[Met_help] [rt.rap.ucar.edu #83986] History for Help with grid_stat on Neighborhood statistics
John Halley Gotway via RT
met_help at ucar.edu
Mon Jun 11 12:22:01 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John,
I am trying to run grid_stat in MET to calculate Neighborhood statistics for
probability of precipitation from some ensemble data with the observations
coming from Stage IV analyses. Unfortunately, I am getting no output (just
headers) in many of the ASCII files from the program when I run it. Could
you help me figure out what I am doing wrong? I have provided the command
line specifics I use as well as the output log file that is created when
running that command. I also have attached the configuration file. I am
having trouble using ftp to send the raw data from any of the clients
available to us. I believe this restriction occurred recently since Bob
Craig had mentioned that he had been able to connect externally in the
past.
I should mention that I have pre-processed the Stage IV analyses already to
be either 0's or 1's for the threshold (>=0.1") since I was unclear as to
whether MET was able to do that on the fly when running grid_stat.
Thanks,
Chris
Python command:
Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
'precip/post-processed-obs.nc', \
'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
Call(metcall)
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 14:49:07 2018
Hi Chris,
I see that you're having difficulty computing neighborhood statistics
in
Grid-Stat.
First, Grid-Stat definitely does have the ability to threshold the
StageIV
data on the fly. So pre-processing to 0's and 1's first is not
needed.
Instead, in the "obs" dictionary of the config file, you'd just set:
cat_thresh = [ >=2.54 ];
Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
If for some reason you have data that's already 0's and 1's, you'd
just use
a different threshold in the config file:
cat_thresh = [ ==1 ];
This tells MET that anywhere you see a 1 in the data, the event is
occurring. Otherwise, it's not.
Now on the to real issue. Looking at your config file, I see that
you're
processing probabilistic data. When "prob = TRUE", MET only computes
the
probabilistic line types (PCT, PSTD, PRC, and PJC). That's why you're
getting no output in the NBRCNT line type, which is the one that
contains
Fractions Skill Score (FSS).
In the attached config file, I've made a few changes:
- Set the obtype to indicate that you're verifying against StageIV
- Defined 2 fcst.field entries.
- In the first we process QP010 as a probability field. Here
cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
for probabilistic verification from 0 to 1 of width 0.1.
- In the second we process it as a field of scalars. Here
cat_thresh
is set to >=0.5. So that's the probability threshold I'm using for
defining events for fractions skill score.
- Defined 2 corresponding obs.field entries to verify the forecast
data.
In both the verifying threshold is >=2.54. This assumes that you pass
the
raw StageIV data as input.
- In the output_flag section, I turned off the line types that don't
apply.
Lastly, while setting "level = [ "R19" ];" works as a way of grabbing
record number 19 from the input file, it's not recommended. Instead,
it'd
be better to figure out how to set the level string so that MET always
grabs the same record, regardless if it's in spot number 19 or 20 or
any
other location in the GRIB file.
Hope this helps clarify. What other questions do you have?
Thanks,
John Halley Gotway
On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> Transaction: Ticket created by christopher.melick at us.af.mil
> Queue: met_help
> Subject: Help with grid_stat on Neighborhood statistics
> Owner: Nobody
> Requestors: christopher.melick at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
>
> Hi John,
>
> I am trying to run grid_stat in MET to calculate Neighborhood
statistics
> for
> probability of precipitation from some ensemble data with the
observations
> coming from Stage IV analyses. Unfortunately, I am getting no
output
> (just
> headers) in many of the ASCII files from the program when I run it.
Could
> you help me figure out what I am doing wrong? I have provided the
command
> line specifics I use as well as the output log file that is created
when
> running that command. I also have attached the configuration file.
I am
> having trouble using ftp to send the raw data from any of the
clients
> available to us. I believe this restriction occurred recently
since Bob
> Craig had mentioned that he had been able to connect externally in
the
> past.
>
> I should mention that I have pre-processed the Stage IV analyses
already to
> be either 0's or 1's for the threshold (>=0.1") since I was unclear
as to
> whether MET was able to do that on the fly when running grid_stat.
>
> Thanks,
> Chris
>
> Python command:
> Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> 'precip/post-processed-obs.nc', \
> 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> Call(metcall)
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 07 15:26:40 2018
Hi John,
Your explanation and suggestions are very helpful and do explain my
problems quite substantially. I will have to give it a try to see
what happens.
I do have a couple of other minor questions dealing with how the
neighborhood verification is performed:
1.) I noticed in the documentation that there is a flag for defining
the shape of the neighborhood area (i.e. circular or square). What is
the default if you don't specify?
2.) In the literature, there are a couple of different ways to handle
generating neighborhood probabilities (e.g., Schwartz and Sobash 2017;
https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I was
curious is the only way to get the Fractions Skill Score (FSS) output
is to define a neighborhood in which MET calculates a fractional
coverage of the threshold event? I have previously explored computing
the neighborhood maximum value within a radius of each grid point
before calculating a probability.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 3:49 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I see that you're having difficulty computing neighborhood statistics
in Grid-Stat.
First, Grid-Stat definitely does have the ability to threshold the
StageIV data on the fly. So pre-processing to 0's and 1's first is
not needed.
Instead, in the "obs" dictionary of the config file, you'd just set:
cat_thresh = [ >=2.54 ];
Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
If for some reason you have data that's already 0's and 1's, you'd
just use a different threshold in the config file:
cat_thresh = [ ==1 ];
This tells MET that anywhere you see a 1 in the data, the event is
occurring. Otherwise, it's not.
Now on the to real issue. Looking at your config file, I see that
you're processing probabilistic data. When "prob = TRUE", MET only
computes the probabilistic line types (PCT, PSTD, PRC, and PJC).
That's why you're getting no output in the NBRCNT line type, which is
the one that contains Fractions Skill Score (FSS).
In the attached config file, I've made a few changes:
- Set the obtype to indicate that you're verifying against StageIV
- Defined 2 fcst.field entries.
- In the first we process QP010 as a probability field. Here
cat_thresh is set to ==0.10 which is shorthand notation for defining
bins for probabilistic verification from 0 to 1 of width 0.1.
- In the second we process it as a field of scalars. Here
cat_thresh is set to >=0.5. So that's the probability threshold I'm
using for defining events for fractions skill score.
- Defined 2 corresponding obs.field entries to verify the forecast
data.
In both the verifying threshold is >=2.54. This assumes that you pass
the raw StageIV data as input.
- In the output_flag section, I turned off the line types that don't
apply.
Lastly, while setting "level = [ "R19" ];" works as a way of grabbing
record number 19 from the input file, it's not recommended. Instead,
it'd be better to figure out how to set the level string so that MET
always grabs the same record, regardless if it's in spot number 19 or
20 or any other location in the GRIB file.
Hope this helps clarify. What other questions do you have?
Thanks,
John Halley Gotway
On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> Transaction: Ticket created by christopher.melick at us.af.mil
> Queue: met_help
> Subject: Help with grid_stat on Neighborhood statistics
> Owner: Nobody
> Requestors: christopher.melick at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
>
>
> Hi John,
>
> I am trying to run grid_stat in MET to calculate Neighborhood
> statistics for probability of precipitation from some ensemble data
> with the observations
> coming from Stage IV analyses. Unfortunately, I am getting no
output
> (just
> headers) in many of the ASCII files from the program when I run it.
Could
> you help me figure out what I am doing wrong? I have provided the
> command line specifics I use as well as the output log file that is
created when
> running that command. I also have attached the configuration file.
I am
> having trouble using ftp to send the raw data from any of the
clients
> available to us. I believe this restriction occurred recently
since Bob
> Craig had mentioned that he had been able to connect externally in
> the past.
>
> I should mention that I have pre-processed the Stage IV analyses
> already to be either 0's or 1's for the threshold (>=0.1") since I
was
> unclear as to whether MET was able to do that on the fly when
running grid_stat.
>
> Thanks,
> Chris
>
> Python command:
> Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> 'precip/post-processed-obs.nc', \
> 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> Call(metcall)
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 15:48:30 2018
Chris,
(1) The default shape is a square since that's what MET previously
did.
The circle is the *new* option.
(2) For this one, the short answer is yes. FSS is computed in MET by
doing
the following...
- For each field separately (i.e. fcst and obs), apply the threshold
(cat_thresh) to convert the input data to a 0/1 bitmap.
- Apply the neighborhood width and shape to compute a fractional
coverage
value for each grid point.
- Compare the fcst and obs fractional coverage fields to compute
FSS.
But there are some other options that you may find useful. The
"interp"
section in the config file is used in Grid-Stat as a smoothing
operation.
After reading the raw input data, you can specify if/how to smooth
that
data prior to computing stats. For example, since you mentioned
maximums,
you could replace the value at each grid point with the maximum of the
values in a 5x5 box surrounding that point:
interp = {
field = FCST;
vld_thresh = 1.0;
shape = SQUARE;
type = [
{ method = MEAREST; width = 1; }, // i.e. no smoothing
{ method = MAX; width = 5; }, // i.e. max of 25 closest points
{ method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
points
];
}
Using the example above, you'll get 3 times the amount of output from
MET.
The first smoother really is no smoothing at all. It would run on the
raw
input data. The second smoother replaces the value at each grid point
with
the maximum of the 25 surrounding points. The third smoother replaces
the
value at each grid point with the mean of those 25.
I worked with Craig Schwartz when he was running MET for that paper.
My
recollection is that he was able to use the configuration options of
Grid-Stat to accomplish the types of processing he was after. But I
don't
recall all the details.
If you explain exactly what you're trying to do, it may be possible to
configure Grid-Stat to accomplish that.
Thanks,
John
On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your explanation and suggestions are very helpful and do explain my
> problems quite substantially. I will have to give it a try to see
what
> happens.
>
> I do have a couple of other minor questions dealing with how the
> neighborhood verification is performed:
> 1.) I noticed in the documentation that there is a flag for
defining the
> shape of the neighborhood area (i.e. circular or square). What is
the
> default if you don't specify?
> 2.) In the literature, there are a couple of different ways to
handle
> generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> curious is the only way to get the Fractions Skill Score (FSS)
output is to
> define a neighborhood in which MET calculates a fractional coverage
of the
> threshold event? I have previously explored computing the
neighborhood
> maximum value within a radius of each grid point before calculating
a
> probability.
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 3:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I see that you're having difficulty computing neighborhood
statistics in
> Grid-Stat.
>
> First, Grid-Stat definitely does have the ability to threshold the
StageIV
> data on the fly. So pre-processing to 0's and 1's first is not
needed.
> Instead, in the "obs" dictionary of the config file, you'd just set:
> cat_thresh = [ >=2.54 ];
>
> Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
>
> If for some reason you have data that's already 0's and 1's, you'd
just
> use a different threshold in the config file:
> cat_thresh = [ ==1 ];
>
> This tells MET that anywhere you see a 1 in the data, the event is
> occurring. Otherwise, it's not.
>
> Now on the to real issue. Looking at your config file, I see that
you're
> processing probabilistic data. When "prob = TRUE", MET only
computes the
> probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> getting no output in the NBRCNT line type, which is the one that
contains
> Fractions Skill Score (FSS).
>
> In the attached config file, I've made a few changes:
> - Set the obtype to indicate that you're verifying against StageIV
> - Defined 2 fcst.field entries.
> - In the first we process QP010 as a probability field. Here
> cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
> for probabilistic verification from 0 to 1 of width 0.1.
> - In the second we process it as a field of scalars. Here
> cat_thresh is set to >=0.5. So that's the probability threshold I'm
using
> for defining events for fractions skill score.
> - Defined 2 corresponding obs.field entries to verify the forecast
data.
> In both the verifying threshold is >=2.54. This assumes that you
pass the
> raw StageIV data as input.
> - In the output_flag section, I turned off the line types that
don't
> apply.
>
> Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> record number 19 from the input file, it's not recommended.
Instead, it'd
> be better to figure out how to set the level string so that MET
always
> grabs the same record, regardless if it's in spot number 19 or 20 or
any
> other location in the GRIB file.
>
> Hope this helps clarify. What other questions do you have?
>
> Thanks,
>
> John Halley Gotway
>
>
>
> On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > Transaction: Ticket created by christopher.melick at us.af.mil
> > Queue: met_help
> > Subject: Help with grid_stat on Neighborhood statistics
> > Owner: Nobody
> > Requestors: christopher.melick at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >
> >
> >
> > Hi John,
> >
> > I am trying to run grid_stat in MET to calculate Neighborhood
> > statistics for probability of precipitation from some ensemble
data
> > with the observations
> > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > (just
> > headers) in many of the ASCII files from the program when I run
it.
> Could
> > you help me figure out what I am doing wrong? I have provided the
> > command line specifics I use as well as the output log file that
is
> created when
> > running that command. I also have attached the configuration
file. I
> am
> > having trouble using ftp to send the raw data from any of the
clients
> > available to us. I believe this restriction occurred recently
since Bob
> > Craig had mentioned that he had been able to connect externally
in
> > the past.
> >
> > I should mention that I have pre-processed the Stage IV analyses
> > already to be either 0's or 1's for the threshold (>=0.1") since I
was
> > unclear as to whether MET was able to do that on the fly when
running
> grid_stat.
> >
> > Thanks,
> > Chris
> >
> > Python command:
> > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > 'precip/post-processed-obs.nc', \
> > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > Call(metcall)
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 07 16:14:07 2018
Hi John,
Your suggestion for accomplishing #2 sounds promising. I need to
think the details through some. On a somewhat related note, is there
a Gaussian smoother flag for smoothing the probabilistic fields? I
know that was mentioned in Craig's paper and I have pursued that
option in the past (but just not using MET).
I guess my basic question is can you apply a neighborhood width that
just includes the grid point maybe because you already have inherently
a probabilistic field with spatial uncertainty? Would that be a width
of 1 or does that also include the nearest grid point? A good
example of this would be a probabilistic forecast issued from the
Storm Prediction Center where the probabilities implicitly assume a
40-km radius of influence (so there is no need to compute a fractional
coverage of the event).
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 4:49 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
(1) The default shape is a square since that's what MET previously
did.
The circle is the *new* option.
(2) For this one, the short answer is yes. FSS is computed in MET by
doing the following...
- For each field separately (i.e. fcst and obs), apply the threshold
(cat_thresh) to convert the input data to a 0/1 bitmap.
- Apply the neighborhood width and shape to compute a fractional
coverage value for each grid point.
- Compare the fcst and obs fractional coverage fields to compute
FSS.
But there are some other options that you may find useful. The
"interp"
section in the config file is used in Grid-Stat as a smoothing
operation.
After reading the raw input data, you can specify if/how to smooth
that data prior to computing stats. For example, since you mentioned
maximums, you could replace the value at each grid point with the
maximum of the values in a 5x5 box surrounding that point:
interp = {
field = FCST;
vld_thresh = 1.0;
shape = SQUARE;
type = [
{ method = MEAREST; width = 1; }, // i.e. no smoothing
{ method = MAX; width = 5; }, // i.e. max of 25 closest points
{ method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest points
];
}
Using the example above, you'll get 3 times the amount of output from
MET.
The first smoother really is no smoothing at all. It would run on the
raw input data. The second smoother replaces the value at each grid
point with the maximum of the 25 surrounding points. The third
smoother replaces the value at each grid point with the mean of those
25.
I worked with Craig Schwartz when he was running MET for that paper.
My recollection is that he was able to use the configuration options
of Grid-Stat to accomplish the types of processing he was after. But
I don't recall all the details.
If you explain exactly what you're trying to do, it may be possible to
configure Grid-Stat to accomplish that.
Thanks,
John
On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your explanation and suggestions are very helpful and do explain my
> problems quite substantially. I will have to give it a try to see
> what happens.
>
> I do have a couple of other minor questions dealing with how the
> neighborhood verification is performed:
> 1.) I noticed in the documentation that there is a flag for
defining
> the shape of the neighborhood area (i.e. circular or square). What
is
> the default if you don't specify?
> 2.) In the literature, there are a couple of different ways to
handle
> generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> curious is the only way to get the Fractions Skill Score (FSS)
output
> is to define a neighborhood in which MET calculates a fractional
> coverage of the threshold event? I have previously explored
computing
> the neighborhood maximum value within a radius of each grid point
> before calculating a probability.
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 3:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I see that you're having difficulty computing neighborhood
statistics in
> Grid-Stat.
>
> First, Grid-Stat definitely does have the ability to threshold the
StageIV
> data on the fly. So pre-processing to 0's and 1's first is not
needed.
> Instead, in the "obs" dictionary of the config file, you'd just set:
> cat_thresh = [ >=2.54 ];
>
> Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
>
> If for some reason you have data that's already 0's and 1's, you'd
just
> use a different threshold in the config file:
> cat_thresh = [ ==1 ];
>
> This tells MET that anywhere you see a 1 in the data, the event is
> occurring. Otherwise, it's not.
>
> Now on the to real issue. Looking at your config file, I see that
you're
> processing probabilistic data. When "prob = TRUE", MET only
computes the
> probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> getting no output in the NBRCNT line type, which is the one that
contains
> Fractions Skill Score (FSS).
>
> In the attached config file, I've made a few changes:
> - Set the obtype to indicate that you're verifying against StageIV
> - Defined 2 fcst.field entries.
> - In the first we process QP010 as a probability field. Here
> cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
> for probabilistic verification from 0 to 1 of width 0.1.
> - In the second we process it as a field of scalars. Here
> cat_thresh is set to >=0.5. So that's the probability threshold I'm
using
> for defining events for fractions skill score.
> - Defined 2 corresponding obs.field entries to verify the forecast
data.
> In both the verifying threshold is >=2.54. This assumes that you
pass the
> raw StageIV data as input.
> - In the output_flag section, I turned off the line types that
don't
> apply.
>
> Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> record number 19 from the input file, it's not recommended.
Instead, it'd
> be better to figure out how to set the level string so that MET
always
> grabs the same record, regardless if it's in spot number 19 or 20 or
any
> other location in the GRIB file.
>
> Hope this helps clarify. What other questions do you have?
>
> Thanks,
>
> John Halley Gotway
>
>
>
> On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > Transaction: Ticket created by christopher.melick at us.af.mil
> > Queue: met_help
> > Subject: Help with grid_stat on Neighborhood statistics
> > Owner: Nobody
> > Requestors: christopher.melick at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >
> >
> >
> > Hi John,
> >
> > I am trying to run grid_stat in MET to calculate Neighborhood
> > statistics for probability of precipitation from some ensemble
data
> > with the observations
> > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > (just
> > headers) in many of the ASCII files from the program when I run
it.
> Could
> > you help me figure out what I am doing wrong? I have provided the
> > command line specifics I use as well as the output log file that
is
> created when
> > running that command. I also have attached the configuration
file. I
> am
> > having trouble using ftp to send the raw data from any of the
clients
> > available to us. I believe this restriction occurred recently
since Bob
> > Craig had mentioned that he had been able to connect externally
in
> > the past.
> >
> > I should mention that I have pre-processed the Stage IV analyses
> > already to be either 0's or 1's for the threshold (>=0.1") since I
was
> > unclear as to whether MET was able to do that on the fly when
running
> grid_stat.
> >
> > Thanks,
> > Chris
> >
> > Python command:
> > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > 'precip/post-processed-obs.nc', \
> > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > Call(metcall)
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 16:35:49 2018
Chris,
In fact, SPC's 40-km radius probabilities were the motivation for
adding
the shape = circle option to MET.
For your first question, no, there is not currently a Gaussian
smoother
option for the "interp" section in Grid-Stat. Currently, the only
valid
options there are MIN, MAX, MEDIAN, and UW_MEAN (for unweighted mean).
There is a distance-weighted mean option for point data where the
weight is
1/distance squared. But that doesn't work for Grid-Stat because we'd
end
up dividing by 0.
I can see how adding a gaussian smoother would be useful.
Here's what we've done in the past with this SPC 40-km probabilities:
(1) Evaluate that field as a probability field (i.e. set "prob =
TRUE;" in
the config file).
(2) Use the "interp" section to define a smoothing operation on the
observations:
interp = {
field = OBS;
vld_thresh = 1.0;
shape = CIRCLE;
type = [
{ method = MAX; width = 11; }
];
}
Or set width to some circle diameter in grid units that works out
to
40km in the real world.
In this example, we post-process the observations to make them
comparable
the event for which the probability was defined.
But I think I understand what you'd like to do... interpret as the
probability value at each grid point as if it was a fractional
coverage
value. So don't apply a threshold and neighborhood size to the
forecast
data at all. Just pass it directly to the routine that computes FSS.
Unfortunately no, I don't think there's a way of configuring Grid-Stat
to
do that in met-6.1. If you think this logic would be generally
useful, we
could consider adding a flag to the config file to enable that logic.
Sorry I don't have better answers for you.
John
On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your suggestion for accomplishing #2 sounds promising. I need to
think
> the details through some. On a somewhat related note, is there a
Gaussian
> smoother flag for smoothing the probabilistic fields? I know that
was
> mentioned in Craig's paper and I have pursued that option in the
past (but
> just not using MET).
>
> I guess my basic question is can you apply a neighborhood width that
just
> includes the grid point maybe because you already have inherently a
> probabilistic field with spatial uncertainty? Would that be a width
of 1
> or does that also include the nearest grid point? A good example
of this
> would be a probabilistic forecast issued from the Storm Prediction
Center
> where the probabilities implicitly assume a 40-km radius of
influence (so
> there is no need to compute a fractional coverage of the event).
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 4:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> (1) The default shape is a square since that's what MET previously
did.
> The circle is the *new* option.
>
> (2) For this one, the short answer is yes. FSS is computed in MET
by
> doing the following...
> - For each field separately (i.e. fcst and obs), apply the
threshold
> (cat_thresh) to convert the input data to a 0/1 bitmap.
> - Apply the neighborhood width and shape to compute a fractional
> coverage value for each grid point.
> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>
> But there are some other options that you may find useful. The
"interp"
> section in the config file is used in Grid-Stat as a smoothing
operation.
> After reading the raw input data, you can specify if/how to smooth
that
> data prior to computing stats. For example, since you mentioned
maximums,
> you could replace the value at each grid point with the maximum of
the
> values in a 5x5 box surrounding that point:
>
> interp = {
> field = FCST;
> vld_thresh = 1.0;
> shape = SQUARE;
>
> type = [
> { method = MEAREST; width = 1; }, // i.e. no smoothing
> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
> points
> ];
> }
>
> Using the example above, you'll get 3 times the amount of output
from MET.
> The first smoother really is no smoothing at all. It would run on
the raw
> input data. The second smoother replaces the value at each grid
point with
> the maximum of the 25 surrounding points. The third smoother
replaces the
> value at each grid point with the mean of those 25.
>
> I worked with Craig Schwartz when he was running MET for that paper.
My
> recollection is that he was able to use the configuration options of
> Grid-Stat to accomplish the types of processing he was after. But I
don't
> recall all the details.
>
> If you explain exactly what you're trying to do, it may be possible
to
> configure Grid-Stat to accomplish that.
>
> Thanks,
> John
>
> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Your explanation and suggestions are very helpful and do explain
my
> > problems quite substantially. I will have to give it a try to see
> > what happens.
> >
> > I do have a couple of other minor questions dealing with how the
> > neighborhood verification is performed:
> > 1.) I noticed in the documentation that there is a flag for
defining
> > the shape of the neighborhood area (i.e. circular or square).
What is
> > the default if you don't specify?
> > 2.) In the literature, there are a couple of different ways to
handle
> > generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> > curious is the only way to get the Fractions Skill Score (FSS)
output
> > is to define a neighborhood in which MET calculates a fractional
> > coverage of the threshold event? I have previously explored
computing
> > the neighborhood maximum value within a radius of each grid point
> > before calculating a probability.
> >
> > Thanks,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 7, 2018 3:49 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I see that you're having difficulty computing neighborhood
statistics in
> > Grid-Stat.
> >
> > First, Grid-Stat definitely does have the ability to threshold the
> StageIV
> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> > cat_thresh = [ >=2.54 ];
> >
> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> inches.
> >
> > If for some reason you have data that's already 0's and 1's, you'd
just
> > use a different threshold in the config file:
> > cat_thresh = [ ==1 ];
> >
> > This tells MET that anywhere you see a 1 in the data, the event is
> > occurring. Otherwise, it's not.
> >
> > Now on the to real issue. Looking at your config file, I see that
you're
> > processing probabilistic data. When "prob = TRUE", MET only
computes the
> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> > getting no output in the NBRCNT line type, which is the one that
contains
> > Fractions Skill Score (FSS).
> >
> > In the attached config file, I've made a few changes:
> > - Set the obtype to indicate that you're verifying against
StageIV
> > - Defined 2 fcst.field entries.
> > - In the first we process QP010 as a probability field.
Here
> > cat_thresh is set to ==0.10 which is shorthand notation for
defining bins
> > for probabilistic verification from 0 to 1 of width 0.1.
> > - In the second we process it as a field of scalars. Here
> > cat_thresh is set to >=0.5. So that's the probability threshold
I'm
> using
> > for defining events for fractions skill score.
> > - Defined 2 corresponding obs.field entries to verify the
forecast data.
> > In both the verifying threshold is >=2.54. This assumes that you
pass the
> > raw StageIV data as input.
> > - In the output_flag section, I turned off the line types that
don't
> > apply.
> >
> > Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> > record number 19 from the input file, it's not recommended.
Instead,
> it'd
> > be better to figure out how to set the level string so that MET
always
> > grabs the same record, regardless if it's in spot number 19 or 20
or any
> > other location in the GRIB file.
> >
> > Hope this helps clarify. What other questions do you have?
> >
> > Thanks,
> >
> > John Halley Gotway
> >
> >
> >
> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > Transaction: Ticket created by christopher.melick at us.af.mil
> > > Queue: met_help
> > > Subject: Help with grid_stat on Neighborhood statistics
> > > Owner: Nobody
> > > Requestors: christopher.melick at us.af.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > >
> > >
> > >
> > > Hi John,
> > >
> > > I am trying to run grid_stat in MET to calculate Neighborhood
> > > statistics for probability of precipitation from some ensemble
data
> > > with the observations
> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > > (just
> > > headers) in many of the ASCII files from the program when I run
it.
> > Could
> > > you help me figure out what I am doing wrong? I have provided
the
> > > command line specifics I use as well as the output log file that
is
> > created when
> > > running that command. I also have attached the configuration
file. I
> > am
> > > having trouble using ftp to send the raw data from any of the
clients
> > > available to us. I believe this restriction occurred recently
since
> Bob
> > > Craig had mentioned that he had been able to connect externally
in
> > > the past.
> > >
> > > I should mention that I have pre-processed the Stage IV analyses
> > > already to be either 0's or 1's for the threshold (>=0.1") since
I was
> > > unclear as to whether MET was able to do that on the fly when
running
> > grid_stat.
> > >
> > > Thanks,
> > > Chris
> > >
> > > Python command:
> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > > 'precip/post-processed-obs.nc', \
> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > > Call(metcall)
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 18:13:44 2018
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input
probability data. We could add a “field” option to the “nbrhd”
section,
just like we have in the “interp” section. It would be set to FCST,
OBS,
or BOTH (default) and would control the logic as to which fields
should be
passed through the logic for deriving fractional coverage fields. In
your
case, you’d set it to OBS and we’d just use the raw input forecast
probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
adding
> the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
smoother
> option for the "interp" section in Grid-Stat. Currently, the only
valid
> options there are MIN, MAX, MEDIAN, and UW_MEAN (for unweighted
mean).
> There is a distance-weighted mean option for point data where the
weight is
> 1/distance squared. But that doesn't work for Grid-Stat because
we'd end
> up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
TRUE;" in
> the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
to
> 40km in the real world.
>
> In this example, we post-process the observations to make them
comparable
> the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
coverage
> value. So don't apply a threshold and neighborhood size to the
forecast
> data at all. Just pass it directly to the routine that computes
FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat to
> do that in met-6.1. If you think this logic would be generally
useful, we
> could consider adding a flag to the config file to enable that
logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past (but
>> just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that just
>> includes the grid point maybe because you already have inherently a
>> probabilistic field with spatial uncertainty? Would that be a
width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
Center
>> where the probabilities implicitly assume a 40-km radius of
influence (so
>> there is no need to compute a fractional coverage of the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian
>> Meteorologist, Verification Exploitation Team
>> 16th Weather Squadron (16 WS/WXEV)
>> 557th Weather Wing, Offutt AFB, NE
>> DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
>> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
that
>> data prior to computing stats. For example, since you mentioned
maximums,
>> you could replace the value at each grid point with the maximum of
the
>> values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
>> points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
the
>> raw input data. The second smoother replaces the value at each
grid point
>> with the maximum of the 25 surrounding points. The third smoother
replaces
>> the value at each grid point with the mean of those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper. My
>> recollection is that he was able to use the configuration options
of
>> Grid-Stat to accomplish the types of processing he was after. But
I don't
>> recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
to
>> configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
defining
>> > the shape of the neighborhood area (i.e. circular or square).
What is
>> > the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
handle
>> > generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I was
>> > curious is the only way to get the Fractions Skill Score (FSS)
output
>> > is to define a neighborhood in which MET calculates a fractional
>> > coverage of the threshold event? I have previously explored
computing
>> > the neighborhood maximum value within a radius of each grid point
>> > before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
statistics in
>> > Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd just
>> > use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
>> > getting no output in the NBRCNT line type, which is the one that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
don't
>> > apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
>> > record number 19 from the input file, it's not recommended.
Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
always
>> > grabs the same record, regardless if it's in spot number 19 or 20
or any
>> > other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
data
>> > > with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
the
>> > > command line specifics I use as well as the output log file
that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally in
>> > > the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since I was
>> > > unclear as to whether MET was able to do that on the fly when
running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Thu Feb 08 08:00:11 2018
Hi John,
That sounds like a great solution to the situation I proposed for
computing FSS. My only other suggestion would be to add an additional
option in the case where both the OBS and FCST have been processed
before- hand to be probabilistic. In that case, you could call it
NONE. The default could still be BOTH, though.
How does that sound?
As for now, I will use what is available and it is completely
appropriate. I just wanted to make sure how everything was done
behind the scenes so I can explain to others how FSS is determined (if
it ever comes up). It would be nice to have down the road, though,
more flexibility built into how that metric is computed (not just for
myself but for others as well). I greatly appreciate your assistance.
On a slightly different note, is there a particular reference you
would recommend for citing MET when using the statistical output in
presentations?
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 7:14 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input probability data. We could add a “field” option to the “nbrhd”
section, just like we have in the “interp” section. It would be set
to FCST, OBS, or BOTH (default) and would control the logic as to
which fields should be passed through the logic for deriving
fractional coverage fields. In your case, you’d set it to OBS and
we’d just use the raw input forecast probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
> adding the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
> smoother option for the "interp" section in Grid-Stat. Currently,
the
> only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
unweighted mean).
> There is a distance-weighted mean option for point data where the
> weight is 1/distance squared. But that doesn't work for Grid-Stat
> because we'd end up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
> TRUE;" in the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
> to 40km in the real world.
>
> In this example, we post-process the observations to make them
> comparable the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
> coverage value. So don't apply a threshold and neighborhood size to
> the forecast data at all. Just pass it directly to the routine that
computes FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> to do that in met-6.1. If you think this logic would be generally
> useful, we could consider adding a flag to the config file to enable
that logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past
>> (but just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that
>> just includes the grid point maybe because you already have
>> inherently a probabilistic field with spatial uncertainty? Would
that be a width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
>> Center where the probabilities implicitly assume a 40-km radius of
>> influence (so there is no need to compute a fractional coverage of
the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
>> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
>> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
>> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
>> WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
>> threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
>> that data prior to computing stats. For example, since you
mentioned
>> maximums, you could replace the value at each grid point with the
>> maximum of the values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
>> closest points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
>> the raw input data. The second smoother replaces the value at each
>> grid point with the maximum of the 25 surrounding points. The
third
>> smoother replaces the value at each grid point with the mean of
those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper.
>> My recollection is that he was able to use the configuration
options
>> of Grid-Stat to accomplish the types of processing he was after.
But
>> I don't recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
>> to configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
>> > defining the shape of the neighborhood area (i.e. circular or
>> > square). What is the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
>> > handle generating neighborhood probabilities (e.g., Schwartz and
>> > Sobash 2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
>> > was curious is the only way to get the Fractions Skill Score
(FSS)
>> > output is to define a neighborhood in which MET calculates a
>> > fractional coverage of the threshold event? I have previously
>> > explored computing the neighborhood maximum value within a radius
of each grid point before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
>> > statistics in Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd
>> > just use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
>> > computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
>> > you're getting no output in the NBRCNT line type, which is the
one
>> > that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
>> > defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
>> > I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
>> > pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
>> > don't apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
>> > grabbing record number 19 from the input file, it's not
>> > recommended. Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
>> > always grabs the same record, regardless if it's in spot number
19
>> > or 20 or any other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT
>> > < met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
>> > > data with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
>> > > the command line specifics I use as well as the output log file
>> > > that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally
>> > > in the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since
>> > > I was unclear as to whether MET was able to do that on the fly
>> > > when running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
>> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
>> > > 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Thu Feb 08 14:09:23 2018
Hi John,
Getting back to your initial response, I was able to run grid_stat
with your changes in the attached configuration file to get FSS output
(as well as the other statistics for the neighborhood type results).
However, I do have a question about the forecast categorical threshold
(cat_thresh is set to >=0.5) used in determining the neighborhood
probabilities from the ensemble probabilities: What is the actual
formula that MET uses to create the final field to calculate FSS? Is
it a strict neighborhood fractional coverage of ensemble probabilities
above 0.5 in some set radius of influence? Or is it some other
operation or does the threshold represent something else? I know in
Craig's paper, they were calling the last step a neighborhood average
of the ensemble probabilities to calculate NEP (since the event had
already been defined at the grid point in determining the ensemble
probability [i.e. likelihood of exceeding 0.1" of precipitation]).
Maybe, I should understand how Craig was using MET for his paper.
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 7:14 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input probability data. We could add a “field” option to the “nbrhd”
section, just like we have in the “interp” section. It would be set
to FCST, OBS, or BOTH (default) and would control the logic as to
which fields should be passed through the logic for deriving
fractional coverage fields. In your case, you’d set it to OBS and
we’d just use the raw input forecast probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
> adding the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
> smoother option for the "interp" section in Grid-Stat. Currently,
the
> only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
unweighted mean).
> There is a distance-weighted mean option for point data where the
> weight is 1/distance squared. But that doesn't work for Grid-Stat
> because we'd end up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
> TRUE;" in the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
> to 40km in the real world.
>
> In this example, we post-process the observations to make them
> comparable the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
> coverage value. So don't apply a threshold and neighborhood size to
> the forecast data at all. Just pass it directly to the routine that
computes FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> to do that in met-6.1. If you think this logic would be generally
> useful, we could consider adding a flag to the config file to enable
that logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past
>> (but just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that
>> just includes the grid point maybe because you already have
>> inherently a probabilistic field with spatial uncertainty? Would
that be a width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
>> Center where the probabilities implicitly assume a 40-km radius of
>> influence (so there is no need to compute a fractional coverage of
the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
>> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
>> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
>> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
>> WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
>> threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
>> that data prior to computing stats. For example, since you
mentioned
>> maximums, you could replace the value at each grid point with the
>> maximum of the values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
>> closest points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
>> the raw input data. The second smoother replaces the value at each
>> grid point with the maximum of the 25 surrounding points. The
third
>> smoother replaces the value at each grid point with the mean of
those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper.
>> My recollection is that he was able to use the configuration
options
>> of Grid-Stat to accomplish the types of processing he was after.
But
>> I don't recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
>> to configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
>> > defining the shape of the neighborhood area (i.e. circular or
>> > square). What is the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
>> > handle generating neighborhood probabilities (e.g., Schwartz and
>> > Sobash 2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
>> > was curious is the only way to get the Fractions Skill Score
(FSS)
>> > output is to define a neighborhood in which MET calculates a
>> > fractional coverage of the threshold event? I have previously
>> > explored computing the neighborhood maximum value within a radius
of each grid point before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
>> > statistics in Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd
>> > just use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
>> > computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
>> > you're getting no output in the NBRCNT line type, which is the
one
>> > that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
>> > defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
>> > I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
>> > pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
>> > don't apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
>> > grabbing record number 19 from the input file, it's not
>> > recommended. Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
>> > always grabs the same record, regardless if it's in spot number
19
>> > or 20 or any other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT
>> > < met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
>> > > data with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
>> > > the command line specifics I use as well as the output log file
>> > > that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally
>> > > in the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since
>> > > I was unclear as to whether MET was able to do that on the fly
>> > > when running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
>> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
>> > > 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Thu Feb 08 14:24:26 2018
Chris,
It sounds like you understand how the fractional coverage field is
computed.
- Read the raw field... probability values between 0 and 1 in your
case.
- Apply the categorical threshold to that data to generate a field of
0's
and 1's... >0.5 in your case.
- For each grid point, replace the value at that grid point with the
"fraction" of 1's in the neighborhood around that point... e.g. the
event
occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
point, so fractional coverage = 4/9.
- Compare the forecast fractional coverage values to the observed
fractional coverage values to compute FSS.
Yes, I do think it would be a good idea to talk to Craig about how he
configured MET for his work.
Thanks,
John
On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Getting back to your initial response, I was able to run grid_stat
with
> your changes in the attached configuration file to get FSS output
(as well
> as the other statistics for the neighborhood type results).
>
> However, I do have a question about the forecast categorical
threshold
> (cat_thresh is set to >=0.5) used in determining the neighborhood
> probabilities from the ensemble probabilities: What is the actual
formula
> that MET uses to create the final field to calculate FSS? Is it a
strict
> neighborhood fractional coverage of ensemble probabilities above 0.5
in
> some set radius of influence? Or is it some other operation or does
the
> threshold represent something else? I know in Craig's paper, they
were
> calling the last step a neighborhood average of the ensemble
probabilities
> to calculate NEP (since the event had already been defined at the
grid
> point in determining the ensemble probability [i.e. likelihood of
exceeding
> 0.1" of precipitation]). Maybe, I should understand how Craig was
using
> MET for his paper.
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 7:14 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I was thinking about what changes would be needed to compute FSS on
input
> probability data. We could add a “field” option to the “nbrhd”
section,
> just like we have in the “interp” section. It would be set to FCST,
OBS,
> or BOTH (default) and would control the logic as to which fields
should be
> passed through the logic for deriving fractional coverage fields.
In your
> case, you’d set it to OBS and we’d just use the raw input forecast
> probability values.
>
> Would that logic make sense to you?
>
> Thanks
> John
>
> On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Chris,
> >
> > In fact, SPC's 40-km radius probabilities were the motivation for
> > adding the shape = circle option to MET.
> >
> > For your first question, no, there is not currently a Gaussian
> > smoother option for the "interp" section in Grid-Stat. Currently,
the
> > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> unweighted mean).
> > There is a distance-weighted mean option for point data where the
> > weight is 1/distance squared. But that doesn't work for Grid-Stat
> > because we'd end up dividing by 0.
> >
> > I can see how adding a gaussian smoother would be useful.
> >
> > Here's what we've done in the past with this SPC 40-km
probabilities:
> >
> > (1) Evaluate that field as a probability field (i.e. set "prob =
> > TRUE;" in the config file).
> > (2) Use the "interp" section to define a smoothing operation on
the
> > observations:
> > interp = {
> > field = OBS;
> > vld_thresh = 1.0;
> > shape = CIRCLE;
> > type = [
> > { method = MAX; width = 11; }
> > ];
> > }
> > Or set width to some circle diameter in grid units that works
out
> > to 40km in the real world.
> >
> > In this example, we post-process the observations to make them
> > comparable the event for which the probability was defined.
> >
> > But I think I understand what you'd like to do... interpret as the
> > probability value at each grid point as if it was a fractional
> > coverage value. So don't apply a threshold and neighborhood size
to
> > the forecast data at all. Just pass it directly to the routine
that
> computes FSS.
> >
> > Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> > to do that in met-6.1. If you think this logic would be generally
> > useful, we could consider adding a flag to the config file to
enable
> that logic.
> >
> > Sorry I don't have better answers for you.
> >
> > John
> >
> >
> >
> > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >>
> >> Hi John,
> >>
> >> Your suggestion for accomplishing #2 sounds promising. I need to
think
> >> the details through some. On a somewhat related note, is there
a
> Gaussian
> >> smoother flag for smoothing the probabilistic fields? I know
that was
> >> mentioned in Craig's paper and I have pursued that option in the
past
> >> (but just not using MET).
> >>
> >> I guess my basic question is can you apply a neighborhood width
that
> >> just includes the grid point maybe because you already have
> >> inherently a probabilistic field with spatial uncertainty? Would
that
> be a width of 1
> >> or does that also include the nearest grid point? A good
example of
> this
> >> would be a probabilistic forecast issued from the Storm
Prediction
> >> Center where the probabilities implicitly assume a 40-km radius
of
> >> influence (so there is no need to compute a fractional coverage
of the
> event).
> >>
> >> Thanks again,
> >> Chris
> >>
> >>
> >> //SIGNED//
> >>
> >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, February 7, 2018 4:49 PM
> >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> christopher.melick at us.af.mil>
> >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> >> WS/WXN <robert.craig.2 at us.af.mil>
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> grid_stat on Neighborhood statistics
> >>
> >> Chris,
> >>
> >> (1) The default shape is a square since that's what MET
previously did.
> >> The circle is the *new* option.
> >>
> >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> >> doing the following...
> >> - For each field separately (i.e. fcst and obs), apply the
> >> threshold
> >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> >> - Apply the neighborhood width and shape to compute a
fractional
> >> coverage value for each grid point.
> >> - Compare the fcst and obs fractional coverage fields to
compute FSS.
> >>
> >> But there are some other options that you may find useful. The
"interp"
> >> section in the config file is used in Grid-Stat as a smoothing
> operation.
> >> After reading the raw input data, you can specify if/how to
smooth
> >> that data prior to computing stats. For example, since you
mentioned
> >> maximums, you could replace the value at each grid point with the
> >> maximum of the values in a 5x5 box surrounding that point:
> >>
> >> interp = {
> >> field = FCST;
> >> vld_thresh = 1.0;
> >> shape = SQUARE;
> >>
> >> type = [
> >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> >> closest points
> >> ];
> >> }
> >>
> >> Using the example above, you'll get 3 times the amount of output
from
> MET.
> >> The first smoother really is no smoothing at all. It would run
on
> >> the raw input data. The second smoother replaces the value at
each
> >> grid point with the maximum of the 25 surrounding points. The
third
> >> smoother replaces the value at each grid point with the mean of
those
> 25.
> >>
> >> I worked with Craig Schwartz when he was running MET for that
paper.
> >> My recollection is that he was able to use the configuration
options
> >> of Grid-Stat to accomplish the types of processing he was after.
But
> >> I don't recall all the details.
> >>
> >> If you explain exactly what you're trying to do, it may be
possible
> >> to configure Grid-Stat to accomplish that.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >> >
> >> > Hi John,
> >> >
> >> > Your explanation and suggestions are very helpful and do
explain my
> >> > problems quite substantially. I will have to give it a try to
see
> >> > what happens.
> >> >
> >> > I do have a couple of other minor questions dealing with how
the
> >> > neighborhood verification is performed:
> >> > 1.) I noticed in the documentation that there is a flag for
> >> > defining the shape of the neighborhood area (i.e. circular or
> >> > square). What is the default if you don't specify?
> >> > 2.) In the literature, there are a couple of different ways to
> >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> >> > Sobash 2017;
> >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
> >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> >> > output is to define a neighborhood in which MET calculates a
> >> > fractional coverage of the threshold event? I have previously
> >> > explored computing the neighborhood maximum value within a
radius of
> each grid point before calculating a probability.
> >> >
> >> > Thanks,
> >> > Chris
> >> >
> >> >
> >> > //SIGNED//
> >> >
> >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, February 7, 2018 3:49 PM
> >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> > christopher.melick at us.af.mil>
> >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> >> matthew.dawson.8 at us.af.mil>;
> >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> > grid_stat on Neighborhood statistics
> >> >
> >> > Hi Chris,
> >> >
> >> > I see that you're having difficulty computing neighborhood
> >> > statistics in Grid-Stat.
> >> >
> >> > First, Grid-Stat definitely does have the ability to threshold
the
> >> StageIV
> >> > data on the fly. So pre-processing to 0's and 1's first is not
> needed.
> >> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> >> > cat_thresh = [ >=2.54 ];
> >> >
> >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> >> inches.
> >> >
> >> > If for some reason you have data that's already 0's and 1's,
you'd
> >> > just use a different threshold in the config file:
> >> > cat_thresh = [ ==1 ];
> >> >
> >> > This tells MET that anywhere you see a 1 in the data, the event
is
> >> > occurring. Otherwise, it's not.
> >> >
> >> > Now on the to real issue. Looking at your config file, I see
that
> >> you're
> >> > processing probabilistic data. When "prob = TRUE", MET only
> >> > computes
> >> the
> >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
> >> > you're getting no output in the NBRCNT line type, which is the
one
> >> > that
> >> contains
> >> > Fractions Skill Score (FSS).
> >> >
> >> > In the attached config file, I've made a few changes:
> >> > - Set the obtype to indicate that you're verifying against
StageIV
> >> > - Defined 2 fcst.field entries.
> >> > - In the first we process QP010 as a probability field.
Here
> >> > cat_thresh is set to ==0.10 which is shorthand notation for
> >> > defining
> >> bins
> >> > for probabilistic verification from 0 to 1 of width 0.1.
> >> > - In the second we process it as a field of scalars.
Here
> >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> >> > I'm
> >> using
> >> > for defining events for fractions skill score.
> >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> >> data.
> >> > In both the verifying threshold is >=2.54. This assumes that
you
> >> > pass
> >> the
> >> > raw StageIV data as input.
> >> > - In the output_flag section, I turned off the line types that
> >> > don't apply.
> >> >
> >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> >> > grabbing record number 19 from the input file, it's not
> >> > recommended. Instead,
> >> it'd
> >> > be better to figure out how to set the level string so that MET
> >> > always grabs the same record, regardless if it's in spot number
19
> >> > or 20 or any other location in the GRIB file.
> >> >
> >> > Hope this helps clarify. What other questions do you have?
> >> >
> >> > Thanks,
> >> >
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> >> > < met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: Help with grid_stat on Neighborhood statistics
> >> > > Owner: Nobody
> >> > > Requestors: christopher.melick at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >> > > >
> >> > >
> >> > >
> >> > > Hi John,
> >> > >
> >> > > I am trying to run grid_stat in MET to calculate Neighborhood
> >> > > statistics for probability of precipitation from some
ensemble
> >> > > data with the observations
> >> > > coming from Stage IV analyses. Unfortunately, I am getting
no
> output
> >> > > (just
> >> > > headers) in many of the ASCII files from the program when I
run it.
> >> > Could
> >> > > you help me figure out what I am doing wrong? I have
provided
> >> > > the command line specifics I use as well as the output log
file
> >> > > that is
> >> > created when
> >> > > running that command. I also have attached the
configuration file.
> >> I
> >> > am
> >> > > having trouble using ftp to send the raw data from any of the
> clients
> >> > > available to us. I believe this restriction occurred
recently
> since
> >> Bob
> >> > > Craig had mentioned that he had been able to connect
externally
> >> > > in the past.
> >> > >
> >> > > I should mention that I have pre-processed the Stage IV
analyses
> >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> >> > > I was unclear as to whether MET was able to do that on the
fly
> >> > > when running
> >> > grid_stat.
> >> > >
> >> > > Thanks,
> >> > > Chris
> >> > >
> >> > > Python command:
> >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> >> > > 'precip/post-processed-obs.nc', \
> >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> >> > > Call(metcall)
> >> > >
> >> > > //SIGNED//
> >> > >
> >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> >> > > 294-3693
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >>
> >>
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Fri Feb 09 08:10:22 2018
Hi John,
Yes, thanks for the explanations. That clears up any possible
ambiguity and confusion. I kind of thought that was the procedure.
When do you think that the next MET would be released with the
enhancement that we discussed for the neighborhood statistics? Is it
an easy or difficult update to the software?
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 8, 2018 3:24 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
It sounds like you understand how the fractional coverage field is
computed.
- Read the raw field... probability values between 0 and 1 in your
case.
- Apply the categorical threshold to that data to generate a field of
0's and 1's... >0.5 in your case.
- For each grid point, replace the value at that grid point with the
"fraction" of 1's in the neighborhood around that point... e.g. the
event occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current point, so fractional coverage = 4/9.
- Compare the forecast fractional coverage values to the observed
fractional coverage values to compute FSS.
Yes, I do think it would be a good idea to talk to Craig about how he
configured MET for his work.
Thanks,
John
On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Getting back to your initial response, I was able to run grid_stat
> with your changes in the attached configuration file to get FSS
output
> (as well as the other statistics for the neighborhood type results).
>
> However, I do have a question about the forecast categorical
threshold
> (cat_thresh is set to >=0.5) used in determining the neighborhood
> probabilities from the ensemble probabilities: What is the actual
> formula that MET uses to create the final field to calculate FSS?
Is
> it a strict neighborhood fractional coverage of ensemble
probabilities
> above 0.5 in some set radius of influence? Or is it some other
> operation or does the threshold represent something else? I know in
> Craig's paper, they were calling the last step a neighborhood
average
> of the ensemble probabilities to calculate NEP (since the event had
> already been defined at the grid point in determining the ensemble
probability [i.e. likelihood of exceeding
> 0.1" of precipitation]). Maybe, I should understand how Craig was
using
> MET for his paper.
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 7:14 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I was thinking about what changes would be needed to compute FSS on
input
> probability data. We could add a “field” option to the “nbrhd”
section,
> just like we have in the “interp” section. It would be set to FCST,
OBS,
> or BOTH (default) and would control the logic as to which fields
should be
> passed through the logic for deriving fractional coverage fields.
In your
> case, you’d set it to OBS and we’d just use the raw input forecast
> probability values.
>
> Would that logic make sense to you?
>
> Thanks
> John
>
> On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Chris,
> >
> > In fact, SPC's 40-km radius probabilities were the motivation for
> > adding the shape = circle option to MET.
> >
> > For your first question, no, there is not currently a Gaussian
> > smoother option for the "interp" section in Grid-Stat. Currently,
the
> > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> unweighted mean).
> > There is a distance-weighted mean option for point data where the
> > weight is 1/distance squared. But that doesn't work for Grid-Stat
> > because we'd end up dividing by 0.
> >
> > I can see how adding a gaussian smoother would be useful.
> >
> > Here's what we've done in the past with this SPC 40-km
probabilities:
> >
> > (1) Evaluate that field as a probability field (i.e. set "prob =
> > TRUE;" in the config file).
> > (2) Use the "interp" section to define a smoothing operation on
the
> > observations:
> > interp = {
> > field = OBS;
> > vld_thresh = 1.0;
> > shape = CIRCLE;
> > type = [
> > { method = MAX; width = 11; }
> > ];
> > }
> > Or set width to some circle diameter in grid units that works
out
> > to 40km in the real world.
> >
> > In this example, we post-process the observations to make them
> > comparable the event for which the probability was defined.
> >
> > But I think I understand what you'd like to do... interpret as the
> > probability value at each grid point as if it was a fractional
> > coverage value. So don't apply a threshold and neighborhood size
to
> > the forecast data at all. Just pass it directly to the routine
that
> computes FSS.
> >
> > Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> > to do that in met-6.1. If you think this logic would be generally
> > useful, we could consider adding a flag to the config file to
enable
> that logic.
> >
> > Sorry I don't have better answers for you.
> >
> > John
> >
> >
> >
> > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >>
> >> Hi John,
> >>
> >> Your suggestion for accomplishing #2 sounds promising. I need to
think
> >> the details through some. On a somewhat related note, is there
a
> Gaussian
> >> smoother flag for smoothing the probabilistic fields? I know
that was
> >> mentioned in Craig's paper and I have pursued that option in the
past
> >> (but just not using MET).
> >>
> >> I guess my basic question is can you apply a neighborhood width
that
> >> just includes the grid point maybe because you already have
> >> inherently a probabilistic field with spatial uncertainty? Would
that
> be a width of 1
> >> or does that also include the nearest grid point? A good
example of
> this
> >> would be a probabilistic forecast issued from the Storm
Prediction
> >> Center where the probabilities implicitly assume a 40-km radius
of
> >> influence (so there is no need to compute a fractional coverage
of the
> event).
> >>
> >> Thanks again,
> >> Chris
> >>
> >>
> >> //SIGNED//
> >>
> >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, February 7, 2018 4:49 PM
> >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> christopher.melick at us.af.mil>
> >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> >> WS/WXN <robert.craig.2 at us.af.mil>
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> grid_stat on Neighborhood statistics
> >>
> >> Chris,
> >>
> >> (1) The default shape is a square since that's what MET
previously did.
> >> The circle is the *new* option.
> >>
> >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> >> doing the following...
> >> - For each field separately (i.e. fcst and obs), apply the
> >> threshold
> >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> >> - Apply the neighborhood width and shape to compute a
fractional
> >> coverage value for each grid point.
> >> - Compare the fcst and obs fractional coverage fields to
compute FSS.
> >>
> >> But there are some other options that you may find useful. The
"interp"
> >> section in the config file is used in Grid-Stat as a smoothing
> operation.
> >> After reading the raw input data, you can specify if/how to
smooth
> >> that data prior to computing stats. For example, since you
mentioned
> >> maximums, you could replace the value at each grid point with the
> >> maximum of the values in a 5x5 box surrounding that point:
> >>
> >> interp = {
> >> field = FCST;
> >> vld_thresh = 1.0;
> >> shape = SQUARE;
> >>
> >> type = [
> >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> >> closest points
> >> ];
> >> }
> >>
> >> Using the example above, you'll get 3 times the amount of output
from
> MET.
> >> The first smoother really is no smoothing at all. It would run
on
> >> the raw input data. The second smoother replaces the value at
each
> >> grid point with the maximum of the 25 surrounding points. The
third
> >> smoother replaces the value at each grid point with the mean of
those
> 25.
> >>
> >> I worked with Craig Schwartz when he was running MET for that
paper.
> >> My recollection is that he was able to use the configuration
options
> >> of Grid-Stat to accomplish the types of processing he was after.
But
> >> I don't recall all the details.
> >>
> >> If you explain exactly what you're trying to do, it may be
possible
> >> to configure Grid-Stat to accomplish that.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >> >
> >> > Hi John,
> >> >
> >> > Your explanation and suggestions are very helpful and do
explain my
> >> > problems quite substantially. I will have to give it a try to
see
> >> > what happens.
> >> >
> >> > I do have a couple of other minor questions dealing with how
the
> >> > neighborhood verification is performed:
> >> > 1.) I noticed in the documentation that there is a flag for
> >> > defining the shape of the neighborhood area (i.e. circular or
> >> > square). What is the default if you don't specify?
> >> > 2.) In the literature, there are a couple of different ways to
> >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> >> > Sobash 2017;
> >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
> >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> >> > output is to define a neighborhood in which MET calculates a
> >> > fractional coverage of the threshold event? I have previously
> >> > explored computing the neighborhood maximum value within a
radius of
> each grid point before calculating a probability.
> >> >
> >> > Thanks,
> >> > Chris
> >> >
> >> >
> >> > //SIGNED//
> >> >
> >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, February 7, 2018 3:49 PM
> >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> > christopher.melick at us.af.mil>
> >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> >> matthew.dawson.8 at us.af.mil>;
> >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> > grid_stat on Neighborhood statistics
> >> >
> >> > Hi Chris,
> >> >
> >> > I see that you're having difficulty computing neighborhood
> >> > statistics in Grid-Stat.
> >> >
> >> > First, Grid-Stat definitely does have the ability to threshold
the
> >> StageIV
> >> > data on the fly. So pre-processing to 0's and 1's first is not
> needed.
> >> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> >> > cat_thresh = [ >=2.54 ];
> >> >
> >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> >> inches.
> >> >
> >> > If for some reason you have data that's already 0's and 1's,
you'd
> >> > just use a different threshold in the config file:
> >> > cat_thresh = [ ==1 ];
> >> >
> >> > This tells MET that anywhere you see a 1 in the data, the event
is
> >> > occurring. Otherwise, it's not.
> >> >
> >> > Now on the to real issue. Looking at your config file, I see
that
> >> you're
> >> > processing probabilistic data. When "prob = TRUE", MET only
> >> > computes
> >> the
> >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
> >> > you're getting no output in the NBRCNT line type, which is the
one
> >> > that
> >> contains
> >> > Fractions Skill Score (FSS).
> >> >
> >> > In the attached config file, I've made a few changes:
> >> > - Set the obtype to indicate that you're verifying against
StageIV
> >> > - Defined 2 fcst.field entries.
> >> > - In the first we process QP010 as a probability field.
Here
> >> > cat_thresh is set to ==0.10 which is shorthand notation for
> >> > defining
> >> bins
> >> > for probabilistic verification from 0 to 1 of width 0.1.
> >> > - In the second we process it as a field of scalars.
Here
> >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> >> > I'm
> >> using
> >> > for defining events for fractions skill score.
> >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> >> data.
> >> > In both the verifying threshold is >=2.54. This assumes that
you
> >> > pass
> >> the
> >> > raw StageIV data as input.
> >> > - In the output_flag section, I turned off the line types that
> >> > don't apply.
> >> >
> >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> >> > grabbing record number 19 from the input file, it's not
> >> > recommended. Instead,
> >> it'd
> >> > be better to figure out how to set the level string so that MET
> >> > always grabs the same record, regardless if it's in spot number
19
> >> > or 20 or any other location in the GRIB file.
> >> >
> >> > Hope this helps clarify. What other questions do you have?
> >> >
> >> > Thanks,
> >> >
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> >> > < met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: Help with grid_stat on Neighborhood statistics
> >> > > Owner: Nobody
> >> > > Requestors: christopher.melick at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >> > > >
> >> > >
> >> > >
> >> > > Hi John,
> >> > >
> >> > > I am trying to run grid_stat in MET to calculate Neighborhood
> >> > > statistics for probability of precipitation from some
ensemble
> >> > > data with the observations
> >> > > coming from Stage IV analyses. Unfortunately, I am getting
no
> output
> >> > > (just
> >> > > headers) in many of the ASCII files from the program when I
run it.
> >> > Could
> >> > > you help me figure out what I am doing wrong? I have
provided
> >> > > the command line specifics I use as well as the output log
file
> >> > > that is
> >> > created when
> >> > > running that command. I also have attached the
configuration file.
> >> I
> >> > am
> >> > > having trouble using ftp to send the raw data from any of the
> clients
> >> > > available to us. I believe this restriction occurred
recently
> since
> >> Bob
> >> > > Craig had mentioned that he had been able to connect
externally
> >> > > in the past.
> >> > >
> >> > > I should mention that I have pre-processed the Stage IV
analyses
> >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> >> > > I was unclear as to whether MET was able to do that on the
fly
> >> > > when running
> >> > grid_stat.
> >> > >
> >> > > Thanks,
> >> > > Chris
> >> > >
> >> > > Python command:
> >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> >> > > 'precip/post-processed-obs.nc', \
> >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> >> > > Call(metcall)
> >> > >
> >> > > //SIGNED//
> >> > >
> >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> >> > > 294-3693
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >>
> >>
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 14 09:35:34 2018
Chris,
Just wanted to close the loop on this ticket. I'll try to get it into
the
next release due out next in March.
Thanks,
John
On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Yes, thanks for the explanations. That clears up any possible
ambiguity
> and confusion. I kind of thought that was the procedure.
>
> When do you think that the next MET would be released with the
enhancement
> that we discussed for the neighborhood statistics? Is it an easy or
> difficult update to the software?
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, February 8, 2018 3:24 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> It sounds like you understand how the fractional coverage field is
> computed.
>
> - Read the raw field... probability values between 0 and 1 in your
case.
> - Apply the categorical threshold to that data to generate a field
of 0's
> and 1's... >0.5 in your case.
> - For each grid point, replace the value at that grid point with the
> "fraction" of 1's in the neighborhood around that point... e.g. the
event
> occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
> point, so fractional coverage = 4/9.
> - Compare the forecast fractional coverage values to the observed
> fractional coverage values to compute FSS.
>
> Yes, I do think it would be a good idea to talk to Craig about how
he
> configured MET for his work.
>
> Thanks,
> John
>
>
> On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Getting back to your initial response, I was able to run grid_stat
> > with your changes in the attached configuration file to get FSS
output
> > (as well as the other statistics for the neighborhood type
results).
> >
> > However, I do have a question about the forecast categorical
threshold
> > (cat_thresh is set to >=0.5) used in determining the neighborhood
> > probabilities from the ensemble probabilities: What is the actual
> > formula that MET uses to create the final field to calculate FSS?
Is
> > it a strict neighborhood fractional coverage of ensemble
probabilities
> > above 0.5 in some set radius of influence? Or is it some other
> > operation or does the threshold represent something else? I know
in
> > Craig's paper, they were calling the last step a neighborhood
average
> > of the ensemble probabilities to calculate NEP (since the event
had
> > already been defined at the grid point in determining the ensemble
> probability [i.e. likelihood of exceeding
> > 0.1" of precipitation]). Maybe, I should understand how Craig
was using
> > MET for his paper.
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 7, 2018 7:14 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I was thinking about what changes would be needed to compute FSS
on input
> > probability data. We could add a “field” option to the “nbrhd”
section,
> > just like we have in the “interp” section. It would be set to
FCST, OBS,
> > or BOTH (default) and would control the logic as to which fields
should
> be
> > passed through the logic for deriving fractional coverage fields.
In
> your
> > case, you’d set it to OBS and we’d just use the raw input forecast
> > probability values.
> >
> > Would that logic make sense to you?
> >
> > Thanks
> > John
> >
> > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Chris,
> > >
> > > In fact, SPC's 40-km radius probabilities were the motivation
for
> > > adding the shape = circle option to MET.
> > >
> > > For your first question, no, there is not currently a Gaussian
> > > smoother option for the "interp" section in Grid-Stat.
Currently, the
> > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> > unweighted mean).
> > > There is a distance-weighted mean option for point data where
the
> > > weight is 1/distance squared. But that doesn't work for Grid-
Stat
> > > because we'd end up dividing by 0.
> > >
> > > I can see how adding a gaussian smoother would be useful.
> > >
> > > Here's what we've done in the past with this SPC 40-km
probabilities:
> > >
> > > (1) Evaluate that field as a probability field (i.e. set "prob =
> > > TRUE;" in the config file).
> > > (2) Use the "interp" section to define a smoothing operation on
the
> > > observations:
> > > interp = {
> > > field = OBS;
> > > vld_thresh = 1.0;
> > > shape = CIRCLE;
> > > type = [
> > > { method = MAX; width = 11; }
> > > ];
> > > }
> > > Or set width to some circle diameter in grid units that works
out
> > > to 40km in the real world.
> > >
> > > In this example, we post-process the observations to make them
> > > comparable the event for which the probability was defined.
> > >
> > > But I think I understand what you'd like to do... interpret as
the
> > > probability value at each grid point as if it was a fractional
> > > coverage value. So don't apply a threshold and neighborhood
size to
> > > the forecast data at all. Just pass it directly to the routine
that
> > computes FSS.
> > >
> > > Unfortunately no, I don't think there's a way of configuring
Grid-Stat
> > > to do that in met-6.1. If you think this logic would be
generally
> > > useful, we could consider adding a flag to the config file to
enable
> > that logic.
> > >
> > > Sorry I don't have better answers for you.
> > >
> > > John
> > >
> > >
> > >
> > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >>
> > >> Hi John,
> > >>
> > >> Your suggestion for accomplishing #2 sounds promising. I need
to
> think
> > >> the details through some. On a somewhat related note, is
there a
> > Gaussian
> > >> smoother flag for smoothing the probabilistic fields? I know
that
> was
> > >> mentioned in Craig's paper and I have pursued that option in
the past
> > >> (but just not using MET).
> > >>
> > >> I guess my basic question is can you apply a neighborhood width
that
> > >> just includes the grid point maybe because you already have
> > >> inherently a probabilistic field with spatial uncertainty?
Would that
> > be a width of 1
> > >> or does that also include the nearest grid point? A good
example of
> > this
> > >> would be a probabilistic forecast issued from the Storm
Prediction
> > >> Center where the probabilities implicitly assume a 40-km radius
of
> > >> influence (so there is no need to compute a fractional coverage
of the
> > event).
> > >>
> > >> Thanks again,
> > >> Chris
> > >>
> > >>
> > >> //SIGNED//
> > >>
> > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > >> christopher.melick at us.af.mil>
> > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> > >> WS/WXN <robert.craig.2 at us.af.mil>
> > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > >> grid_stat on Neighborhood statistics
> > >>
> > >> Chris,
> > >>
> > >> (1) The default shape is a square since that's what MET
previously
> did.
> > >> The circle is the *new* option.
> > >>
> > >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> > >> doing the following...
> > >> - For each field separately (i.e. fcst and obs), apply the
> > >> threshold
> > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > >> - Apply the neighborhood width and shape to compute a
fractional
> > >> coverage value for each grid point.
> > >> - Compare the fcst and obs fractional coverage fields to
compute
> FSS.
> > >>
> > >> But there are some other options that you may find useful. The
> "interp"
> > >> section in the config file is used in Grid-Stat as a smoothing
> > operation.
> > >> After reading the raw input data, you can specify if/how to
smooth
> > >> that data prior to computing stats. For example, since you
mentioned
> > >> maximums, you could replace the value at each grid point with
the
> > >> maximum of the values in a 5x5 box surrounding that point:
> > >>
> > >> interp = {
> > >> field = FCST;
> > >> vld_thresh = 1.0;
> > >> shape = SQUARE;
> > >>
> > >> type = [
> > >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> > >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> > >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> > >> closest points
> > >> ];
> > >> }
> > >>
> > >> Using the example above, you'll get 3 times the amount of
output from
> > MET.
> > >> The first smoother really is no smoothing at all. It would run
on
> > >> the raw input data. The second smoother replaces the value at
each
> > >> grid point with the maximum of the 25 surrounding points. The
third
> > >> smoother replaces the value at each grid point with the mean of
those
> > 25.
> > >>
> > >> I worked with Craig Schwartz when he was running MET for that
paper.
> > >> My recollection is that he was able to use the configuration
options
> > >> of Grid-Stat to accomplish the types of processing he was
after. But
> > >> I don't recall all the details.
> > >>
> > >> If you explain exactly what you're trying to do, it may be
possible
> > >> to configure Grid-Stat to accomplish that.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil
via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > >> >
> > >> > Hi John,
> > >> >
> > >> > Your explanation and suggestions are very helpful and do
explain my
> > >> > problems quite substantially. I will have to give it a try
to see
> > >> > what happens.
> > >> >
> > >> > I do have a couple of other minor questions dealing with how
the
> > >> > neighborhood verification is performed:
> > >> > 1.) I noticed in the documentation that there is a flag for
> > >> > defining the shape of the neighborhood area (i.e. circular or
> > >> > square). What is the default if you don't specify?
> > >> > 2.) In the literature, there are a couple of different ways
to
> > >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> > >> > Sobash 2017;
> > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1). I
> > >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> > >> > output is to define a neighborhood in which MET calculates a
> > >> > fractional coverage of the threshold event? I have
previously
> > >> > explored computing the neighborhood maximum value within a
radius of
> > each grid point before calculating a probability.
> > >> >
> > >> > Thanks,
> > >> > Chris
> > >> >
> > >> >
> > >> > //SIGNED//
> > >> >
> > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >> >
> > >> >
> > >> >
> > >> > -----Original Message-----
> > >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > >> > christopher.melick at us.af.mil>
> > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > >> matthew.dawson.8 at us.af.mil>;
> > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > >> > grid_stat on Neighborhood statistics
> > >> >
> > >> > Hi Chris,
> > >> >
> > >> > I see that you're having difficulty computing neighborhood
> > >> > statistics in Grid-Stat.
> > >> >
> > >> > First, Grid-Stat definitely does have the ability to
threshold the
> > >> StageIV
> > >> > data on the fly. So pre-processing to 0's and 1's first is
not
> > needed.
> > >> > Instead, in the "obs" dictionary of the config file, you'd
just set:
> > >> > cat_thresh = [ >=2.54 ];
> > >> >
> > >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> > >> inches.
> > >> >
> > >> > If for some reason you have data that's already 0's and 1's,
you'd
> > >> > just use a different threshold in the config file:
> > >> > cat_thresh = [ ==1 ];
> > >> >
> > >> > This tells MET that anywhere you see a 1 in the data, the
event is
> > >> > occurring. Otherwise, it's not.
> > >> >
> > >> > Now on the to real issue. Looking at your config file, I see
that
> > >> you're
> > >> > processing probabilistic data. When "prob = TRUE", MET only
> > >> > computes
> > >> the
> > >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's
why
> > >> > you're getting no output in the NBRCNT line type, which is
the one
> > >> > that
> > >> contains
> > >> > Fractions Skill Score (FSS).
> > >> >
> > >> > In the attached config file, I've made a few changes:
> > >> > - Set the obtype to indicate that you're verifying against
StageIV
> > >> > - Defined 2 fcst.field entries.
> > >> > - In the first we process QP010 as a probability field.
Here
> > >> > cat_thresh is set to ==0.10 which is shorthand notation for
> > >> > defining
> > >> bins
> > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > >> > - In the second we process it as a field of scalars.
Here
> > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > >> > I'm
> > >> using
> > >> > for defining events for fractions skill score.
> > >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> > >> data.
> > >> > In both the verifying threshold is >=2.54. This assumes that
you
> > >> > pass
> > >> the
> > >> > raw StageIV data as input.
> > >> > - In the output_flag section, I turned off the line types
that
> > >> > don't apply.
> > >> >
> > >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> > >> > grabbing record number 19 from the input file, it's not
> > >> > recommended. Instead,
> > >> it'd
> > >> > be better to figure out how to set the level string so that
MET
> > >> > always grabs the same record, regardless if it's in spot
number 19
> > >> > or 20 or any other location in the GRIB file.
> > >> >
> > >> > Hope this helps clarify. What other questions do you have?
> > >> >
> > >> > Thanks,
> > >> >
> > >> > John Halley Gotway
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> > >> > < met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> > >> > > Queue: met_help
> > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > >> > > Owner: Nobody
> > >> > > Requestors: christopher.melick at us.af.mil
> > >> > > Status: new
> > >> > > Ticket <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >> > > >
> > >> > >
> > >> > >
> > >> > > Hi John,
> > >> > >
> > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > >> > > statistics for probability of precipitation from some
ensemble
> > >> > > data with the observations
> > >> > > coming from Stage IV analyses. Unfortunately, I am
getting no
> > output
> > >> > > (just
> > >> > > headers) in many of the ASCII files from the program when I
run
> it.
> > >> > Could
> > >> > > you help me figure out what I am doing wrong? I have
provided
> > >> > > the command line specifics I use as well as the output log
file
> > >> > > that is
> > >> > created when
> > >> > > running that command. I also have attached the
configuration
> file.
> > >> I
> > >> > am
> > >> > > having trouble using ftp to send the raw data from any of
the
> > clients
> > >> > > available to us. I believe this restriction occurred
recently
> > since
> > >> Bob
> > >> > > Craig had mentioned that he had been able to connect
externally
> > >> > > in the past.
> > >> > >
> > >> > > I should mention that I have pre-processed the Stage IV
analyses
> > >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> > >> > > I was unclear as to whether MET was able to do that on the
fly
> > >> > > when running
> > >> > grid_stat.
> > >> > >
> > >> > > Thanks,
> > >> > > Chris
> > >> > >
> > >> > > Python command:
> > >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > >> > > 'precip/post-processed-obs.nc', \
> > >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> > >> > > Call(metcall)
> > >> > >
> > >> > > //SIGNED//
> > >> > >
> > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> > >> > > 294-3693
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 14 09:39:21 2018
John,
That sounds like an excellent plan. Thanks for your assistance and
trying to accommodate and make changes to the MET software so quickly.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 14, 2018 10:36 AM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
Just wanted to close the loop on this ticket. I'll try to get it into
the next release due out next in March.
Thanks,
John
On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Yes, thanks for the explanations. That clears up any possible
> ambiguity and confusion. I kind of thought that was the procedure.
>
> When do you think that the next MET would be released with the
> enhancement that we discussed for the neighborhood statistics? Is
it
> an easy or difficult update to the software?
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, February 8, 2018 3:24 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> It sounds like you understand how the fractional coverage field is
> computed.
>
> - Read the raw field... probability values between 0 and 1 in your
case.
> - Apply the categorical threshold to that data to generate a field
of 0's
> and 1's... >0.5 in your case.
> - For each grid point, replace the value at that grid point with the
> "fraction" of 1's in the neighborhood around that point... e.g. the
event
> occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
> point, so fractional coverage = 4/9.
> - Compare the forecast fractional coverage values to the observed
> fractional coverage values to compute FSS.
>
> Yes, I do think it would be a good idea to talk to Craig about how
he
> configured MET for his work.
>
> Thanks,
> John
>
>
> On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Getting back to your initial response, I was able to run grid_stat
> > with your changes in the attached configuration file to get FSS
output
> > (as well as the other statistics for the neighborhood type
results).
> >
> > However, I do have a question about the forecast categorical
threshold
> > (cat_thresh is set to >=0.5) used in determining the neighborhood
> > probabilities from the ensemble probabilities: What is the actual
> > formula that MET uses to create the final field to calculate FSS?
Is
> > it a strict neighborhood fractional coverage of ensemble
probabilities
> > above 0.5 in some set radius of influence? Or is it some other
> > operation or does the threshold represent something else? I know
in
> > Craig's paper, they were calling the last step a neighborhood
average
> > of the ensemble probabilities to calculate NEP (since the event
had
> > already been defined at the grid point in determining the ensemble
> probability [i.e. likelihood of exceeding
> > 0.1" of precipitation]). Maybe, I should understand how Craig
was using
> > MET for his paper.
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 7, 2018 7:14 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I was thinking about what changes would be needed to compute FSS
on input
> > probability data. We could add a “field” option to the “nbrhd”
section,
> > just like we have in the “interp” section. It would be set to
FCST, OBS,
> > or BOTH (default) and would control the logic as to which fields
should
> be
> > passed through the logic for deriving fractional coverage fields.
In
> your
> > case, you’d set it to OBS and we’d just use the raw input forecast
> > probability values.
> >
> > Would that logic make sense to you?
> >
> > Thanks
> > John
> >
> > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Chris,
> > >
> > > In fact, SPC's 40-km radius probabilities were the motivation
for
> > > adding the shape = circle option to MET.
> > >
> > > For your first question, no, there is not currently a Gaussian
> > > smoother option for the "interp" section in Grid-Stat.
Currently, the
> > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> > unweighted mean).
> > > There is a distance-weighted mean option for point data where
the
> > > weight is 1/distance squared. But that doesn't work for Grid-
Stat
> > > because we'd end up dividing by 0.
> > >
> > > I can see how adding a gaussian smoother would be useful.
> > >
> > > Here's what we've done in the past with this SPC 40-km
probabilities:
> > >
> > > (1) Evaluate that field as a probability field (i.e. set "prob =
> > > TRUE;" in the config file).
> > > (2) Use the "interp" section to define a smoothing operation on
the
> > > observations:
> > > interp = {
> > > field = OBS;
> > > vld_thresh = 1.0;
> > > shape = CIRCLE;
> > > type = [
> > > { method = MAX; width = 11; }
> > > ];
> > > }
> > > Or set width to some circle diameter in grid units that works
out
> > > to 40km in the real world.
> > >
> > > In this example, we post-process the observations to make them
> > > comparable the event for which the probability was defined.
> > >
> > > But I think I understand what you'd like to do... interpret as
the
> > > probability value at each grid point as if it was a fractional
> > > coverage value. So don't apply a threshold and neighborhood
size to
> > > the forecast data at all. Just pass it directly to the routine
that
> > computes FSS.
> > >
> > > Unfortunately no, I don't think there's a way of configuring
Grid-Stat
> > > to do that in met-6.1. If you think this logic would be
generally
> > > useful, we could consider adding a flag to the config file to
enable
> > that logic.
> > >
> > > Sorry I don't have better answers for you.
> > >
> > > John
> > >
> > >
> > >
> > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >>
> > >> Hi John,
> > >>
> > >> Your suggestion for accomplishing #2 sounds promising. I need
to
> think
> > >> the details through some. On a somewhat related note, is
there a
> > Gaussian
> > >> smoother flag for smoothing the probabilistic fields? I know
that
> was
> > >> mentioned in Craig's paper and I have pursued that option in
the past
> > >> (but just not using MET).
> > >>
> > >> I guess my basic question is can you apply a neighborhood width
that
> > >> just includes the grid point maybe because you already have
> > >> inherently a probabilistic field with spatial uncertainty?
Would that
> > be a width of 1
> > >> or does that also include the nearest grid point? A good
example of
> > this
> > >> would be a probabilistic forecast issued from the Storm
Prediction
> > >> Center where the probabilities implicitly assume a 40-km radius
of
> > >> influence (so there is no need to compute a fractional coverage
of the
> > event).
> > >>
> > >> Thanks again,
> > >> Chris
> > >>
> > >>
> > >> //SIGNED//
> > >>
> > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > >> christopher.melick at us.af.mil>
> > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> > >> WS/WXN <robert.craig.2 at us.af.mil>
> > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > >> grid_stat on Neighborhood statistics
> > >>
> > >> Chris,
> > >>
> > >> (1) The default shape is a square since that's what MET
previously
> did.
> > >> The circle is the *new* option.
> > >>
> > >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> > >> doing the following...
> > >> - For each field separately (i.e. fcst and obs), apply the
> > >> threshold
> > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > >> - Apply the neighborhood width and shape to compute a
fractional
> > >> coverage value for each grid point.
> > >> - Compare the fcst and obs fractional coverage fields to
compute
> FSS.
> > >>
> > >> But there are some other options that you may find useful. The
> "interp"
> > >> section in the config file is used in Grid-Stat as a smoothing
> > operation.
> > >> After reading the raw input data, you can specify if/how to
smooth
> > >> that data prior to computing stats. For example, since you
mentioned
> > >> maximums, you could replace the value at each grid point with
the
> > >> maximum of the values in a 5x5 box surrounding that point:
> > >>
> > >> interp = {
> > >> field = FCST;
> > >> vld_thresh = 1.0;
> > >> shape = SQUARE;
> > >>
> > >> type = [
> > >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> > >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> > >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> > >> closest points
> > >> ];
> > >> }
> > >>
> > >> Using the example above, you'll get 3 times the amount of
output from
> > MET.
> > >> The first smoother really is no smoothing at all. It would run
on
> > >> the raw input data. The second smoother replaces the value at
each
> > >> grid point with the maximum of the 25 surrounding points. The
third
> > >> smoother replaces the value at each grid point with the mean of
those
> > 25.
> > >>
> > >> I worked with Craig Schwartz when he was running MET for that
paper.
> > >> My recollection is that he was able to use the configuration
options
> > >> of Grid-Stat to accomplish the types of processing he was
after. But
> > >> I don't recall all the details.
> > >>
> > >> If you explain exactly what you're trying to do, it may be
possible
> > >> to configure Grid-Stat to accomplish that.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil
via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > >> >
> > >> > Hi John,
> > >> >
> > >> > Your explanation and suggestions are very helpful and do
explain my
> > >> > problems quite substantially. I will have to give it a try
to see
> > >> > what happens.
> > >> >
> > >> > I do have a couple of other minor questions dealing with how
the
> > >> > neighborhood verification is performed:
> > >> > 1.) I noticed in the documentation that there is a flag for
> > >> > defining the shape of the neighborhood area (i.e. circular or
> > >> > square). What is the default if you don't specify?
> > >> > 2.) In the literature, there are a couple of different ways
to
> > >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> > >> > Sobash 2017;
> > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1). I
> > >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> > >> > output is to define a neighborhood in which MET calculates a
> > >> > fractional coverage of the threshold event? I have
previously
> > >> > explored computing the neighborhood maximum value within a
radius of
> > each grid point before calculating a probability.
> > >> >
> > >> > Thanks,
> > >> > Chris
> > >> >
> > >> >
> > >> > //SIGNED//
> > >> >
> > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >> >
> > >> >
> > >> >
> > >> > -----Original Message-----
> > >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > >> > christopher.melick at us.af.mil>
> > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > >> matthew.dawson.8 at us.af.mil>;
> > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > >> > grid_stat on Neighborhood statistics
> > >> >
> > >> > Hi Chris,
> > >> >
> > >> > I see that you're having difficulty computing neighborhood
> > >> > statistics in Grid-Stat.
> > >> >
> > >> > First, Grid-Stat definitely does have the ability to
threshold the
> > >> StageIV
> > >> > data on the fly. So pre-processing to 0's and 1's first is
not
> > needed.
> > >> > Instead, in the "obs" dictionary of the config file, you'd
just set:
> > >> > cat_thresh = [ >=2.54 ];
> > >> >
> > >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> > >> inches.
> > >> >
> > >> > If for some reason you have data that's already 0's and 1's,
you'd
> > >> > just use a different threshold in the config file:
> > >> > cat_thresh = [ ==1 ];
> > >> >
> > >> > This tells MET that anywhere you see a 1 in the data, the
event is
> > >> > occurring. Otherwise, it's not.
> > >> >
> > >> > Now on the to real issue. Looking at your config file, I see
that
> > >> you're
> > >> > processing probabilistic data. When "prob = TRUE", MET only
> > >> > computes
> > >> the
> > >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's
why
> > >> > you're getting no output in the NBRCNT line type, which is
the one
> > >> > that
> > >> contains
> > >> > Fractions Skill Score (FSS).
> > >> >
> > >> > In the attached config file, I've made a few changes:
> > >> > - Set the obtype to indicate that you're verifying against
StageIV
> > >> > - Defined 2 fcst.field entries.
> > >> > - In the first we process QP010 as a probability field.
Here
> > >> > cat_thresh is set to ==0.10 which is shorthand notation for
> > >> > defining
> > >> bins
> > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > >> > - In the second we process it as a field of scalars.
Here
> > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > >> > I'm
> > >> using
> > >> > for defining events for fractions skill score.
> > >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> > >> data.
> > >> > In both the verifying threshold is >=2.54. This assumes that
you
> > >> > pass
> > >> the
> > >> > raw StageIV data as input.
> > >> > - In the output_flag section, I turned off the line types
that
> > >> > don't apply.
> > >> >
> > >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> > >> > grabbing record number 19 from the input file, it's not
> > >> > recommended. Instead,
> > >> it'd
> > >> > be better to figure out how to set the level string so that
MET
> > >> > always grabs the same record, regardless if it's in spot
number 19
> > >> > or 20 or any other location in the GRIB file.
> > >> >
> > >> > Hope this helps clarify. What other questions do you have?
> > >> >
> > >> > Thanks,
> > >> >
> > >> > John Halley Gotway
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> > >> > < met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> > >> > > Queue: met_help
> > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > >> > > Owner: Nobody
> > >> > > Requestors: christopher.melick at us.af.mil
> > >> > > Status: new
> > >> > > Ticket <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >> > > >
> > >> > >
> > >> > >
> > >> > > Hi John,
> > >> > >
> > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > >> > > statistics for probability of precipitation from some
ensemble
> > >> > > data with the observations
> > >> > > coming from Stage IV analyses. Unfortunately, I am
getting no
> > output
> > >> > > (just
> > >> > > headers) in many of the ASCII files from the program when I
run
> it.
> > >> > Could
> > >> > > you help me figure out what I am doing wrong? I have
provided
> > >> > > the command line specifics I use as well as the output log
file
> > >> > > that is
> > >> > created when
> > >> > > running that command. I also have attached the
configuration
> file.
> > >> I
> > >> > am
> > >> > > having trouble using ftp to send the raw data from any of
the
> > clients
> > >> > > available to us. I believe this restriction occurred
recently
> > since
> > >> Bob
> > >> > > Craig had mentioned that he had been able to connect
externally
> > >> > > in the past.
> > >> > >
> > >> > > I should mention that I have pre-processed the Stage IV
analyses
> > >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> > >> > > I was unclear as to whether MET was able to do that on the
fly
> > >> > > when running
> > >> > grid_stat.
> > >> > >
> > >> > > Thanks,
> > >> > > Chris
> > >> > >
> > >> > > Python command:
> > >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > >> > > 'precip/post-processed-obs.nc', \
> > >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> > >> > > Call(metcall)
> > >> > >
> > >> > > //SIGNED//
> > >> > >
> > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> > >> > > 294-3693
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 14 14:19:54 2018
Hi Chris,
I'm working on the interp.field option we discussed.
I have a quick question for you. The NBRCNT line type that is
computed by
grid_stat contains columns for F_RATE and O_RATE. That is the ratio
of
grid points over a masking region at which the "event" is occurring in
the
forecast and observation fields respectively.
When grid_stat computes the fractional coverage field, we can compute
that. But if grid_stat does *NOT* compute the fractional coverage
fields,
then we don't know what those ratio's are.
- If interp.field = FCST, then O_RATE = NA.
- If interp.field = OBS, then F_RATE = NA.
- If interp.field = NONE, then F_RATE = O_RATE = NA.
Make sense?
Thanks,
John
On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> That sounds like an excellent plan. Thanks for your assistance and
trying
> to accommodate and make changes to the MET software so quickly.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 14, 2018 10:36 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Just wanted to close the loop on this ticket. I'll try to get it
into the
> next release due out next in March.
>
> Thanks,
> John
>
> On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Yes, thanks for the explanations. That clears up any possible
> > ambiguity and confusion. I kind of thought that was the
procedure.
> >
> > When do you think that the next MET would be released with the
> > enhancement that we discussed for the neighborhood statistics? Is
it
> > an easy or difficult update to the software?
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, February 8, 2018 3:24 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > It sounds like you understand how the fractional coverage field is
> > computed.
> >
> > - Read the raw field... probability values between 0 and 1 in your
case.
> > - Apply the categorical threshold to that data to generate a field
of 0's
> > and 1's... >0.5 in your case.
> > - For each grid point, replace the value at that grid point with
the
> > "fraction" of 1's in the neighborhood around that point... e.g.
the event
> > occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
> > point, so fractional coverage = 4/9.
> > - Compare the forecast fractional coverage values to the observed
> > fractional coverage values to compute FSS.
> >
> > Yes, I do think it would be a good idea to talk to Craig about how
he
> > configured MET for his work.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Getting back to your initial response, I was able to run
grid_stat
> > > with your changes in the attached configuration file to get FSS
output
> > > (as well as the other statistics for the neighborhood type
results).
> > >
> > > However, I do have a question about the forecast categorical
threshold
> > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > probabilities from the ensemble probabilities: What is the
actual
> > > formula that MET uses to create the final field to calculate
FSS? Is
> > > it a strict neighborhood fractional coverage of ensemble
probabilities
> > > above 0.5 in some set radius of influence? Or is it some other
> > > operation or does the threshold represent something else? I
know in
> > > Craig's paper, they were calling the last step a neighborhood
average
> > > of the ensemble probabilities to calculate NEP (since the event
had
> > > already been defined at the grid point in determining the
ensemble
> > probability [i.e. likelihood of exceeding
> > > 0.1" of precipitation]). Maybe, I should understand how Craig
was
> using
> > > MET for his paper.
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > I was thinking about what changes would be needed to compute FSS
on
> input
> > > probability data. We could add a “field” option to the “nbrhd”
> section,
> > > just like we have in the “interp” section. It would be set to
FCST,
> OBS,
> > > or BOTH (default) and would control the logic as to which fields
should
> > be
> > > passed through the logic for deriving fractional coverage
fields. In
> > your
> > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > probability values.
> > >
> > > Would that logic make sense to you?
> > >
> > > Thanks
> > > John
> > >
> > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Chris,
> > > >
> > > > In fact, SPC's 40-km radius probabilities were the motivation
for
> > > > adding the shape = circle option to MET.
> > > >
> > > > For your first question, no, there is not currently a Gaussian
> > > > smoother option for the "interp" section in Grid-Stat.
Currently,
> the
> > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > unweighted mean).
> > > > There is a distance-weighted mean option for point data where
the
> > > > weight is 1/distance squared. But that doesn't work for Grid-
Stat
> > > > because we'd end up dividing by 0.
> > > >
> > > > I can see how adding a gaussian smoother would be useful.
> > > >
> > > > Here's what we've done in the past with this SPC 40-km
probabilities:
> > > >
> > > > (1) Evaluate that field as a probability field (i.e. set "prob
=
> > > > TRUE;" in the config file).
> > > > (2) Use the "interp" section to define a smoothing operation
on the
> > > > observations:
> > > > interp = {
> > > > field = OBS;
> > > > vld_thresh = 1.0;
> > > > shape = CIRCLE;
> > > > type = [
> > > > { method = MAX; width = 11; }
> > > > ];
> > > > }
> > > > Or set width to some circle diameter in grid units that
works out
> > > > to 40km in the real world.
> > > >
> > > > In this example, we post-process the observations to make them
> > > > comparable the event for which the probability was defined.
> > > >
> > > > But I think I understand what you'd like to do... interpret as
the
> > > > probability value at each grid point as if it was a fractional
> > > > coverage value. So don't apply a threshold and neighborhood
size to
> > > > the forecast data at all. Just pass it directly to the
routine that
> > > computes FSS.
> > > >
> > > > Unfortunately no, I don't think there's a way of configuring
> Grid-Stat
> > > > to do that in met-6.1. If you think this logic would be
generally
> > > > useful, we could consider adding a flag to the config file to
enable
> > > that logic.
> > > >
> > > > Sorry I don't have better answers for you.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >>
> > > >> Hi John,
> > > >>
> > > >> Your suggestion for accomplishing #2 sounds promising. I
need to
> > think
> > > >> the details through some. On a somewhat related note, is
there a
> > > Gaussian
> > > >> smoother flag for smoothing the probabilistic fields? I
know that
> > was
> > > >> mentioned in Craig's paper and I have pursued that option in
the
> past
> > > >> (but just not using MET).
> > > >>
> > > >> I guess my basic question is can you apply a neighborhood
width that
> > > >> just includes the grid point maybe because you already have
> > > >> inherently a probabilistic field with spatial uncertainty?
Would
> that
> > > be a width of 1
> > > >> or does that also include the nearest grid point? A good
example
> of
> > > this
> > > >> would be a probabilistic forecast issued from the Storm
Prediction
> > > >> Center where the probabilities implicitly assume a 40-km
radius of
> > > >> influence (so there is no need to compute a fractional
coverage of
> the
> > > event).
> > > >>
> > > >> Thanks again,
> > > >> Chris
> > > >>
> > > >>
> > > >> //SIGNED//
> > > >>
> > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> christopher.melick at us.af.mil>
> > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC
16
> > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > >> grid_stat on Neighborhood statistics
> > > >>
> > > >> Chris,
> > > >>
> > > >> (1) The default shape is a square since that's what MET
previously
> > did.
> > > >> The circle is the *new* option.
> > > >>
> > > >> (2) For this one, the short answer is yes. FSS is computed
in MET
> by
> > > >> doing the following...
> > > >> - For each field separately (i.e. fcst and obs), apply the
> > > >> threshold
> > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > >> - Apply the neighborhood width and shape to compute a
fractional
> > > >> coverage value for each grid point.
> > > >> - Compare the fcst and obs fractional coverage fields to
compute
> > FSS.
> > > >>
> > > >> But there are some other options that you may find useful.
The
> > "interp"
> > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > operation.
> > > >> After reading the raw input data, you can specify if/how to
smooth
> > > >> that data prior to computing stats. For example, since you
> mentioned
> > > >> maximums, you could replace the value at each grid point with
the
> > > >> maximum of the values in a 5x5 box surrounding that point:
> > > >>
> > > >> interp = {
> > > >> field = FCST;
> > > >> vld_thresh = 1.0;
> > > >> shape = SQUARE;
> > > >>
> > > >> type = [
> > > >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> points
> > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of the
25
> > > >> closest points
> > > >> ];
> > > >> }
> > > >>
> > > >> Using the example above, you'll get 3 times the amount of
output
> from
> > > MET.
> > > >> The first smoother really is no smoothing at all. It would
run on
> > > >> the raw input data. The second smoother replaces the value
at each
> > > >> grid point with the maximum of the 25 surrounding points.
The third
> > > >> smoother replaces the value at each grid point with the mean
of
> those
> > > 25.
> > > >>
> > > >> I worked with Craig Schwartz when he was running MET for that
paper.
> > > >> My recollection is that he was able to use the configuration
options
> > > >> of Grid-Stat to accomplish the types of processing he was
after.
> But
> > > >> I don't recall all the details.
> > > >>
> > > >> If you explain exactly what you're trying to do, it may be
possible
> > > >> to configure Grid-Stat to accomplish that.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil
via
> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > >> >
> > > >> > Hi John,
> > > >> >
> > > >> > Your explanation and suggestions are very helpful and do
explain
> my
> > > >> > problems quite substantially. I will have to give it a try
to see
> > > >> > what happens.
> > > >> >
> > > >> > I do have a couple of other minor questions dealing with
how the
> > > >> > neighborhood verification is performed:
> > > >> > 1.) I noticed in the documentation that there is a flag
for
> > > >> > defining the shape of the neighborhood area (i.e. circular
or
> > > >> > square). What is the default if you don't specify?
> > > >> > 2.) In the literature, there are a couple of different
ways to
> > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz and
> > > >> > Sobash 2017;
> > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1).
> I
> > > >> > was curious is the only way to get the Fractions Skill
Score (FSS)
> > > >> > output is to define a neighborhood in which MET calculates
a
> > > >> > fractional coverage of the threshold event? I have
previously
> > > >> > explored computing the neighborhood maximum value within a
radius
> of
> > > each grid point before calculating a probability.
> > > >> >
> > > >> > Thanks,
> > > >> > Chris
> > > >> >
> > > >> >
> > > >> > //SIGNED//
> > > >> >
> > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> > christopher.melick at us.af.mil>
> > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > >> matthew.dawson.8 at us.af.mil>;
> > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil>
> > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > >> > grid_stat on Neighborhood statistics
> > > >> >
> > > >> > Hi Chris,
> > > >> >
> > > >> > I see that you're having difficulty computing neighborhood
> > > >> > statistics in Grid-Stat.
> > > >> >
> > > >> > First, Grid-Stat definitely does have the ability to
threshold the
> > > >> StageIV
> > > >> > data on the fly. So pre-processing to 0's and 1's first is
not
> > > needed.
> > > >> > Instead, in the "obs" dictionary of the config file, you'd
just
> set:
> > > >> > cat_thresh = [ >=2.54 ];
> > > >> >
> > > >> > Since StageIV precip is millimeters, >=2.54 mm is the same
as
> >=0.1
> > > >> inches.
> > > >> >
> > > >> > If for some reason you have data that's already 0's and
1's, you'd
> > > >> > just use a different threshold in the config file:
> > > >> > cat_thresh = [ ==1 ];
> > > >> >
> > > >> > This tells MET that anywhere you see a 1 in the data, the
event is
> > > >> > occurring. Otherwise, it's not.
> > > >> >
> > > >> > Now on the to real issue. Looking at your config file, I
see that
> > > >> you're
> > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > >> > computes
> > > >> the
> > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's
why
> > > >> > you're getting no output in the NBRCNT line type, which is
the one
> > > >> > that
> > > >> contains
> > > >> > Fractions Skill Score (FSS).
> > > >> >
> > > >> > In the attached config file, I've made a few changes:
> > > >> > - Set the obtype to indicate that you're verifying against
> StageIV
> > > >> > - Defined 2 fcst.field entries.
> > > >> > - In the first we process QP010 as a probability
field.
> Here
> > > >> > cat_thresh is set to ==0.10 which is shorthand notation for
> > > >> > defining
> > > >> bins
> > > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > > >> > - In the second we process it as a field of scalars.
Here
> > > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > > >> > I'm
> > > >> using
> > > >> > for defining events for fractions skill score.
> > > >> > - Defined 2 corresponding obs.field entries to verify the
> forecast
> > > >> data.
> > > >> > In both the verifying threshold is >=2.54. This assumes
that you
> > > >> > pass
> > > >> the
> > > >> > raw StageIV data as input.
> > > >> > - In the output_flag section, I turned off the line types
that
> > > >> > don't apply.
> > > >> >
> > > >> > Lastly, while setting "level = [ "R19" ];" works as a way
of
> > > >> > grabbing record number 19 from the input file, it's not
> > > >> > recommended. Instead,
> > > >> it'd
> > > >> > be better to figure out how to set the level string so that
MET
> > > >> > always grabs the same record, regardless if it's in spot
number 19
> > > >> > or 20 or any other location in the GRIB file.
> > > >> >
> > > >> > Hope this helps clarify. What other questions do you have?
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > John Halley Gotway
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil via
> RT
> > > >> > < met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > >> > > Queue: met_help
> > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > >> > > Owner: Nobody
> > > >> > > Requestors: christopher.melick at us.af.mil
> > > >> > > Status: new
> > > >> > > Ticket <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > Hi John,
> > > >> > >
> > > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > > >> > > statistics for probability of precipitation from some
ensemble
> > > >> > > data with the observations
> > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting no
> > > output
> > > >> > > (just
> > > >> > > headers) in many of the ASCII files from the program when
I run
> > it.
> > > >> > Could
> > > >> > > you help me figure out what I am doing wrong? I have
provided
> > > >> > > the command line specifics I use as well as the output
log file
> > > >> > > that is
> > > >> > created when
> > > >> > > running that command. I also have attached the
configuration
> > file.
> > > >> I
> > > >> > am
> > > >> > > having trouble using ftp to send the raw data from any of
the
> > > clients
> > > >> > > available to us. I believe this restriction occurred
recently
> > > since
> > > >> Bob
> > > >> > > Craig had mentioned that he had been able to connect
externally
> > > >> > > in the past.
> > > >> > >
> > > >> > > I should mention that I have pre-processed the Stage IV
analyses
> > > >> > > already to be either 0's or 1's for the threshold
(>=0.1") since
> > > >> > > I was unclear as to whether MET was able to do that on
the fly
> > > >> > > when running
> > > >> > grid_stat.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Chris
> > > >> > >
> > > >> > > Python command:
> > > >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > > >> > > 'precip/post-processed-obs.nc', \
> > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-
v', '5]
> > > >> > > Call(metcall)
> > > >> > >
> > > >> > > //SIGNED//
> > > >> > >
> > > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > >> > > Verification Exploitation Team 16th Weather Squadron (16
> WS/WXEV)
> > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > >> > > 294-3693
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 14 14:48:14 2018
Hi John,
Yes, that does make perfect sense.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 14, 2018 3:20 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I'm working on the interp.field option we discussed.
I have a quick question for you. The NBRCNT line type that is
computed by
grid_stat contains columns for F_RATE and O_RATE. That is the ratio
of grid points over a masking region at which the "event" is occurring
in the forecast and observation fields respectively.
When grid_stat computes the fractional coverage field, we can compute
that. But if grid_stat does *NOT* compute the fractional coverage
fields, then we don't know what those ratio's are.
- If interp.field = FCST, then O_RATE = NA.
- If interp.field = OBS, then F_RATE = NA.
- If interp.field = NONE, then F_RATE = O_RATE = NA.
Make sense?
Thanks,
John
On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> That sounds like an excellent plan. Thanks for your assistance and
> trying to accommodate and make changes to the MET software so
quickly.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 14, 2018 10:36 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Just wanted to close the loop on this ticket. I'll try to get it
into the
> next release due out next in March.
>
> Thanks,
> John
>
> On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Yes, thanks for the explanations. That clears up any possible
> > ambiguity and confusion. I kind of thought that was the
procedure.
> >
> > When do you think that the next MET would be released with the
> > enhancement that we discussed for the neighborhood statistics? Is
it
> > an easy or difficult update to the software?
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, February 8, 2018 3:24 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > It sounds like you understand how the fractional coverage field is
> > computed.
> >
> > - Read the raw field... probability values between 0 and 1 in your
case.
> > - Apply the categorical threshold to that data to generate a field
of 0's
> > and 1's... >0.5 in your case.
> > - For each grid point, replace the value at that grid point with
the
> > "fraction" of 1's in the neighborhood around that point... e.g.
the event
> > occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
> > point, so fractional coverage = 4/9.
> > - Compare the forecast fractional coverage values to the observed
> > fractional coverage values to compute FSS.
> >
> > Yes, I do think it would be a good idea to talk to Craig about how
he
> > configured MET for his work.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Getting back to your initial response, I was able to run
grid_stat
> > > with your changes in the attached configuration file to get FSS
output
> > > (as well as the other statistics for the neighborhood type
results).
> > >
> > > However, I do have a question about the forecast categorical
threshold
> > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > probabilities from the ensemble probabilities: What is the
actual
> > > formula that MET uses to create the final field to calculate
FSS? Is
> > > it a strict neighborhood fractional coverage of ensemble
probabilities
> > > above 0.5 in some set radius of influence? Or is it some other
> > > operation or does the threshold represent something else? I
know in
> > > Craig's paper, they were calling the last step a neighborhood
average
> > > of the ensemble probabilities to calculate NEP (since the event
had
> > > already been defined at the grid point in determining the
ensemble
> > probability [i.e. likelihood of exceeding
> > > 0.1" of precipitation]). Maybe, I should understand how Craig
was
> using
> > > MET for his paper.
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > I was thinking about what changes would be needed to compute FSS
on
> input
> > > probability data. We could add a “field” option to the “nbrhd”
> section,
> > > just like we have in the “interp” section. It would be set to
FCST,
> OBS,
> > > or BOTH (default) and would control the logic as to which fields
should
> > be
> > > passed through the logic for deriving fractional coverage
fields. In
> > your
> > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > probability values.
> > >
> > > Would that logic make sense to you?
> > >
> > > Thanks
> > > John
> > >
> > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Chris,
> > > >
> > > > In fact, SPC's 40-km radius probabilities were the motivation
for
> > > > adding the shape = circle option to MET.
> > > >
> > > > For your first question, no, there is not currently a Gaussian
> > > > smoother option for the "interp" section in Grid-Stat.
Currently,
> the
> > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > unweighted mean).
> > > > There is a distance-weighted mean option for point data where
the
> > > > weight is 1/distance squared. But that doesn't work for Grid-
Stat
> > > > because we'd end up dividing by 0.
> > > >
> > > > I can see how adding a gaussian smoother would be useful.
> > > >
> > > > Here's what we've done in the past with this SPC 40-km
probabilities:
> > > >
> > > > (1) Evaluate that field as a probability field (i.e. set "prob
=
> > > > TRUE;" in the config file).
> > > > (2) Use the "interp" section to define a smoothing operation
on the
> > > > observations:
> > > > interp = {
> > > > field = OBS;
> > > > vld_thresh = 1.0;
> > > > shape = CIRCLE;
> > > > type = [
> > > > { method = MAX; width = 11; }
> > > > ];
> > > > }
> > > > Or set width to some circle diameter in grid units that
works out
> > > > to 40km in the real world.
> > > >
> > > > In this example, we post-process the observations to make them
> > > > comparable the event for which the probability was defined.
> > > >
> > > > But I think I understand what you'd like to do... interpret as
the
> > > > probability value at each grid point as if it was a fractional
> > > > coverage value. So don't apply a threshold and neighborhood
size to
> > > > the forecast data at all. Just pass it directly to the
routine that
> > > computes FSS.
> > > >
> > > > Unfortunately no, I don't think there's a way of configuring
> Grid-Stat
> > > > to do that in met-6.1. If you think this logic would be
generally
> > > > useful, we could consider adding a flag to the config file to
enable
> > > that logic.
> > > >
> > > > Sorry I don't have better answers for you.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >>
> > > >> Hi John,
> > > >>
> > > >> Your suggestion for accomplishing #2 sounds promising. I
need to
> > think
> > > >> the details through some. On a somewhat related note, is
there a
> > > Gaussian
> > > >> smoother flag for smoothing the probabilistic fields? I
know that
> > was
> > > >> mentioned in Craig's paper and I have pursued that option in
the
> past
> > > >> (but just not using MET).
> > > >>
> > > >> I guess my basic question is can you apply a neighborhood
width that
> > > >> just includes the grid point maybe because you already have
> > > >> inherently a probabilistic field with spatial uncertainty?
Would
> that
> > > be a width of 1
> > > >> or does that also include the nearest grid point? A good
example
> of
> > > this
> > > >> would be a probabilistic forecast issued from the Storm
Prediction
> > > >> Center where the probabilities implicitly assume a 40-km
radius of
> > > >> influence (so there is no need to compute a fractional
coverage of
> the
> > > event).
> > > >>
> > > >> Thanks again,
> > > >> Chris
> > > >>
> > > >>
> > > >> //SIGNED//
> > > >>
> > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> christopher.melick at us.af.mil>
> > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC
16
> > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > >> grid_stat on Neighborhood statistics
> > > >>
> > > >> Chris,
> > > >>
> > > >> (1) The default shape is a square since that's what MET
previously
> > did.
> > > >> The circle is the *new* option.
> > > >>
> > > >> (2) For this one, the short answer is yes. FSS is computed
in MET
> by
> > > >> doing the following...
> > > >> - For each field separately (i.e. fcst and obs), apply the
> > > >> threshold
> > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > >> - Apply the neighborhood width and shape to compute a
fractional
> > > >> coverage value for each grid point.
> > > >> - Compare the fcst and obs fractional coverage fields to
compute
> > FSS.
> > > >>
> > > >> But there are some other options that you may find useful.
The
> > "interp"
> > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > operation.
> > > >> After reading the raw input data, you can specify if/how to
smooth
> > > >> that data prior to computing stats. For example, since you
> mentioned
> > > >> maximums, you could replace the value at each grid point with
the
> > > >> maximum of the values in a 5x5 box surrounding that point:
> > > >>
> > > >> interp = {
> > > >> field = FCST;
> > > >> vld_thresh = 1.0;
> > > >> shape = SQUARE;
> > > >>
> > > >> type = [
> > > >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> points
> > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of the
25
> > > >> closest points
> > > >> ];
> > > >> }
> > > >>
> > > >> Using the example above, you'll get 3 times the amount of
output
> from
> > > MET.
> > > >> The first smoother really is no smoothing at all. It would
run on
> > > >> the raw input data. The second smoother replaces the value
at each
> > > >> grid point with the maximum of the 25 surrounding points.
The third
> > > >> smoother replaces the value at each grid point with the mean
of
> those
> > > 25.
> > > >>
> > > >> I worked with Craig Schwartz when he was running MET for that
paper.
> > > >> My recollection is that he was able to use the configuration
options
> > > >> of Grid-Stat to accomplish the types of processing he was
after.
> But
> > > >> I don't recall all the details.
> > > >>
> > > >> If you explain exactly what you're trying to do, it may be
possible
> > > >> to configure Grid-Stat to accomplish that.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil
via
> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > >> >
> > > >> > Hi John,
> > > >> >
> > > >> > Your explanation and suggestions are very helpful and do
explain
> my
> > > >> > problems quite substantially. I will have to give it a try
to see
> > > >> > what happens.
> > > >> >
> > > >> > I do have a couple of other minor questions dealing with
how the
> > > >> > neighborhood verification is performed:
> > > >> > 1.) I noticed in the documentation that there is a flag
for
> > > >> > defining the shape of the neighborhood area (i.e. circular
or
> > > >> > square). What is the default if you don't specify?
> > > >> > 2.) In the literature, there are a couple of different
ways to
> > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz and
> > > >> > Sobash 2017;
> > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1).
> I
> > > >> > was curious is the only way to get the Fractions Skill
Score (FSS)
> > > >> > output is to define a neighborhood in which MET calculates
a
> > > >> > fractional coverage of the threshold event? I have
previously
> > > >> > explored computing the neighborhood maximum value within a
radius
> of
> > > each grid point before calculating a probability.
> > > >> >
> > > >> > Thanks,
> > > >> > Chris
> > > >> >
> > > >> >
> > > >> > //SIGNED//
> > > >> >
> > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> > christopher.melick at us.af.mil>
> > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > >> matthew.dawson.8 at us.af.mil>;
> > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil>
> > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > >> > grid_stat on Neighborhood statistics
> > > >> >
> > > >> > Hi Chris,
> > > >> >
> > > >> > I see that you're having difficulty computing neighborhood
> > > >> > statistics in Grid-Stat.
> > > >> >
> > > >> > First, Grid-Stat definitely does have the ability to
threshold the
> > > >> StageIV
> > > >> > data on the fly. So pre-processing to 0's and 1's first is
not
> > > needed.
> > > >> > Instead, in the "obs" dictionary of the config file, you'd
just
> set:
> > > >> > cat_thresh = [ >=2.54 ];
> > > >> >
> > > >> > Since StageIV precip is millimeters, >=2.54 mm is the same
as
> >=0.1
> > > >> inches.
> > > >> >
> > > >> > If for some reason you have data that's already 0's and
1's, you'd
> > > >> > just use a different threshold in the config file:
> > > >> > cat_thresh = [ ==1 ];
> > > >> >
> > > >> > This tells MET that anywhere you see a 1 in the data, the
event is
> > > >> > occurring. Otherwise, it's not.
> > > >> >
> > > >> > Now on the to real issue. Looking at your config file, I
see that
> > > >> you're
> > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > >> > computes
> > > >> the
> > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's
why
> > > >> > you're getting no output in the NBRCNT line type, which is
the one
> > > >> > that
> > > >> contains
> > > >> > Fractions Skill Score (FSS).
> > > >> >
> > > >> > In the attached config file, I've made a few changes:
> > > >> > - Set the obtype to indicate that you're verifying against
> StageIV
> > > >> > - Defined 2 fcst.field entries.
> > > >> > - In the first we process QP010 as a probability
field.
> Here
> > > >> > cat_thresh is set to ==0.10 which is shorthand notation for
> > > >> > defining
> > > >> bins
> > > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > > >> > - In the second we process it as a field of scalars.
Here
> > > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > > >> > I'm
> > > >> using
> > > >> > for defining events for fractions skill score.
> > > >> > - Defined 2 corresponding obs.field entries to verify the
> forecast
> > > >> data.
> > > >> > In both the verifying threshold is >=2.54. This assumes
that you
> > > >> > pass
> > > >> the
> > > >> > raw StageIV data as input.
> > > >> > - In the output_flag section, I turned off the line types
that
> > > >> > don't apply.
> > > >> >
> > > >> > Lastly, while setting "level = [ "R19" ];" works as a way
of
> > > >> > grabbing record number 19 from the input file, it's not
> > > >> > recommended. Instead,
> > > >> it'd
> > > >> > be better to figure out how to set the level string so that
MET
> > > >> > always grabs the same record, regardless if it's in spot
number 19
> > > >> > or 20 or any other location in the GRIB file.
> > > >> >
> > > >> > Hope this helps clarify. What other questions do you have?
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > John Halley Gotway
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil via
> RT
> > > >> > < met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > >> > > Queue: met_help
> > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > >> > > Owner: Nobody
> > > >> > > Requestors: christopher.melick at us.af.mil
> > > >> > > Status: new
> > > >> > > Ticket <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > Hi John,
> > > >> > >
> > > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > > >> > > statistics for probability of precipitation from some
ensemble
> > > >> > > data with the observations
> > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting no
> > > output
> > > >> > > (just
> > > >> > > headers) in many of the ASCII files from the program when
I run
> > it.
> > > >> > Could
> > > >> > > you help me figure out what I am doing wrong? I have
provided
> > > >> > > the command line specifics I use as well as the output
log file
> > > >> > > that is
> > > >> > created when
> > > >> > > running that command. I also have attached the
configuration
> > file.
> > > >> I
> > > >> > am
> > > >> > > having trouble using ftp to send the raw data from any of
the
> > > clients
> > > >> > > available to us. I believe this restriction occurred
recently
> > > since
> > > >> Bob
> > > >> > > Craig had mentioned that he had been able to connect
externally
> > > >> > > in the past.
> > > >> > >
> > > >> > > I should mention that I have pre-processed the Stage IV
analyses
> > > >> > > already to be either 0's or 1's for the threshold
(>=0.1") since
> > > >> > > I was unclear as to whether MET was able to do that on
the fly
> > > >> > > when running
> > > >> > grid_stat.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Chris
> > > >> > >
> > > >> > > Python command:
> > > >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > > >> > > 'precip/post-processed-obs.nc', \
> > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-
v', '5]
> > > >> > > Call(metcall)
> > > >> > >
> > > >> > > //SIGNED//
> > > >> > >
> > > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > >> > > Verification Exploitation Team 16th Weather Squadron (16
> WS/WXEV)
> > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > >> > > 294-3693
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Tue Feb 20 15:38:40 2018
John,
After thinking about it some, I was curious why the F_RATE and O_RATE
can't be calculated in the cases below? Can't the gridded data still
be read in to determine the relative frequency of the events,
regardless whether grid_stat is used to compute fractional coverage
fields? I am basically doing that right now in some Python code I
developed when calculating Fractions Skill Score.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 14, 2018 3:20 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I'm working on the interp.field option we discussed.
I have a quick question for you. The NBRCNT line type that is
computed by
grid_stat contains columns for F_RATE and O_RATE. That is the ratio
of grid points over a masking region at which the "event" is occurring
in the forecast and observation fields respectively.
When grid_stat computes the fractional coverage field, we can compute
that. But if grid_stat does *NOT* compute the fractional coverage
fields, then we don't know what those ratio's are.
- If interp.field = FCST, then O_RATE = NA.
- If interp.field = OBS, then F_RATE = NA.
- If interp.field = NONE, then F_RATE = O_RATE = NA.
Make sense?
Thanks,
John
On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> That sounds like an excellent plan. Thanks for your assistance and
> trying to accommodate and make changes to the MET software so
quickly.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 14, 2018 10:36 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Just wanted to close the loop on this ticket. I'll try to get it
into the
> next release due out next in March.
>
> Thanks,
> John
>
> On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Yes, thanks for the explanations. That clears up any possible
> > ambiguity and confusion. I kind of thought that was the
procedure.
> >
> > When do you think that the next MET would be released with the
> > enhancement that we discussed for the neighborhood statistics? Is
it
> > an easy or difficult update to the software?
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, February 8, 2018 3:24 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > It sounds like you understand how the fractional coverage field is
> > computed.
> >
> > - Read the raw field... probability values between 0 and 1 in your
case.
> > - Apply the categorical threshold to that data to generate a field
of 0's
> > and 1's... >0.5 in your case.
> > - For each grid point, replace the value at that grid point with
the
> > "fraction" of 1's in the neighborhood around that point... e.g.
the event
> > occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
> > point, so fractional coverage = 4/9.
> > - Compare the forecast fractional coverage values to the observed
> > fractional coverage values to compute FSS.
> >
> > Yes, I do think it would be a good idea to talk to Craig about how
he
> > configured MET for his work.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Getting back to your initial response, I was able to run
grid_stat
> > > with your changes in the attached configuration file to get FSS
output
> > > (as well as the other statistics for the neighborhood type
results).
> > >
> > > However, I do have a question about the forecast categorical
threshold
> > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > probabilities from the ensemble probabilities: What is the
actual
> > > formula that MET uses to create the final field to calculate
FSS? Is
> > > it a strict neighborhood fractional coverage of ensemble
probabilities
> > > above 0.5 in some set radius of influence? Or is it some other
> > > operation or does the threshold represent something else? I
know in
> > > Craig's paper, they were calling the last step a neighborhood
average
> > > of the ensemble probabilities to calculate NEP (since the event
had
> > > already been defined at the grid point in determining the
ensemble
> > probability [i.e. likelihood of exceeding
> > > 0.1" of precipitation]). Maybe, I should understand how Craig
was
> using
> > > MET for his paper.
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > I was thinking about what changes would be needed to compute FSS
on
> input
> > > probability data. We could add a “field” option to the “nbrhd”
> section,
> > > just like we have in the “interp” section. It would be set to
FCST,
> OBS,
> > > or BOTH (default) and would control the logic as to which fields
should
> > be
> > > passed through the logic for deriving fractional coverage
fields. In
> > your
> > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > probability values.
> > >
> > > Would that logic make sense to you?
> > >
> > > Thanks
> > > John
> > >
> > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Chris,
> > > >
> > > > In fact, SPC's 40-km radius probabilities were the motivation
for
> > > > adding the shape = circle option to MET.
> > > >
> > > > For your first question, no, there is not currently a Gaussian
> > > > smoother option for the "interp" section in Grid-Stat.
Currently,
> the
> > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > unweighted mean).
> > > > There is a distance-weighted mean option for point data where
the
> > > > weight is 1/distance squared. But that doesn't work for Grid-
Stat
> > > > because we'd end up dividing by 0.
> > > >
> > > > I can see how adding a gaussian smoother would be useful.
> > > >
> > > > Here's what we've done in the past with this SPC 40-km
probabilities:
> > > >
> > > > (1) Evaluate that field as a probability field (i.e. set "prob
=
> > > > TRUE;" in the config file).
> > > > (2) Use the "interp" section to define a smoothing operation
on the
> > > > observations:
> > > > interp = {
> > > > field = OBS;
> > > > vld_thresh = 1.0;
> > > > shape = CIRCLE;
> > > > type = [
> > > > { method = MAX; width = 11; }
> > > > ];
> > > > }
> > > > Or set width to some circle diameter in grid units that
works out
> > > > to 40km in the real world.
> > > >
> > > > In this example, we post-process the observations to make them
> > > > comparable the event for which the probability was defined.
> > > >
> > > > But I think I understand what you'd like to do... interpret as
the
> > > > probability value at each grid point as if it was a fractional
> > > > coverage value. So don't apply a threshold and neighborhood
size to
> > > > the forecast data at all. Just pass it directly to the
routine that
> > > computes FSS.
> > > >
> > > > Unfortunately no, I don't think there's a way of configuring
> Grid-Stat
> > > > to do that in met-6.1. If you think this logic would be
generally
> > > > useful, we could consider adding a flag to the config file to
enable
> > > that logic.
> > > >
> > > > Sorry I don't have better answers for you.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >>
> > > >> Hi John,
> > > >>
> > > >> Your suggestion for accomplishing #2 sounds promising. I
need to
> > think
> > > >> the details through some. On a somewhat related note, is
there a
> > > Gaussian
> > > >> smoother flag for smoothing the probabilistic fields? I
know that
> > was
> > > >> mentioned in Craig's paper and I have pursued that option in
the
> past
> > > >> (but just not using MET).
> > > >>
> > > >> I guess my basic question is can you apply a neighborhood
width that
> > > >> just includes the grid point maybe because you already have
> > > >> inherently a probabilistic field with spatial uncertainty?
Would
> that
> > > be a width of 1
> > > >> or does that also include the nearest grid point? A good
example
> of
> > > this
> > > >> would be a probabilistic forecast issued from the Storm
Prediction
> > > >> Center where the probabilities implicitly assume a 40-km
radius of
> > > >> influence (so there is no need to compute a fractional
coverage of
> the
> > > event).
> > > >>
> > > >> Thanks again,
> > > >> Chris
> > > >>
> > > >>
> > > >> //SIGNED//
> > > >>
> > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> christopher.melick at us.af.mil>
> > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC
16
> > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > >> grid_stat on Neighborhood statistics
> > > >>
> > > >> Chris,
> > > >>
> > > >> (1) The default shape is a square since that's what MET
previously
> > did.
> > > >> The circle is the *new* option.
> > > >>
> > > >> (2) For this one, the short answer is yes. FSS is computed
in MET
> by
> > > >> doing the following...
> > > >> - For each field separately (i.e. fcst and obs), apply the
> > > >> threshold
> > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > >> - Apply the neighborhood width and shape to compute a
fractional
> > > >> coverage value for each grid point.
> > > >> - Compare the fcst and obs fractional coverage fields to
compute
> > FSS.
> > > >>
> > > >> But there are some other options that you may find useful.
The
> > "interp"
> > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > operation.
> > > >> After reading the raw input data, you can specify if/how to
smooth
> > > >> that data prior to computing stats. For example, since you
> mentioned
> > > >> maximums, you could replace the value at each grid point with
the
> > > >> maximum of the values in a 5x5 box surrounding that point:
> > > >>
> > > >> interp = {
> > > >> field = FCST;
> > > >> vld_thresh = 1.0;
> > > >> shape = SQUARE;
> > > >>
> > > >> type = [
> > > >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> points
> > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of the
25
> > > >> closest points
> > > >> ];
> > > >> }
> > > >>
> > > >> Using the example above, you'll get 3 times the amount of
output
> from
> > > MET.
> > > >> The first smoother really is no smoothing at all. It would
run on
> > > >> the raw input data. The second smoother replaces the value
at each
> > > >> grid point with the maximum of the 25 surrounding points.
The third
> > > >> smoother replaces the value at each grid point with the mean
of
> those
> > > 25.
> > > >>
> > > >> I worked with Craig Schwartz when he was running MET for that
paper.
> > > >> My recollection is that he was able to use the configuration
options
> > > >> of Grid-Stat to accomplish the types of processing he was
after.
> But
> > > >> I don't recall all the details.
> > > >>
> > > >> If you explain exactly what you're trying to do, it may be
possible
> > > >> to configure Grid-Stat to accomplish that.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil
via
> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > >> >
> > > >> > Hi John,
> > > >> >
> > > >> > Your explanation and suggestions are very helpful and do
explain
> my
> > > >> > problems quite substantially. I will have to give it a try
to see
> > > >> > what happens.
> > > >> >
> > > >> > I do have a couple of other minor questions dealing with
how the
> > > >> > neighborhood verification is performed:
> > > >> > 1.) I noticed in the documentation that there is a flag
for
> > > >> > defining the shape of the neighborhood area (i.e. circular
or
> > > >> > square). What is the default if you don't specify?
> > > >> > 2.) In the literature, there are a couple of different
ways to
> > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz and
> > > >> > Sobash 2017;
> > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1).
> I
> > > >> > was curious is the only way to get the Fractions Skill
Score (FSS)
> > > >> > output is to define a neighborhood in which MET calculates
a
> > > >> > fractional coverage of the threshold event? I have
previously
> > > >> > explored computing the neighborhood maximum value within a
radius
> of
> > > each grid point before calculating a probability.
> > > >> >
> > > >> > Thanks,
> > > >> > Chris
> > > >> >
> > > >> >
> > > >> > //SIGNED//
> > > >> >
> > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > >> > christopher.melick at us.af.mil>
> > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > >> matthew.dawson.8 at us.af.mil>;
> > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil>
> > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > >> > grid_stat on Neighborhood statistics
> > > >> >
> > > >> > Hi Chris,
> > > >> >
> > > >> > I see that you're having difficulty computing neighborhood
> > > >> > statistics in Grid-Stat.
> > > >> >
> > > >> > First, Grid-Stat definitely does have the ability to
threshold the
> > > >> StageIV
> > > >> > data on the fly. So pre-processing to 0's and 1's first is
not
> > > needed.
> > > >> > Instead, in the "obs" dictionary of the config file, you'd
just
> set:
> > > >> > cat_thresh = [ >=2.54 ];
> > > >> >
> > > >> > Since StageIV precip is millimeters, >=2.54 mm is the same
as
> >=0.1
> > > >> inches.
> > > >> >
> > > >> > If for some reason you have data that's already 0's and
1's, you'd
> > > >> > just use a different threshold in the config file:
> > > >> > cat_thresh = [ ==1 ];
> > > >> >
> > > >> > This tells MET that anywhere you see a 1 in the data, the
event is
> > > >> > occurring. Otherwise, it's not.
> > > >> >
> > > >> > Now on the to real issue. Looking at your config file, I
see that
> > > >> you're
> > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > >> > computes
> > > >> the
> > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's
why
> > > >> > you're getting no output in the NBRCNT line type, which is
the one
> > > >> > that
> > > >> contains
> > > >> > Fractions Skill Score (FSS).
> > > >> >
> > > >> > In the attached config file, I've made a few changes:
> > > >> > - Set the obtype to indicate that you're verifying against
> StageIV
> > > >> > - Defined 2 fcst.field entries.
> > > >> > - In the first we process QP010 as a probability
field.
> Here
> > > >> > cat_thresh is set to ==0.10 which is shorthand notation for
> > > >> > defining
> > > >> bins
> > > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > > >> > - In the second we process it as a field of scalars.
Here
> > > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > > >> > I'm
> > > >> using
> > > >> > for defining events for fractions skill score.
> > > >> > - Defined 2 corresponding obs.field entries to verify the
> forecast
> > > >> data.
> > > >> > In both the verifying threshold is >=2.54. This assumes
that you
> > > >> > pass
> > > >> the
> > > >> > raw StageIV data as input.
> > > >> > - In the output_flag section, I turned off the line types
that
> > > >> > don't apply.
> > > >> >
> > > >> > Lastly, while setting "level = [ "R19" ];" works as a way
of
> > > >> > grabbing record number 19 from the input file, it's not
> > > >> > recommended. Instead,
> > > >> it'd
> > > >> > be better to figure out how to set the level string so that
MET
> > > >> > always grabs the same record, regardless if it's in spot
number 19
> > > >> > or 20 or any other location in the GRIB file.
> > > >> >
> > > >> > Hope this helps clarify. What other questions do you have?
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > John Halley Gotway
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil via
> RT
> > > >> > < met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > >> > > Queue: met_help
> > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > >> > > Owner: Nobody
> > > >> > > Requestors: christopher.melick at us.af.mil
> > > >> > > Status: new
> > > >> > > Ticket <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > Hi John,
> > > >> > >
> > > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > > >> > > statistics for probability of precipitation from some
ensemble
> > > >> > > data with the observations
> > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting no
> > > output
> > > >> > > (just
> > > >> > > headers) in many of the ASCII files from the program when
I run
> > it.
> > > >> > Could
> > > >> > > you help me figure out what I am doing wrong? I have
provided
> > > >> > > the command line specifics I use as well as the output
log file
> > > >> > > that is
> > > >> > created when
> > > >> > > running that command. I also have attached the
configuration
> > file.
> > > >> I
> > > >> > am
> > > >> > > having trouble using ftp to send the raw data from any of
the
> > > clients
> > > >> > > available to us. I believe this restriction occurred
recently
> > > since
> > > >> Bob
> > > >> > > Craig had mentioned that he had been able to connect
externally
> > > >> > > in the past.
> > > >> > >
> > > >> > > I should mention that I have pre-processed the Stage IV
analyses
> > > >> > > already to be either 0's or 1's for the threshold
(>=0.1") since
> > > >> > > I was unclear as to whether MET was able to do that on
the fly
> > > >> > > when running
> > > >> > grid_stat.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Chris
> > > >> > >
> > > >> > > Python command:
> > > >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > > >> > > 'precip/post-processed-obs.nc', \
> > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-
v', '5]
> > > >> > > Call(metcall)
> > > >> > >
> > > >> > > //SIGNED//
> > > >> > >
> > > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > >> > > Verification Exploitation Team 16th Weather Squadron (16
> WS/WXEV)
> > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > >> > > 294-3693
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Tue Feb 20 19:19:53 2018
Chris,
Your email got me thinking. Thinking through some toy examples in my
mind,
I can see that probably yes, the mean of the fractional coverage field
*should* be the same as the frequency of the event.
I ran MET in the debugger and was able to stop at the point in the
logic
where this is computed. At that point, we're passing around arrays of
fractional coverage field values as well as arrays of 0's and 1's
indicating whether the event did or did not occur at each grid point.
Comparing the mean of these arrays, I find that they are close but not
identical. Sometimes one is larger, sometimes the other.
I can think of 2 reasons why they wouldn't be the same, but these may
not
fully explain it:
(1) In Grid-Stat, the fractional coverage field is computed once over
the
full verification domain. Then, the masking regions are processed as
subsets of these grid points. For a grid point that falls just barely
inside that mask, around 1/2 of the grid points used to compute the
fractional coverage value came from *OUTSIDE* that masking region.
The 0/1
arrays are only for points inside the current mask. So the mean of
the 0/1
array represents only the points inside the mask while the mean of the
fractional coverage includes info from outside the mask.
(2) When data values are missing in the forecast or observation
fields,
depending on how the user has set the vld_thresh config file option,
the
"denominator" used to compute the fractional coverage field value
could
change from grid point to grid point. That would likely throw off the
mean
of the fractional coverage field.
Now what is the *CORRECT* way of computing the forecast and
observation
rates? In papers about this, I see references to the "base_rate" but
no
explicit equation.
One very nice feature about the way MET is currently doing it is that
F_RATE and O_RATE remain constant regardless of the neighborhood size.
Therefore, UFSS and AFSS remain constant as well. Using the other
method,
O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood size
since the number of points included from outside the mask would
change.
You mentioned that you had issues with negative UFSS and AFSS values.
Perhaps it's related to how you're computing the F_RATE and O_RATE?
Lots of little details to consider here!
Thanks,
John
On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> After thinking about it some, I was curious why the F_RATE and
O_RATE
> can't be calculated in the cases below? Can't the gridded data
still be
> read in to determine the relative frequency of the events,
regardless
> whether grid_stat is used to compute fractional coverage fields? I
am
> basically doing that right now in some Python code I developed when
> calculating Fractions Skill Score.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 14, 2018 3:20 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I'm working on the interp.field option we discussed.
>
> I have a quick question for you. The NBRCNT line type that is
computed by
> grid_stat contains columns for F_RATE and O_RATE. That is the ratio
of
> grid points over a masking region at which the "event" is occurring
in the
> forecast and observation fields respectively.
>
> When grid_stat computes the fractional coverage field, we can
compute
> that. But if grid_stat does *NOT* compute the fractional coverage
fields,
> then we don't know what those ratio's are.
>
> - If interp.field = FCST, then O_RATE = NA.
> - If interp.field = OBS, then F_RATE = NA.
> - If interp.field = NONE, then F_RATE = O_RATE = NA.
>
> Make sense?
>
> Thanks,
> John
>
> On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > That sounds like an excellent plan. Thanks for your assistance
and
> > trying to accommodate and make changes to the MET software so
quickly.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 14, 2018 10:36 AM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > Just wanted to close the loop on this ticket. I'll try to get it
into
> the
> > next release due out next in March.
> >
> > Thanks,
> > John
> >
> > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Yes, thanks for the explanations. That clears up any possible
> > > ambiguity and confusion. I kind of thought that was the
procedure.
> > >
> > > When do you think that the next MET would be released with the
> > > enhancement that we discussed for the neighborhood statistics?
Is it
> > > an easy or difficult update to the software?
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, February 8, 2018 3:24 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > It sounds like you understand how the fractional coverage field
is
> > > computed.
> > >
> > > - Read the raw field... probability values between 0 and 1 in
your
> case.
> > > - Apply the categorical threshold to that data to generate a
field of
> 0's
> > > and 1's... >0.5 in your case.
> > > - For each grid point, replace the value at that grid point with
the
> > > "fraction" of 1's in the neighborhood around that point... e.g.
the
> event
> > > occurs at 4 of the 9 grid points in a 3x3 square surrounding the
> current
> > > point, so fractional coverage = 4/9.
> > > - Compare the forecast fractional coverage values to the
observed
> > > fractional coverage values to compute FSS.
> > >
> > > Yes, I do think it would be a good idea to talk to Craig about
how he
> > > configured MET for his work.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > Getting back to your initial response, I was able to run
grid_stat
> > > > with your changes in the attached configuration file to get
FSS
> output
> > > > (as well as the other statistics for the neighborhood type
results).
> > > >
> > > > However, I do have a question about the forecast categorical
> threshold
> > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > probabilities from the ensemble probabilities: What is the
actual
> > > > formula that MET uses to create the final field to calculate
FSS? Is
> > > > it a strict neighborhood fractional coverage of ensemble
> probabilities
> > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > operation or does the threshold represent something else? I
know in
> > > > Craig's paper, they were calling the last step a neighborhood
average
> > > > of the ensemble probabilities to calculate NEP (since the
event had
> > > > already been defined at the grid point in determining the
ensemble
> > > probability [i.e. likelihood of exceeding
> > > > 0.1" of precipitation]). Maybe, I should understand how
Craig was
> > using
> > > > MET for his paper.
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Hi Chris,
> > > >
> > > > I was thinking about what changes would be needed to compute
FSS on
> > input
> > > > probability data. We could add a “field” option to the
“nbrhd”
> > section,
> > > > just like we have in the “interp” section. It would be set to
FCST,
> > OBS,
> > > > or BOTH (default) and would control the logic as to which
fields
> should
> > > be
> > > > passed through the logic for deriving fractional coverage
fields. In
> > > your
> > > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > > probability values.
> > > >
> > > > Would that logic make sense to you?
> > > >
> > > > Thanks
> > > > John
> > > >
> > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > In fact, SPC's 40-km radius probabilities were the
motivation for
> > > > > adding the shape = circle option to MET.
> > > > >
> > > > > For your first question, no, there is not currently a
Gaussian
> > > > > smoother option for the "interp" section in Grid-Stat.
Currently,
> > the
> > > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > > unweighted mean).
> > > > > There is a distance-weighted mean option for point data
where the
> > > > > weight is 1/distance squared. But that doesn't work for
Grid-Stat
> > > > > because we'd end up dividing by 0.
> > > > >
> > > > > I can see how adding a gaussian smoother would be useful.
> > > > >
> > > > > Here's what we've done in the past with this SPC 40-km
> probabilities:
> > > > >
> > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
> > > > > TRUE;" in the config file).
> > > > > (2) Use the "interp" section to define a smoothing operation
on the
> > > > > observations:
> > > > > interp = {
> > > > > field = OBS;
> > > > > vld_thresh = 1.0;
> > > > > shape = CIRCLE;
> > > > > type = [
> > > > > { method = MAX; width = 11; }
> > > > > ];
> > > > > }
> > > > > Or set width to some circle diameter in grid units that
works
> out
> > > > > to 40km in the real world.
> > > > >
> > > > > In this example, we post-process the observations to make
them
> > > > > comparable the event for which the probability was defined.
> > > > >
> > > > > But I think I understand what you'd like to do... interpret
as the
> > > > > probability value at each grid point as if it was a
fractional
> > > > > coverage value. So don't apply a threshold and neighborhood
size
> to
> > > > > the forecast data at all. Just pass it directly to the
routine
> that
> > > > computes FSS.
> > > > >
> > > > > Unfortunately no, I don't think there's a way of configuring
> > Grid-Stat
> > > > > to do that in met-6.1. If you think this logic would be
generally
> > > > > useful, we could consider adding a flag to the config file
to
> enable
> > > > that logic.
> > > > >
> > > > > Sorry I don't have better answers for you.
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >>
> > > > >> Hi John,
> > > > >>
> > > > >> Your suggestion for accomplishing #2 sounds promising. I
need to
> > > think
> > > > >> the details through some. On a somewhat related note, is
there a
> > > > Gaussian
> > > > >> smoother flag for smoothing the probabilistic fields? I
know
> that
> > > was
> > > > >> mentioned in Craig's paper and I have pursued that option
in the
> > past
> > > > >> (but just not using MET).
> > > > >>
> > > > >> I guess my basic question is can you apply a neighborhood
width
> that
> > > > >> just includes the grid point maybe because you already have
> > > > >> inherently a probabilistic field with spatial uncertainty?
Would
> > that
> > > > be a width of 1
> > > > >> or does that also include the nearest grid point? A good
> example
> > of
> > > > this
> > > > >> would be a probabilistic forecast issued from the Storm
Prediction
> > > > >> Center where the probabilities implicitly assume a 40-km
radius of
> > > > >> influence (so there is no need to compute a fractional
coverage of
> > the
> > > > event).
> > > > >>
> > > > >> Thanks again,
> > > > >> Chris
> > > > >>
> > > > >>
> > > > >> //SIGNED//
> > > > >>
> > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >>
> > > > >>
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > >> christopher.melick at us.af.mil>
> > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC 16
> > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > >> grid_stat on Neighborhood statistics
> > > > >>
> > > > >> Chris,
> > > > >>
> > > > >> (1) The default shape is a square since that's what MET
previously
> > > did.
> > > > >> The circle is the *new* option.
> > > > >>
> > > > >> (2) For this one, the short answer is yes. FSS is computed
in MET
> > by
> > > > >> doing the following...
> > > > >> - For each field separately (i.e. fcst and obs), apply
the
> > > > >> threshold
> > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > >> - Apply the neighborhood width and shape to compute a
fractional
> > > > >> coverage value for each grid point.
> > > > >> - Compare the fcst and obs fractional coverage fields to
compute
> > > FSS.
> > > > >>
> > > > >> But there are some other options that you may find useful.
The
> > > "interp"
> > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > operation.
> > > > >> After reading the raw input data, you can specify if/how to
smooth
> > > > >> that data prior to computing stats. For example, since you
> > mentioned
> > > > >> maximums, you could replace the value at each grid point
with the
> > > > >> maximum of the values in a 5x5 box surrounding that point:
> > > > >>
> > > > >> interp = {
> > > > >> field = FCST;
> > > > >> vld_thresh = 1.0;
> > > > >> shape = SQUARE;
> > > > >>
> > > > >> type = [
> > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > points
> > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
> > > > >> closest points
> > > > >> ];
> > > > >> }
> > > > >>
> > > > >> Using the example above, you'll get 3 times the amount of
output
> > from
> > > > MET.
> > > > >> The first smoother really is no smoothing at all. It would
run on
> > > > >> the raw input data. The second smoother replaces the value
at
> each
> > > > >> grid point with the maximum of the 25 surrounding points.
The
> third
> > > > >> smoother replaces the value at each grid point with the
mean of
> > those
> > > > 25.
> > > > >>
> > > > >> I worked with Craig Schwartz when he was running MET for
that
> paper.
> > > > >> My recollection is that he was able to use the
configuration
> options
> > > > >> of Grid-Stat to accomplish the types of processing he was
after.
> > But
> > > > >> I don't recall all the details.
> > > > >>
> > > > >> If you explain exactly what you're trying to do, it may be
> possible
> > > > >> to configure Grid-Stat to accomplish that.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil via
> > RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >> >
> > > > >> > Hi John,
> > > > >> >
> > > > >> > Your explanation and suggestions are very helpful and do
explain
> > my
> > > > >> > problems quite substantially. I will have to give it a
try to
> see
> > > > >> > what happens.
> > > > >> >
> > > > >> > I do have a couple of other minor questions dealing with
how the
> > > > >> > neighborhood verification is performed:
> > > > >> > 1.) I noticed in the documentation that there is a flag
for
> > > > >> > defining the shape of the neighborhood area (i.e.
circular or
> > > > >> > square). What is the default if you don't specify?
> > > > >> > 2.) In the literature, there are a couple of different
ways to
> > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz and
> > > > >> > Sobash 2017;
> > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1).
> > I
> > > > >> > was curious is the only way to get the Fractions Skill
Score
> (FSS)
> > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > >> > fractional coverage of the threshold event? I have
previously
> > > > >> > explored computing the neighborhood maximum value within
a
> radius
> > of
> > > > each grid point before calculating a probability.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Chris
> > > > >> >
> > > > >> >
> > > > >> > //SIGNED//
> > > > >> >
> > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> Weather
> > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > -----Original Message-----
> > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > >> > christopher.melick at us.af.mil>
> > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > >> matthew.dawson.8 at us.af.mil>;
> > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil>
> > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > >> > grid_stat on Neighborhood statistics
> > > > >> >
> > > > >> > Hi Chris,
> > > > >> >
> > > > >> > I see that you're having difficulty computing
neighborhood
> > > > >> > statistics in Grid-Stat.
> > > > >> >
> > > > >> > First, Grid-Stat definitely does have the ability to
threshold
> the
> > > > >> StageIV
> > > > >> > data on the fly. So pre-processing to 0's and 1's first
is not
> > > > needed.
> > > > >> > Instead, in the "obs" dictionary of the config file,
you'd just
> > set:
> > > > >> > cat_thresh = [ >=2.54 ];
> > > > >> >
> > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
> > >=0.1
> > > > >> inches.
> > > > >> >
> > > > >> > If for some reason you have data that's already 0's and
1's,
> you'd
> > > > >> > just use a different threshold in the config file:
> > > > >> > cat_thresh = [ ==1 ];
> > > > >> >
> > > > >> > This tells MET that anywhere you see a 1 in the data, the
event
> is
> > > > >> > occurring. Otherwise, it's not.
> > > > >> >
> > > > >> > Now on the to real issue. Looking at your config file, I
see
> that
> > > > >> you're
> > > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > > >> > computes
> > > > >> the
> > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's why
> > > > >> > you're getting no output in the NBRCNT line type, which
is the
> one
> > > > >> > that
> > > > >> contains
> > > > >> > Fractions Skill Score (FSS).
> > > > >> >
> > > > >> > In the attached config file, I've made a few changes:
> > > > >> > - Set the obtype to indicate that you're verifying
against
> > StageIV
> > > > >> > - Defined 2 fcst.field entries.
> > > > >> > - In the first we process QP010 as a probability
field.
> > Here
> > > > >> > cat_thresh is set to ==0.10 which is shorthand notation
for
> > > > >> > defining
> > > > >> bins
> > > > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > > > >> > - In the second we process it as a field of
scalars. Here
> > > > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > > > >> > I'm
> > > > >> using
> > > > >> > for defining events for fractions skill score.
> > > > >> > - Defined 2 corresponding obs.field entries to verify
the
> > forecast
> > > > >> data.
> > > > >> > In both the verifying threshold is >=2.54. This assumes
that you
> > > > >> > pass
> > > > >> the
> > > > >> > raw StageIV data as input.
> > > > >> > - In the output_flag section, I turned off the line
types that
> > > > >> > don't apply.
> > > > >> >
> > > > >> > Lastly, while setting "level = [ "R19" ];" works as a way
of
> > > > >> > grabbing record number 19 from the input file, it's not
> > > > >> > recommended. Instead,
> > > > >> it'd
> > > > >> > be better to figure out how to set the level string so
that MET
> > > > >> > always grabs the same record, regardless if it's in spot
number
> 19
> > > > >> > or 20 or any other location in the GRIB file.
> > > > >> >
> > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > John Halley Gotway
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
> via
> > RT
> > > > >> > < met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > > >> > > Queue: met_help
> > > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > > >> > > Owner: Nobody
> > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > >> > > Status: new
> > > > >> > > Ticket <URL:
> > > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Hi John,
> > > > >> > >
> > > > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > > > >> > > statistics for probability of precipitation from some
ensemble
> > > > >> > > data with the observations
> > > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting
> no
> > > > output
> > > > >> > > (just
> > > > >> > > headers) in many of the ASCII files from the program
when I
> run
> > > it.
> > > > >> > Could
> > > > >> > > you help me figure out what I am doing wrong? I have
provided
> > > > >> > > the command line specifics I use as well as the output
log
> file
> > > > >> > > that is
> > > > >> > created when
> > > > >> > > running that command. I also have attached the
configuration
> > > file.
> > > > >> I
> > > > >> > am
> > > > >> > > having trouble using ftp to send the raw data from any
of the
> > > > clients
> > > > >> > > available to us. I believe this restriction occurred
> recently
> > > > since
> > > > >> Bob
> > > > >> > > Craig had mentioned that he had been able to connect
> externally
> > > > >> > > in the past.
> > > > >> > >
> > > > >> > > I should mention that I have pre-processed the Stage IV
> analyses
> > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> since
> > > > >> > > I was unclear as to whether MET was able to do that on
the fly
> > > > >> > > when running
> > > > >> > grid_stat.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Chris
> > > > >> > >
> > > > >> > > Python command:
> > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > >> > > 'precip/post-processed-obs.nc', \
> > > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp',
'-v',
> '5]
> > > > >> > > Call(metcall)
> > > > >> > >
> > > > >> > > //SIGNED//
> > > > >> > >
> > > > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
> > WS/WXEV)
> > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > > >> > > 294-3693
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 21 08:36:14 2018
Hi John,
I don't get negative values for AFSS but I do get values that are less
than what I calculate for FSS. I approach the procedure for computing
all the metrics such that everything is on the same "playing ground".
Thus, F_RATE and O_RATE are only computed by the mask of defined grid
points on *BOTH* the observation and forecast grids. When I process
the sample time period I sent you, F_RATE >> O_RATE which causes AFSS
to be a small value. Is that not the approach that I should follow?
Were you able to examine my script and raw data file I sent you?
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, February 20, 2018 8:20 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
Your email got me thinking. Thinking through some toy examples in my
mind, I can see that probably yes, the mean of the fractional coverage
field
*should* be the same as the frequency of the event.
I ran MET in the debugger and was able to stop at the point in the
logic where this is computed. At that point, we're passing around
arrays of fractional coverage field values as well as arrays of 0's
and 1's indicating whether the event did or did not occur at each grid
point.
Comparing the mean of these arrays, I find that they are close but not
identical. Sometimes one is larger, sometimes the other.
I can think of 2 reasons why they wouldn't be the same, but these may
not fully explain it:
(1) In Grid-Stat, the fractional coverage field is computed once over
the full verification domain. Then, the masking regions are processed
as subsets of these grid points. For a grid point that falls just
barely inside that mask, around 1/2 of the grid points used to compute
the fractional coverage value came from *OUTSIDE* that masking region.
The 0/1 arrays are only for points inside the current mask. So the
mean of the 0/1 array represents only the points inside the mask while
the mean of the fractional coverage includes info from outside the
mask.
(2) When data values are missing in the forecast or observation
fields, depending on how the user has set the vld_thresh config file
option, the "denominator" used to compute the fractional coverage
field value could change from grid point to grid point. That would
likely throw off the mean of the fractional coverage field.
Now what is the *CORRECT* way of computing the forecast and
observation rates? In papers about this, I see references to the
"base_rate" but no explicit equation.
One very nice feature about the way MET is currently doing it is that
F_RATE and O_RATE remain constant regardless of the neighborhood size.
Therefore, UFSS and AFSS remain constant as well. Using the other
method, O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood size since the number of points included from outside the
mask would change.
You mentioned that you had issues with negative UFSS and AFSS values.
Perhaps it's related to how you're computing the F_RATE and O_RATE?
Lots of little details to consider here!
Thanks,
John
On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> After thinking about it some, I was curious why the F_RATE and
O_RATE
> can't be calculated in the cases below? Can't the gridded data
still
> be read in to determine the relative frequency of the events,
regardless
> whether grid_stat is used to compute fractional coverage fields? I
am
> basically doing that right now in some Python code I developed when
> calculating Fractions Skill Score.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 14, 2018 3:20 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I'm working on the interp.field option we discussed.
>
> I have a quick question for you. The NBRCNT line type that is
computed by
> grid_stat contains columns for F_RATE and O_RATE. That is the ratio
of
> grid points over a masking region at which the "event" is occurring
in the
> forecast and observation fields respectively.
>
> When grid_stat computes the fractional coverage field, we can
compute
> that. But if grid_stat does *NOT* compute the fractional coverage
fields,
> then we don't know what those ratio's are.
>
> - If interp.field = FCST, then O_RATE = NA.
> - If interp.field = OBS, then F_RATE = NA.
> - If interp.field = NONE, then F_RATE = O_RATE = NA.
>
> Make sense?
>
> Thanks,
> John
>
> On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > That sounds like an excellent plan. Thanks for your assistance
and
> > trying to accommodate and make changes to the MET software so
quickly.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 14, 2018 10:36 AM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > Just wanted to close the loop on this ticket. I'll try to get it
into
> the
> > next release due out next in March.
> >
> > Thanks,
> > John
> >
> > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Yes, thanks for the explanations. That clears up any possible
> > > ambiguity and confusion. I kind of thought that was the
procedure.
> > >
> > > When do you think that the next MET would be released with the
> > > enhancement that we discussed for the neighborhood statistics?
Is it
> > > an easy or difficult update to the software?
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, February 8, 2018 3:24 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > It sounds like you understand how the fractional coverage field
is
> > > computed.
> > >
> > > - Read the raw field... probability values between 0 and 1 in
your
> case.
> > > - Apply the categorical threshold to that data to generate a
field of
> 0's
> > > and 1's... >0.5 in your case.
> > > - For each grid point, replace the value at that grid point with
the
> > > "fraction" of 1's in the neighborhood around that point... e.g.
the
> event
> > > occurs at 4 of the 9 grid points in a 3x3 square surrounding the
> current
> > > point, so fractional coverage = 4/9.
> > > - Compare the forecast fractional coverage values to the
observed
> > > fractional coverage values to compute FSS.
> > >
> > > Yes, I do think it would be a good idea to talk to Craig about
how he
> > > configured MET for his work.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > Getting back to your initial response, I was able to run
grid_stat
> > > > with your changes in the attached configuration file to get
FSS
> output
> > > > (as well as the other statistics for the neighborhood type
results).
> > > >
> > > > However, I do have a question about the forecast categorical
> threshold
> > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > probabilities from the ensemble probabilities: What is the
actual
> > > > formula that MET uses to create the final field to calculate
FSS? Is
> > > > it a strict neighborhood fractional coverage of ensemble
> probabilities
> > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > operation or does the threshold represent something else? I
know in
> > > > Craig's paper, they were calling the last step a neighborhood
average
> > > > of the ensemble probabilities to calculate NEP (since the
event had
> > > > already been defined at the grid point in determining the
ensemble
> > > probability [i.e. likelihood of exceeding
> > > > 0.1" of precipitation]). Maybe, I should understand how
Craig was
> > using
> > > > MET for his paper.
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Hi Chris,
> > > >
> > > > I was thinking about what changes would be needed to compute
FSS on
> > input
> > > > probability data. We could add a “field” option to the
“nbrhd”
> > section,
> > > > just like we have in the “interp” section. It would be set to
FCST,
> > OBS,
> > > > or BOTH (default) and would control the logic as to which
fields
> should
> > > be
> > > > passed through the logic for deriving fractional coverage
fields. In
> > > your
> > > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > > probability values.
> > > >
> > > > Would that logic make sense to you?
> > > >
> > > > Thanks
> > > > John
> > > >
> > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > In fact, SPC's 40-km radius probabilities were the
motivation for
> > > > > adding the shape = circle option to MET.
> > > > >
> > > > > For your first question, no, there is not currently a
Gaussian
> > > > > smoother option for the "interp" section in Grid-Stat.
Currently,
> > the
> > > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > > unweighted mean).
> > > > > There is a distance-weighted mean option for point data
where the
> > > > > weight is 1/distance squared. But that doesn't work for
Grid-Stat
> > > > > because we'd end up dividing by 0.
> > > > >
> > > > > I can see how adding a gaussian smoother would be useful.
> > > > >
> > > > > Here's what we've done in the past with this SPC 40-km
> probabilities:
> > > > >
> > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
> > > > > TRUE;" in the config file).
> > > > > (2) Use the "interp" section to define a smoothing operation
on the
> > > > > observations:
> > > > > interp = {
> > > > > field = OBS;
> > > > > vld_thresh = 1.0;
> > > > > shape = CIRCLE;
> > > > > type = [
> > > > > { method = MAX; width = 11; }
> > > > > ];
> > > > > }
> > > > > Or set width to some circle diameter in grid units that
works
> out
> > > > > to 40km in the real world.
> > > > >
> > > > > In this example, we post-process the observations to make
them
> > > > > comparable the event for which the probability was defined.
> > > > >
> > > > > But I think I understand what you'd like to do... interpret
as the
> > > > > probability value at each grid point as if it was a
fractional
> > > > > coverage value. So don't apply a threshold and neighborhood
size
> to
> > > > > the forecast data at all. Just pass it directly to the
routine
> that
> > > > computes FSS.
> > > > >
> > > > > Unfortunately no, I don't think there's a way of configuring
> > Grid-Stat
> > > > > to do that in met-6.1. If you think this logic would be
generally
> > > > > useful, we could consider adding a flag to the config file
to
> enable
> > > > that logic.
> > > > >
> > > > > Sorry I don't have better answers for you.
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >>
> > > > >> Hi John,
> > > > >>
> > > > >> Your suggestion for accomplishing #2 sounds promising. I
need to
> > > think
> > > > >> the details through some. On a somewhat related note, is
there a
> > > > Gaussian
> > > > >> smoother flag for smoothing the probabilistic fields? I
know
> that
> > > was
> > > > >> mentioned in Craig's paper and I have pursued that option
in the
> > past
> > > > >> (but just not using MET).
> > > > >>
> > > > >> I guess my basic question is can you apply a neighborhood
width
> that
> > > > >> just includes the grid point maybe because you already have
> > > > >> inherently a probabilistic field with spatial uncertainty?
Would
> > that
> > > > be a width of 1
> > > > >> or does that also include the nearest grid point? A good
> example
> > of
> > > > this
> > > > >> would be a probabilistic forecast issued from the Storm
Prediction
> > > > >> Center where the probabilities implicitly assume a 40-km
radius of
> > > > >> influence (so there is no need to compute a fractional
coverage of
> > the
> > > > event).
> > > > >>
> > > > >> Thanks again,
> > > > >> Chris
> > > > >>
> > > > >>
> > > > >> //SIGNED//
> > > > >>
> > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >>
> > > > >>
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > >> christopher.melick at us.af.mil>
> > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC 16
> > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > >> grid_stat on Neighborhood statistics
> > > > >>
> > > > >> Chris,
> > > > >>
> > > > >> (1) The default shape is a square since that's what MET
previously
> > > did.
> > > > >> The circle is the *new* option.
> > > > >>
> > > > >> (2) For this one, the short answer is yes. FSS is computed
in MET
> > by
> > > > >> doing the following...
> > > > >> - For each field separately (i.e. fcst and obs), apply
the
> > > > >> threshold
> > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > >> - Apply the neighborhood width and shape to compute a
fractional
> > > > >> coverage value for each grid point.
> > > > >> - Compare the fcst and obs fractional coverage fields to
compute
> > > FSS.
> > > > >>
> > > > >> But there are some other options that you may find useful.
The
> > > "interp"
> > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > operation.
> > > > >> After reading the raw input data, you can specify if/how to
smooth
> > > > >> that data prior to computing stats. For example, since you
> > mentioned
> > > > >> maximums, you could replace the value at each grid point
with the
> > > > >> maximum of the values in a 5x5 box surrounding that point:
> > > > >>
> > > > >> interp = {
> > > > >> field = FCST;
> > > > >> vld_thresh = 1.0;
> > > > >> shape = SQUARE;
> > > > >>
> > > > >> type = [
> > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > points
> > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
> > > > >> closest points
> > > > >> ];
> > > > >> }
> > > > >>
> > > > >> Using the example above, you'll get 3 times the amount of
output
> > from
> > > > MET.
> > > > >> The first smoother really is no smoothing at all. It would
run on
> > > > >> the raw input data. The second smoother replaces the value
at
> each
> > > > >> grid point with the maximum of the 25 surrounding points.
The
> third
> > > > >> smoother replaces the value at each grid point with the
mean of
> > those
> > > > 25.
> > > > >>
> > > > >> I worked with Craig Schwartz when he was running MET for
that
> paper.
> > > > >> My recollection is that he was able to use the
configuration
> options
> > > > >> of Grid-Stat to accomplish the types of processing he was
after.
> > But
> > > > >> I don't recall all the details.
> > > > >>
> > > > >> If you explain exactly what you're trying to do, it may be
> possible
> > > > >> to configure Grid-Stat to accomplish that.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil via
> > RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >> >
> > > > >> > Hi John,
> > > > >> >
> > > > >> > Your explanation and suggestions are very helpful and do
explain
> > my
> > > > >> > problems quite substantially. I will have to give it a
try to
> see
> > > > >> > what happens.
> > > > >> >
> > > > >> > I do have a couple of other minor questions dealing with
how the
> > > > >> > neighborhood verification is performed:
> > > > >> > 1.) I noticed in the documentation that there is a flag
for
> > > > >> > defining the shape of the neighborhood area (i.e.
circular or
> > > > >> > square). What is the default if you don't specify?
> > > > >> > 2.) In the literature, there are a couple of different
ways to
> > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz and
> > > > >> > Sobash 2017;
> > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1).
> > I
> > > > >> > was curious is the only way to get the Fractions Skill
Score
> (FSS)
> > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > >> > fractional coverage of the threshold event? I have
previously
> > > > >> > explored computing the neighborhood maximum value within
a
> radius
> > of
> > > > each grid point before calculating a probability.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Chris
> > > > >> >
> > > > >> >
> > > > >> > //SIGNED//
> > > > >> >
> > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> Weather
> > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > -----Original Message-----
> > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > >> > christopher.melick at us.af.mil>
> > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > >> matthew.dawson.8 at us.af.mil>;
> > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil>
> > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > >> > grid_stat on Neighborhood statistics
> > > > >> >
> > > > >> > Hi Chris,
> > > > >> >
> > > > >> > I see that you're having difficulty computing
neighborhood
> > > > >> > statistics in Grid-Stat.
> > > > >> >
> > > > >> > First, Grid-Stat definitely does have the ability to
threshold
> the
> > > > >> StageIV
> > > > >> > data on the fly. So pre-processing to 0's and 1's first
is not
> > > > needed.
> > > > >> > Instead, in the "obs" dictionary of the config file,
you'd just
> > set:
> > > > >> > cat_thresh = [ >=2.54 ];
> > > > >> >
> > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
> > >=0.1
> > > > >> inches.
> > > > >> >
> > > > >> > If for some reason you have data that's already 0's and
1's,
> you'd
> > > > >> > just use a different threshold in the config file:
> > > > >> > cat_thresh = [ ==1 ];
> > > > >> >
> > > > >> > This tells MET that anywhere you see a 1 in the data, the
event
> is
> > > > >> > occurring. Otherwise, it's not.
> > > > >> >
> > > > >> > Now on the to real issue. Looking at your config file, I
see
> that
> > > > >> you're
> > > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > > >> > computes
> > > > >> the
> > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's why
> > > > >> > you're getting no output in the NBRCNT line type, which
is the
> one
> > > > >> > that
> > > > >> contains
> > > > >> > Fractions Skill Score (FSS).
> > > > >> >
> > > > >> > In the attached config file, I've made a few changes:
> > > > >> > - Set the obtype to indicate that you're verifying
against
> > StageIV
> > > > >> > - Defined 2 fcst.field entries.
> > > > >> > - In the first we process QP010 as a probability
field.
> > Here
> > > > >> > cat_thresh is set to ==0.10 which is shorthand notation
for
> > > > >> > defining
> > > > >> bins
> > > > >> > for probabilistic verification from 0 to 1 of width 0.1.
> > > > >> > - In the second we process it as a field of
scalars. Here
> > > > >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> > > > >> > I'm
> > > > >> using
> > > > >> > for defining events for fractions skill score.
> > > > >> > - Defined 2 corresponding obs.field entries to verify
the
> > forecast
> > > > >> data.
> > > > >> > In both the verifying threshold is >=2.54. This assumes
that you
> > > > >> > pass
> > > > >> the
> > > > >> > raw StageIV data as input.
> > > > >> > - In the output_flag section, I turned off the line
types that
> > > > >> > don't apply.
> > > > >> >
> > > > >> > Lastly, while setting "level = [ "R19" ];" works as a way
of
> > > > >> > grabbing record number 19 from the input file, it's not
> > > > >> > recommended. Instead,
> > > > >> it'd
> > > > >> > be better to figure out how to set the level string so
that MET
> > > > >> > always grabs the same record, regardless if it's in spot
number
> 19
> > > > >> > or 20 or any other location in the GRIB file.
> > > > >> >
> > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > John Halley Gotway
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
> via
> > RT
> > > > >> > < met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > > >> > > Queue: met_help
> > > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > > >> > > Owner: Nobody
> > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > >> > > Status: new
> > > > >> > > Ticket <URL:
> > > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Hi John,
> > > > >> > >
> > > > >> > > I am trying to run grid_stat in MET to calculate
Neighborhood
> > > > >> > > statistics for probability of precipitation from some
ensemble
> > > > >> > > data with the observations
> > > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting
> no
> > > > output
> > > > >> > > (just
> > > > >> > > headers) in many of the ASCII files from the program
when I
> run
> > > it.
> > > > >> > Could
> > > > >> > > you help me figure out what I am doing wrong? I have
provided
> > > > >> > > the command line specifics I use as well as the output
log
> file
> > > > >> > > that is
> > > > >> > created when
> > > > >> > > running that command. I also have attached the
configuration
> > > file.
> > > > >> I
> > > > >> > am
> > > > >> > > having trouble using ftp to send the raw data from any
of the
> > > > clients
> > > > >> > > available to us. I believe this restriction occurred
> recently
> > > > since
> > > > >> Bob
> > > > >> > > Craig had mentioned that he had been able to connect
> externally
> > > > >> > > in the past.
> > > > >> > >
> > > > >> > > I should mention that I have pre-processed the Stage IV
> analyses
> > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> since
> > > > >> > > I was unclear as to whether MET was able to do that on
the fly
> > > > >> > > when running
> > > > >> > grid_stat.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Chris
> > > > >> > >
> > > > >> > > Python command:
> > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > >> > > 'precip/post-processed-obs.nc', \
> > > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp',
'-v',
> '5]
> > > > >> > > Call(metcall)
> > > > >> > >
> > > > >> > > //SIGNED//
> > > > >> > >
> > > > >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
> > WS/WXEV)
> > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > > >> > > 294-3693
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 21 13:59:28 2018
Hi Chris,
Tressa sent an email on this issue, but she didn't include
met_help at ucar.edu.
So I don't think you received it. I've cut-and-pasted it below:
John and Chris,
It is not generally true that the average of the coverage values is
the
base rate. I think it may sometimes be true, if there are no events in
edge
neighborhoods without full coverage.
I worked up the following example with a 4x4 neighborhood:
1 0 1 1
0 0 0 0
1 0 1 1
0 1 0 0
The rate here is 7/16 = .4375
Fractional coverage values for a 3x3 neighborhood are:
1/4 2/6 2/6 2/4
2/6 4/9 4/9 4/6
2/6 3/9 3/9 2/6
2/4 3/6 3/6 2/4
The average of these values is 0.4149306
I suppose if each fraction had the same denominator (e.g. no edges, or
all
non events there) then we would add each event up for 9 different
neighborhoods and then divide by 9. In that case, it should work.
Tressa
On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> I don't get negative values for AFSS but I do get values that are
less
> than what I calculate for FSS. I approach the procedure for
computing all
> the metrics such that everything is on the same "playing ground".
Thus,
> F_RATE and O_RATE are only computed by the mask of defined grid
points on
> *BOTH* the observation and forecast grids. When I process the
sample time
> period I sent you, F_RATE >> O_RATE which causes AFSS to be a small
value.
> Is that not the approach that I should follow? Were you able to
examine
> my script and raw data file I sent you?
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 20, 2018 8:20 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Your email got me thinking. Thinking through some toy examples in
my
> mind, I can see that probably yes, the mean of the fractional
coverage field
> *should* be the same as the frequency of the event.
>
> I ran MET in the debugger and was able to stop at the point in the
logic
> where this is computed. At that point, we're passing around arrays
of
> fractional coverage field values as well as arrays of 0's and 1's
> indicating whether the event did or did not occur at each grid
point.
> Comparing the mean of these arrays, I find that they are close but
not
> identical. Sometimes one is larger, sometimes the other.
>
> I can think of 2 reasons why they wouldn't be the same, but these
may not
> fully explain it:
>
> (1) In Grid-Stat, the fractional coverage field is computed once
over the
> full verification domain. Then, the masking regions are processed
as
> subsets of these grid points. For a grid point that falls just
barely
> inside that mask, around 1/2 of the grid points used to compute the
> fractional coverage value came from *OUTSIDE* that masking region.
The 0/1
> arrays are only for points inside the current mask. So the mean of
the 0/1
> array represents only the points inside the mask while the mean of
the
> fractional coverage includes info from outside the mask.
>
> (2) When data values are missing in the forecast or observation
fields,
> depending on how the user has set the vld_thresh config file option,
the
> "denominator" used to compute the fractional coverage field value
could
> change from grid point to grid point. That would likely throw off
the mean
> of the fractional coverage field.
>
> Now what is the *CORRECT* way of computing the forecast and
observation
> rates? In papers about this, I see references to the "base_rate"
but no
> explicit equation.
>
> One very nice feature about the way MET is currently doing it is
that
> F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> Therefore, UFSS and AFSS remain constant as well. Using the other
method,
> O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood
size
> since the number of points included from outside the mask would
change.
>
> You mentioned that you had issues with negative UFSS and AFSS
values.
> Perhaps it's related to how you're computing the F_RATE and O_RATE?
>
> Lots of little details to consider here!
>
> Thanks,
> John
>
>
> On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > can't be calculated in the cases below? Can't the gridded data
still
> > be read in to determine the relative frequency of the events,
regardless
> > whether grid_stat is used to compute fractional coverage fields?
I am
> > basically doing that right now in some Python code I developed
when
> > calculating Fractions Skill Score.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 14, 2018 3:20 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I'm working on the interp.field option we discussed.
> >
> > I have a quick question for you. The NBRCNT line type that is
computed
> by
> > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
> > grid points over a masking region at which the "event" is
occurring in
> the
> > forecast and observation fields respectively.
> >
> > When grid_stat computes the fractional coverage field, we can
compute
> > that. But if grid_stat does *NOT* compute the fractional coverage
> fields,
> > then we don't know what those ratio's are.
> >
> > - If interp.field = FCST, then O_RATE = NA.
> > - If interp.field = OBS, then F_RATE = NA.
> > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > John,
> > >
> > > That sounds like an excellent plan. Thanks for your assistance
and
> > > trying to accommodate and make changes to the MET software so
quickly.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Just wanted to close the loop on this ticket. I'll try to get
it into
> > the
> > > next release due out next in March.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > Yes, thanks for the explanations. That clears up any possible
> > > > ambiguity and confusion. I kind of thought that was the
procedure.
> > > >
> > > > When do you think that the next MET would be released with the
> > > > enhancement that we discussed for the neighborhood statistics?
Is it
> > > > an easy or difficult update to the software?
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > It sounds like you understand how the fractional coverage
field is
> > > > computed.
> > > >
> > > > - Read the raw field... probability values between 0 and 1 in
your
> > case.
> > > > - Apply the categorical threshold to that data to generate a
field of
> > 0's
> > > > and 1's... >0.5 in your case.
> > > > - For each grid point, replace the value at that grid point
with the
> > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
> > event
> > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
> > current
> > > > point, so fractional coverage = 4/9.
> > > > - Compare the forecast fractional coverage values to the
observed
> > > > fractional coverage values to compute FSS.
> > > >
> > > > Yes, I do think it would be a good idea to talk to Craig about
how he
> > > > configured MET for his work.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Getting back to your initial response, I was able to run
grid_stat
> > > > > with your changes in the attached configuration file to get
FSS
> > output
> > > > > (as well as the other statistics for the neighborhood type
> results).
> > > > >
> > > > > However, I do have a question about the forecast categorical
> > threshold
> > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > > probabilities from the ensemble probabilities: What is the
actual
> > > > > formula that MET uses to create the final field to calculate
FSS?
> Is
> > > > > it a strict neighborhood fractional coverage of ensemble
> > probabilities
> > > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > > operation or does the threshold represent something else? I
know
> in
> > > > > Craig's paper, they were calling the last step a
neighborhood
> average
> > > > > of the ensemble probabilities to calculate NEP (since the
event had
> > > > > already been defined at the grid point in determining the
ensemble
> > > > probability [i.e. likelihood of exceeding
> > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig was
> > > using
> > > > > MET for his paper.
> > > > >
> > > > > Thanks again,
> > > > > Chris
> > > > >
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > I was thinking about what changes would be needed to compute
FSS on
> > > input
> > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > section,
> > > > > just like we have in the “interp” section. It would be set
to
> FCST,
> > > OBS,
> > > > > or BOTH (default) and would control the logic as to which
fields
> > should
> > > > be
> > > > > passed through the logic for deriving fractional coverage
fields.
> In
> > > > your
> > > > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > > > probability values.
> > > > >
> > > > > Would that logic make sense to you?
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > > > Chris,
> > > > > >
> > > > > > In fact, SPC's 40-km radius probabilities were the
motivation for
> > > > > > adding the shape = circle option to MET.
> > > > > >
> > > > > > For your first question, no, there is not currently a
Gaussian
> > > > > > smoother option for the "interp" section in Grid-Stat.
> Currently,
> > > the
> > > > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > > > unweighted mean).
> > > > > > There is a distance-weighted mean option for point data
where the
> > > > > > weight is 1/distance squared. But that doesn't work for
> Grid-Stat
> > > > > > because we'd end up dividing by 0.
> > > > > >
> > > > > > I can see how adding a gaussian smoother would be useful.
> > > > > >
> > > > > > Here's what we've done in the past with this SPC 40-km
> > probabilities:
> > > > > >
> > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
> > > > > > TRUE;" in the config file).
> > > > > > (2) Use the "interp" section to define a smoothing
operation on
> the
> > > > > > observations:
> > > > > > interp = {
> > > > > > field = OBS;
> > > > > > vld_thresh = 1.0;
> > > > > > shape = CIRCLE;
> > > > > > type = [
> > > > > > { method = MAX; width = 11; }
> > > > > > ];
> > > > > > }
> > > > > > Or set width to some circle diameter in grid units that
works
> > out
> > > > > > to 40km in the real world.
> > > > > >
> > > > > > In this example, we post-process the observations to make
them
> > > > > > comparable the event for which the probability was
defined.
> > > > > >
> > > > > > But I think I understand what you'd like to do...
interpret as
> the
> > > > > > probability value at each grid point as if it was a
fractional
> > > > > > coverage value. So don't apply a threshold and
neighborhood size
> > to
> > > > > > the forecast data at all. Just pass it directly to the
routine
> > that
> > > > > computes FSS.
> > > > > >
> > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > Grid-Stat
> > > > > > to do that in met-6.1. If you think this logic would be
> generally
> > > > > > useful, we could consider adding a flag to the config file
to
> > enable
> > > > > that logic.
> > > > > >
> > > > > > Sorry I don't have better answers for you.
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >>
> > > > > >> Hi John,
> > > > > >>
> > > > > >> Your suggestion for accomplishing #2 sounds promising. I
need
> to
> > > > think
> > > > > >> the details through some. On a somewhat related note,
is
> there a
> > > > > Gaussian
> > > > > >> smoother flag for smoothing the probabilistic fields? I
know
> > that
> > > > was
> > > > > >> mentioned in Craig's paper and I have pursued that option
in the
> > > past
> > > > > >> (but just not using MET).
> > > > > >>
> > > > > >> I guess my basic question is can you apply a neighborhood
width
> > that
> > > > > >> just includes the grid point maybe because you already
have
> > > > > >> inherently a probabilistic field with spatial
uncertainty?
> Would
> > > that
> > > > > be a width of 1
> > > > > >> or does that also include the nearest grid point? A
good
> > example
> > > of
> > > > > this
> > > > > >> would be a probabilistic forecast issued from the Storm
> Prediction
> > > > > >> Center where the probabilities implicitly assume a 40-km
radius
> of
> > > > > >> influence (so there is no need to compute a fractional
coverage
> of
> > > the
> > > > > event).
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> Chris
> > > > > >>
> > > > > >>
> > > > > >> //SIGNED//
> > > > > >>
> > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> Weather
> > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> christopher.melick at us.af.mil>
> > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC 16
> > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986] Help
> > > with
> > > > > >> grid_stat on Neighborhood statistics
> > > > > >>
> > > > > >> Chris,
> > > > > >>
> > > > > >> (1) The default shape is a square since that's what MET
> previously
> > > > did.
> > > > > >> The circle is the *new* option.
> > > > > >>
> > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
> MET
> > > by
> > > > > >> doing the following...
> > > > > >> - For each field separately (i.e. fcst and obs), apply
the
> > > > > >> threshold
> > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > > >> - Apply the neighborhood width and shape to compute a
> fractional
> > > > > >> coverage value for each grid point.
> > > > > >> - Compare the fcst and obs fractional coverage fields
to
> compute
> > > > FSS.
> > > > > >>
> > > > > >> But there are some other options that you may find
useful. The
> > > > "interp"
> > > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > > operation.
> > > > > >> After reading the raw input data, you can specify if/how
to
> smooth
> > > > > >> that data prior to computing stats. For example, since
you
> > > mentioned
> > > > > >> maximums, you could replace the value at each grid point
with
> the
> > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > >>
> > > > > >> interp = {
> > > > > >> field = FCST;
> > > > > >> vld_thresh = 1.0;
> > > > > >> shape = SQUARE;
> > > > > >>
> > > > > >> type = [
> > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > > points
> > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
> > > > > >> closest points
> > > > > >> ];
> > > > > >> }
> > > > > >>
> > > > > >> Using the example above, you'll get 3 times the amount of
output
> > > from
> > > > > MET.
> > > > > >> The first smoother really is no smoothing at all. It
would run
> on
> > > > > >> the raw input data. The second smoother replaces the
value at
> > each
> > > > > >> grid point with the maximum of the 25 surrounding points.
The
> > third
> > > > > >> smoother replaces the value at each grid point with the
mean of
> > > those
> > > > > 25.
> > > > > >>
> > > > > >> I worked with Craig Schwartz when he was running MET for
that
> > paper.
> > > > > >> My recollection is that he was able to use the
configuration
> > options
> > > > > >> of Grid-Stat to accomplish the types of processing he was
after.
> > > But
> > > > > >> I don't recall all the details.
> > > > > >>
> > > > > >> If you explain exactly what you're trying to do, it may
be
> > possible
> > > > > >> to configure Grid-Stat to accomplish that.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
> via
> > > RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > >> >
> > > > > >> > Hi John,
> > > > > >> >
> > > > > >> > Your explanation and suggestions are very helpful and
do
> explain
> > > my
> > > > > >> > problems quite substantially. I will have to give it a
try to
> > see
> > > > > >> > what happens.
> > > > > >> >
> > > > > >> > I do have a couple of other minor questions dealing
with how
> the
> > > > > >> > neighborhood verification is performed:
> > > > > >> > 1.) I noticed in the documentation that there is a
flag for
> > > > > >> > defining the shape of the neighborhood area (i.e.
circular or
> > > > > >> > square). What is the default if you don't specify?
> > > > > >> > 2.) In the literature, there are a couple of different
ways
> to
> > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
> and
> > > > > >> > Sobash 2017;
> > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1
> ).
> > > I
> > > > > >> > was curious is the only way to get the Fractions Skill
Score
> > (FSS)
> > > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > > >> > fractional coverage of the threshold event? I have
previously
> > > > > >> > explored computing the neighborhood maximum value
within a
> > radius
> > > of
> > > > > each grid point before calculating a probability.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Chris
> > > > > >> >
> > > > > >> >
> > > > > >> > //SIGNED//
> > > > > >> >
> > > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > -----Original Message-----
> > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> > christopher.melick at us.af.mil>
> > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > robert.craig.2 at us.af.mil>
> > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > >> > grid_stat on Neighborhood statistics
> > > > > >> >
> > > > > >> > Hi Chris,
> > > > > >> >
> > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > >> > statistics in Grid-Stat.
> > > > > >> >
> > > > > >> > First, Grid-Stat definitely does have the ability to
threshold
> > the
> > > > > >> StageIV
> > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
> not
> > > > > needed.
> > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
> just
> > > set:
> > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > >> >
> > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
> > > >=0.1
> > > > > >> inches.
> > > > > >> >
> > > > > >> > If for some reason you have data that's already 0's and
1's,
> > you'd
> > > > > >> > just use a different threshold in the config file:
> > > > > >> > cat_thresh = [ ==1 ];
> > > > > >> >
> > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
> event
> > is
> > > > > >> > occurring. Otherwise, it's not.
> > > > > >> >
> > > > > >> > Now on the to real issue. Looking at your config file,
I see
> > that
> > > > > >> you're
> > > > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > > > >> > computes
> > > > > >> the
> > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
> why
> > > > > >> > you're getting no output in the NBRCNT line type, which
is the
> > one
> > > > > >> > that
> > > > > >> contains
> > > > > >> > Fractions Skill Score (FSS).
> > > > > >> >
> > > > > >> > In the attached config file, I've made a few changes:
> > > > > >> > - Set the obtype to indicate that you're verifying
against
> > > StageIV
> > > > > >> > - Defined 2 fcst.field entries.
> > > > > >> > - In the first we process QP010 as a probability
field.
> > > Here
> > > > > >> > cat_thresh is set to ==0.10 which is shorthand notation
for
> > > > > >> > defining
> > > > > >> bins
> > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > >> > - In the second we process it as a field of
scalars.
> Here
> > > > > >> > cat_thresh is set to >=0.5. So that's the probability
> threshold
> > > > > >> > I'm
> > > > > >> using
> > > > > >> > for defining events for fractions skill score.
> > > > > >> > - Defined 2 corresponding obs.field entries to verify
the
> > > forecast
> > > > > >> data.
> > > > > >> > In both the verifying threshold is >=2.54. This assumes
that
> you
> > > > > >> > pass
> > > > > >> the
> > > > > >> > raw StageIV data as input.
> > > > > >> > - In the output_flag section, I turned off the line
types
> that
> > > > > >> > don't apply.
> > > > > >> >
> > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
> > > > > >> > grabbing record number 19 from the input file, it's not
> > > > > >> > recommended. Instead,
> > > > > >> it'd
> > > > > >> > be better to figure out how to set the level string so
that
> MET
> > > > > >> > always grabs the same record, regardless if it's in
spot
> number
> > 19
> > > > > >> > or 20 or any other location in the GRIB file.
> > > > > >> >
> > > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > John Halley Gotway
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
> > via
> > > RT
> > > > > >> > < met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > > > >> > > Queue: met_help
> > > > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > > > >> > > Owner: Nobody
> > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > >> > > Status: new
> > > > > >> > > Ticket <URL:
> > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Hi John,
> > > > > >> > >
> > > > > >> > > I am trying to run grid_stat in MET to calculate
> Neighborhood
> > > > > >> > > statistics for probability of precipitation from some
> ensemble
> > > > > >> > > data with the observations
> > > > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting
> > no
> > > > > output
> > > > > >> > > (just
> > > > > >> > > headers) in many of the ASCII files from the program
when I
> > run
> > > > it.
> > > > > >> > Could
> > > > > >> > > you help me figure out what I am doing wrong? I have
> provided
> > > > > >> > > the command line specifics I use as well as the
output log
> > file
> > > > > >> > > that is
> > > > > >> > created when
> > > > > >> > > running that command. I also have attached the
> configuration
> > > > file.
> > > > > >> I
> > > > > >> > am
> > > > > >> > > having trouble using ftp to send the raw data from
any of
> the
> > > > > clients
> > > > > >> > > available to us. I believe this restriction
occurred
> > recently
> > > > > since
> > > > > >> Bob
> > > > > >> > > Craig had mentioned that he had been able to connect
> > externally
> > > > > >> > > in the past.
> > > > > >> > >
> > > > > >> > > I should mention that I have pre-processed the Stage
IV
> > analyses
> > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> > since
> > > > > >> > > I was unclear as to whether MET was able to do that
on the
> fly
> > > > > >> > > when running
> > > > > >> > grid_stat.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Chris
> > > > > >> > >
> > > > > >> > > Python command:
> > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp',
'-v',
> > '5]
> > > > > >> > > Call(metcall)
> > > > > >> > >
> > > > > >> > > //SIGNED//
> > > > > >> > >
> > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
> > > WS/WXEV)
> > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > > > >> > > 294-3693
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 21 14:18:08 2018
So based on this info, I think that MET is doing what we want...
computing
the F_RATE and O_RATE as the fraction of events within the masking
region.
It should not be computing the mean of the fractional coverage values.
However, that does mean that the NetCDF matched pairs file you sent us
does
not have sufficient information for recomputing AFSS and UFSS. You'd
need
to know the number of events actually occurring within the masking
regions.
John
On Wed, Feb 21, 2018 at 1:59 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Chris,
>
> Tressa sent an email on this issue, but she didn't include
> met_help at ucar.edu. So I don't think you received it. I've
> cut-and-pasted it below:
>
> John and Chris,
>
> It is not generally true that the average of the coverage values is
the
> base rate. I think it may sometimes be true, if there are no events
in edge
> neighborhoods without full coverage.
>
> I worked up the following example with a 4x4 neighborhood:
>
> 1 0 1 1
> 0 0 0 0
> 1 0 1 1
> 0 1 0 0
>
> The rate here is 7/16 = .4375
>
> Fractional coverage values for a 3x3 neighborhood are:
>
> 1/4 2/6 2/6 2/4
> 2/6 4/9 4/9 4/6
> 2/6 3/9 3/9 2/6
> 2/4 3/6 3/6 2/4
>
> The average of these values is 0.4149306
>
> I suppose if each fraction had the same denominator (e.g. no edges,
or all
> non events there) then we would add each event up for 9 different
> neighborhoods and then divide by 9. In that case, it should work.
>
> Tressa
>
>
> On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> I don't get negative values for AFSS but I do get values that are
less
>> than what I calculate for FSS. I approach the procedure for
computing all
>> the metrics such that everything is on the same "playing ground".
Thus,
>> F_RATE and O_RATE are only computed by the mask of defined grid
points on
>> *BOTH* the observation and forecast grids. When I process the
sample time
>> period I sent you, F_RATE >> O_RATE which causes AFSS to be a small
value.
>> Is that not the approach that I should follow? Were you able to
examine
>> my script and raw data file I sent you?
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian
>> Meteorologist, Verification Exploitation Team
>> 16th Weather Squadron (16 WS/WXEV)
>> 557th Weather Wing, Offutt AFB, NE
>> DSN 271-3693; Comm (402) 294-3693
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Tuesday, February 20, 2018 8:20 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
>> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> Your email got me thinking. Thinking through some toy examples in
my
>> mind, I can see that probably yes, the mean of the fractional
coverage field
>> *should* be the same as the frequency of the event.
>>
>> I ran MET in the debugger and was able to stop at the point in the
logic
>> where this is computed. At that point, we're passing around arrays
of
>> fractional coverage field values as well as arrays of 0's and 1's
>> indicating whether the event did or did not occur at each grid
point.
>> Comparing the mean of these arrays, I find that they are close but
not
>> identical. Sometimes one is larger, sometimes the other.
>>
>> I can think of 2 reasons why they wouldn't be the same, but these
may not
>> fully explain it:
>>
>> (1) In Grid-Stat, the fractional coverage field is computed once
over the
>> full verification domain. Then, the masking regions are processed
as
>> subsets of these grid points. For a grid point that falls just
barely
>> inside that mask, around 1/2 of the grid points used to compute the
>> fractional coverage value came from *OUTSIDE* that masking region.
The 0/1
>> arrays are only for points inside the current mask. So the mean of
the 0/1
>> array represents only the points inside the mask while the mean of
the
>> fractional coverage includes info from outside the mask.
>>
>> (2) When data values are missing in the forecast or observation
fields,
>> depending on how the user has set the vld_thresh config file
option, the
>> "denominator" used to compute the fractional coverage field value
could
>> change from grid point to grid point. That would likely throw off
the mean
>> of the fractional coverage field.
>>
>> Now what is the *CORRECT* way of computing the forecast and
observation
>> rates? In papers about this, I see references to the "base_rate"
but no
>> explicit equation.
>>
>> One very nice feature about the way MET is currently doing it is
that
>> F_RATE and O_RATE remain constant regardless of the neighborhood
size.
>> Therefore, UFSS and AFSS remain constant as well. Using the other
>> method, O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood
>> size since the number of points included from outside the mask
would change.
>>
>> You mentioned that you had issues with negative UFSS and AFSS
values.
>> Perhaps it's related to how you're computing the F_RATE and O_RATE?
>>
>> Lots of little details to consider here!
>>
>> Thanks,
>> John
>>
>>
>> On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > John,
>> >
>> > After thinking about it some, I was curious why the F_RATE and
O_RATE
>> > can't be calculated in the cases below? Can't the gridded data
still
>> > be read in to determine the relative frequency of the events,
regardless
>> > whether grid_stat is used to compute fractional coverage fields?
I am
>> > basically doing that right now in some Python code I developed
when
>> > calculating Fractions Skill Score.
>> >
>> > Thanks,
>> > Chris
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 14, 2018 3:20 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I'm working on the interp.field option we discussed.
>> >
>> > I have a quick question for you. The NBRCNT line type that is
>> computed by
>> > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
>> > grid points over a masking region at which the "event" is
occurring in
>> the
>> > forecast and observation fields respectively.
>> >
>> > When grid_stat computes the fractional coverage field, we can
compute
>> > that. But if grid_stat does *NOT* compute the fractional
coverage
>> fields,
>> > then we don't know what those ratio's are.
>> >
>> > - If interp.field = FCST, then O_RATE = NA.
>> > - If interp.field = OBS, then F_RATE = NA.
>> > - If interp.field = NONE, then F_RATE = O_RATE = NA.
>> >
>> > Make sense?
>> >
>> > Thanks,
>> > John
>> >
>> > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> > >
>> > > John,
>> > >
>> > > That sounds like an excellent plan. Thanks for your assistance
and
>> > > trying to accommodate and make changes to the MET software so
quickly.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > > Sent: Wednesday, February 14, 2018 10:36 AM
>> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > > christopher.melick at us.af.mil>
>> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> > matthew.dawson.8 at us.af.mil>;
>> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> > > grid_stat on Neighborhood statistics
>> > >
>> > > Chris,
>> > >
>> > > Just wanted to close the loop on this ticket. I'll try to get
it into
>> > the
>> > > next release due out next in March.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Yes, thanks for the explanations. That clears up any
possible
>> > > > ambiguity and confusion. I kind of thought that was the
procedure.
>> > > >
>> > > > When do you think that the next MET would be released with
the
>> > > > enhancement that we discussed for the neighborhood
statistics? Is
>> it
>> > > > an easy or difficult update to the software?
>> > > >
>> > > > Thanks again,
>> > > > Chris
>> > > >
>> > > >
>> > > > //SIGNED//
>> > > >
>> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > > > Sent: Thursday, February 8, 2018 3:24 PM
>> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > > > christopher.melick at us.af.mil>
>> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> > > matthew.dawson.8 at us.af.mil>;
>> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
>> with
>> > > > grid_stat on Neighborhood statistics
>> > > >
>> > > > Chris,
>> > > >
>> > > > It sounds like you understand how the fractional coverage
field is
>> > > > computed.
>> > > >
>> > > > - Read the raw field... probability values between 0 and 1 in
your
>> > case.
>> > > > - Apply the categorical threshold to that data to generate a
field
>> of
>> > 0's
>> > > > and 1's... >0.5 in your case.
>> > > > - For each grid point, replace the value at that grid point
with the
>> > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
>> > event
>> > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
>> > current
>> > > > point, so fractional coverage = 4/9.
>> > > > - Compare the forecast fractional coverage values to the
observed
>> > > > fractional coverage values to compute FSS.
>> > > >
>> > > > Yes, I do think it would be a good idea to talk to Craig
about how
>> he
>> > > > configured MET for his work.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > >
>> > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via
>> RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> > > > >
>> > > > > Hi John,
>> > > > >
>> > > > > Getting back to your initial response, I was able to run
grid_stat
>> > > > > with your changes in the attached configuration file to get
FSS
>> > output
>> > > > > (as well as the other statistics for the neighborhood type
>> results).
>> > > > >
>> > > > > However, I do have a question about the forecast
categorical
>> > threshold
>> > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
>> > > > > probabilities from the ensemble probabilities: What is the
actual
>> > > > > formula that MET uses to create the final field to
calculate
>> FSS? Is
>> > > > > it a strict neighborhood fractional coverage of ensemble
>> > probabilities
>> > > > > above 0.5 in some set radius of influence? Or is it some
other
>> > > > > operation or does the threshold represent something else?
I know
>> in
>> > > > > Craig's paper, they were calling the last step a
neighborhood
>> average
>> > > > > of the ensemble probabilities to calculate NEP (since the
event
>> had
>> > > > > already been defined at the grid point in determining the
ensemble
>> > > > probability [i.e. likelihood of exceeding
>> > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig
>> was
>> > > using
>> > > > > MET for his paper.
>> > > > >
>> > > > > Thanks again,
>> > > > > Chris
>> > > > >
>> > > > >
>> > > > > //SIGNED//
>> > > > >
>> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> Verification
>> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > > > >
>> > > > >
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > > > > Sent: Wednesday, February 7, 2018 7:14 PM
>> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > > > > christopher.melick at us.af.mil>
>> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> > > > matthew.dawson.8 at us.af.mil>;
>> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
>> robert.craig.2 at us.af.mil>
>> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
>> with
>> > > > > grid_stat on Neighborhood statistics
>> > > > >
>> > > > > Hi Chris,
>> > > > >
>> > > > > I was thinking about what changes would be needed to
compute FSS
>> on
>> > > input
>> > > > > probability data. We could add a “field” option to the
“nbrhd”
>> > > section,
>> > > > > just like we have in the “interp” section. It would be set
to
>> FCST,
>> > > OBS,
>> > > > > or BOTH (default) and would control the logic as to which
fields
>> > should
>> > > > be
>> > > > > passed through the logic for deriving fractional coverage
>> fields. In
>> > > > your
>> > > > > case, you’d set it to OBS and we’d just use the raw input
forecast
>> > > > > probability values.
>> > > > >
>> > > > > Would that logic make sense to you?
>> > > > >
>> > > > > Thanks
>> > > > > John
>> > > > >
>> > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
>> johnhg at ucar.edu>
>> > > > wrote:
>> > > > >
>> > > > > > Chris,
>> > > > > >
>> > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
>> for
>> > > > > > adding the shape = circle option to MET.
>> > > > > >
>> > > > > > For your first question, no, there is not currently a
Gaussian
>> > > > > > smoother option for the "interp" section in Grid-Stat.
>> Currently,
>> > > the
>> > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN (for
>> > > > > unweighted mean).
>> > > > > > There is a distance-weighted mean option for point data
where
>> the
>> > > > > > weight is 1/distance squared. But that doesn't work for
>> Grid-Stat
>> > > > > > because we'd end up dividing by 0.
>> > > > > >
>> > > > > > I can see how adding a gaussian smoother would be useful.
>> > > > > >
>> > > > > > Here's what we've done in the past with this SPC 40-km
>> > probabilities:
>> > > > > >
>> > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
>> > > > > > TRUE;" in the config file).
>> > > > > > (2) Use the "interp" section to define a smoothing
operation on
>> the
>> > > > > > observations:
>> > > > > > interp = {
>> > > > > > field = OBS;
>> > > > > > vld_thresh = 1.0;
>> > > > > > shape = CIRCLE;
>> > > > > > type = [
>> > > > > > { method = MAX; width = 11; }
>> > > > > > ];
>> > > > > > }
>> > > > > > Or set width to some circle diameter in grid units
that works
>> > out
>> > > > > > to 40km in the real world.
>> > > > > >
>> > > > > > In this example, we post-process the observations to make
them
>> > > > > > comparable the event for which the probability was
defined.
>> > > > > >
>> > > > > > But I think I understand what you'd like to do...
interpret as
>> the
>> > > > > > probability value at each grid point as if it was a
fractional
>> > > > > > coverage value. So don't apply a threshold and
neighborhood
>> size
>> > to
>> > > > > > the forecast data at all. Just pass it directly to the
routine
>> > that
>> > > > > computes FSS.
>> > > > > >
>> > > > > > Unfortunately no, I don't think there's a way of
configuring
>> > > Grid-Stat
>> > > > > > to do that in met-6.1. If you think this logic would be
>> generally
>> > > > > > useful, we could consider adding a flag to the config
file to
>> > enable
>> > > > > that logic.
>> > > > > >
>> > > > > > Sorry I don't have better answers for you.
>> > > > > >
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
>> via
>> > RT
>> > > <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > >>
>> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> >
>> > > > > >>
>> > > > > >> Hi John,
>> > > > > >>
>> > > > > >> Your suggestion for accomplishing #2 sounds promising.
I need
>> to
>> > > > think
>> > > > > >> the details through some. On a somewhat related note,
is
>> there a
>> > > > > Gaussian
>> > > > > >> smoother flag for smoothing the probabilistic fields?
I know
>> > that
>> > > > was
>> > > > > >> mentioned in Craig's paper and I have pursued that
option in
>> the
>> > > past
>> > > > > >> (but just not using MET).
>> > > > > >>
>> > > > > >> I guess my basic question is can you apply a
neighborhood width
>> > that
>> > > > > >> just includes the grid point maybe because you already
have
>> > > > > >> inherently a probabilistic field with spatial
uncertainty?
>> Would
>> > > that
>> > > > > be a width of 1
>> > > > > >> or does that also include the nearest grid point? A
good
>> > example
>> > > of
>> > > > > this
>> > > > > >> would be a probabilistic forecast issued from the Storm
>> Prediction
>> > > > > >> Center where the probabilities implicitly assume a 40-km
>> radius of
>> > > > > >> influence (so there is no need to compute a fractional
>> coverage of
>> > > the
>> > > > > event).
>> > > > > >>
>> > > > > >> Thanks again,
>> > > > > >> Chris
>> > > > > >>
>> > > > > >>
>> > > > > >> //SIGNED//
>> > > > > >>
>> > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > Verification
>> > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
>> Weather
>> > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> -----Original Message-----
>> > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
>> > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
>> > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > > > > >> christopher.melick at us.af.mil>
>> > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
>> > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC
>> 16
>> > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
>> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
>> Help
>> > > with
>> > > > > >> grid_stat on Neighborhood statistics
>> > > > > >>
>> > > > > >> Chris,
>> > > > > >>
>> > > > > >> (1) The default shape is a square since that's what MET
>> previously
>> > > > did.
>> > > > > >> The circle is the *new* option.
>> > > > > >>
>> > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
>> MET
>> > > by
>> > > > > >> doing the following...
>> > > > > >> - For each field separately (i.e. fcst and obs), apply
the
>> > > > > >> threshold
>> > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> > > > > >> - Apply the neighborhood width and shape to compute a
>> fractional
>> > > > > >> coverage value for each grid point.
>> > > > > >> - Compare the fcst and obs fractional coverage fields
to
>> compute
>> > > > FSS.
>> > > > > >>
>> > > > > >> But there are some other options that you may find
useful. The
>> > > > "interp"
>> > > > > >> section in the config file is used in Grid-Stat as a
smoothing
>> > > > > operation.
>> > > > > >> After reading the raw input data, you can specify if/how
to
>> smooth
>> > > > > >> that data prior to computing stats. For example, since
you
>> > > mentioned
>> > > > > >> maximums, you could replace the value at each grid point
with
>> the
>> > > > > >> maximum of the values in a 5x5 box surrounding that
point:
>> > > > > >>
>> > > > > >> interp = {
>> > > > > >> field = FCST;
>> > > > > >> vld_thresh = 1.0;
>> > > > > >> shape = SQUARE;
>> > > > > >>
>> > > > > >> type = [
>> > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
>> > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
>> > > points
>> > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
>> > > > > >> closest points
>> > > > > >> ];
>> > > > > >> }
>> > > > > >>
>> > > > > >> Using the example above, you'll get 3 times the amount
of
>> output
>> > > from
>> > > > > MET.
>> > > > > >> The first smoother really is no smoothing at all. It
would
>> run on
>> > > > > >> the raw input data. The second smoother replaces the
value at
>> > each
>> > > > > >> grid point with the maximum of the 25 surrounding
points. The
>> > third
>> > > > > >> smoother replaces the value at each grid point with the
mean of
>> > > those
>> > > > > 25.
>> > > > > >>
>> > > > > >> I worked with Craig Schwartz when he was running MET for
that
>> > paper.
>> > > > > >> My recollection is that he was able to use the
configuration
>> > options
>> > > > > >> of Grid-Stat to accomplish the types of processing he
was
>> after.
>> > > But
>> > > > > >> I don't recall all the details.
>> > > > > >>
>> > > > > >> If you explain exactly what you're trying to do, it may
be
>> > possible
>> > > > > >> to configure Grid-Stat to accomplish that.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> John
>> > > > > >>
>> > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
>> via
>> > > RT <
>> > > > > >> met_help at ucar.edu> wrote:
>> > > > > >>
>> > > > > >> >
>> > > > > >> > <URL: https://rt.rap.ucar.edu/rt/Tic
>> ket/Display.html?id=83986 >
>> > > > > >> >
>> > > > > >> > Hi John,
>> > > > > >> >
>> > > > > >> > Your explanation and suggestions are very helpful and
do
>> explain
>> > > my
>> > > > > >> > problems quite substantially. I will have to give it
a try
>> to
>> > see
>> > > > > >> > what happens.
>> > > > > >> >
>> > > > > >> > I do have a couple of other minor questions dealing
with how
>> the
>> > > > > >> > neighborhood verification is performed:
>> > > > > >> > 1.) I noticed in the documentation that there is a
flag for
>> > > > > >> > defining the shape of the neighborhood area (i.e.
circular or
>> > > > > >> > square). What is the default if you don't specify?
>> > > > > >> > 2.) In the literature, there are a couple of
different ways
>> to
>> > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
>> and
>> > > > > >> > Sobash 2017;
>> > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-
16-0400.
>> 1).
>> > > I
>> > > > > >> > was curious is the only way to get the Fractions Skill
Score
>> > (FSS)
>> > > > > >> > output is to define a neighborhood in which MET
calculates a
>> > > > > >> > fractional coverage of the threshold event? I have
>> previously
>> > > > > >> > explored computing the neighborhood maximum value
within a
>> > radius
>> > > of
>> > > > > each grid point before calculating a probability.
>> > > > > >> >
>> > > > > >> > Thanks,
>> > > > > >> > Chris
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > //SIGNED//
>> > > > > >> >
>> > > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > > Verification
>> > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
>> > Weather
>> > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > -----Original Message-----
>> > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
>> > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > > > > >> > christopher.melick at us.af.mil>
>> > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> > > > > >> matthew.dawson.8 at us.af.mil>;
>> > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
>> > > robert.craig.2 at us.af.mil>
>> > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
>> > with
>> > > > > >> > grid_stat on Neighborhood statistics
>> > > > > >> >
>> > > > > >> > Hi Chris,
>> > > > > >> >
>> > > > > >> > I see that you're having difficulty computing
neighborhood
>> > > > > >> > statistics in Grid-Stat.
>> > > > > >> >
>> > > > > >> > First, Grid-Stat definitely does have the ability to
>> threshold
>> > the
>> > > > > >> StageIV
>> > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
>> not
>> > > > > needed.
>> > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
>> just
>> > > set:
>> > > > > >> > cat_thresh = [ >=2.54 ];
>> > > > > >> >
>> > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
>> > > >=0.1
>> > > > > >> inches.
>> > > > > >> >
>> > > > > >> > If for some reason you have data that's already 0's
and 1's,
>> > you'd
>> > > > > >> > just use a different threshold in the config file:
>> > > > > >> > cat_thresh = [ ==1 ];
>> > > > > >> >
>> > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
>> event
>> > is
>> > > > > >> > occurring. Otherwise, it's not.
>> > > > > >> >
>> > > > > >> > Now on the to real issue. Looking at your config
file, I see
>> > that
>> > > > > >> you're
>> > > > > >> > processing probabilistic data. When "prob = TRUE",
MET only
>> > > > > >> > computes
>> > > > > >> the
>> > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
>> why
>> > > > > >> > you're getting no output in the NBRCNT line type,
which is
>> the
>> > one
>> > > > > >> > that
>> > > > > >> contains
>> > > > > >> > Fractions Skill Score (FSS).
>> > > > > >> >
>> > > > > >> > In the attached config file, I've made a few changes:
>> > > > > >> > - Set the obtype to indicate that you're verifying
against
>> > > StageIV
>> > > > > >> > - Defined 2 fcst.field entries.
>> > > > > >> > - In the first we process QP010 as a probability
field.
>> > > Here
>> > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation for
>> > > > > >> > defining
>> > > > > >> bins
>> > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
>> > > > > >> > - In the second we process it as a field of
scalars.
>> Here
>> > > > > >> > cat_thresh is set to >=0.5. So that's the probability
>> threshold
>> > > > > >> > I'm
>> > > > > >> using
>> > > > > >> > for defining events for fractions skill score.
>> > > > > >> > - Defined 2 corresponding obs.field entries to verify
the
>> > > forecast
>> > > > > >> data.
>> > > > > >> > In both the verifying threshold is >=2.54. This
assumes that
>> you
>> > > > > >> > pass
>> > > > > >> the
>> > > > > >> > raw StageIV data as input.
>> > > > > >> > - In the output_flag section, I turned off the line
types
>> that
>> > > > > >> > don't apply.
>> > > > > >> >
>> > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
>> > > > > >> > grabbing record number 19 from the input file, it's
not
>> > > > > >> > recommended. Instead,
>> > > > > >> it'd
>> > > > > >> > be better to figure out how to set the level string so
that
>> MET
>> > > > > >> > always grabs the same record, regardless if it's in
spot
>> number
>> > 19
>> > > > > >> > or 20 or any other location in the GRIB file.
>> > > > > >> >
>> > > > > >> > Hope this helps clarify. What other questions do you
have?
>> > > > > >> >
>> > > > > >> > Thanks,
>> > > > > >> >
>> > > > > >> > John Halley Gotway
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
>> > via
>> > > RT
>> > > > > >> > < met_help at ucar.edu> wrote:
>> > > > > >> >
>> > > > > >> > >
>> > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
>> > > > > >> > > Transaction: Ticket created by
>> christopher.melick at us.af.mil
>> > > > > >> > > Queue: met_help
>> > > > > >> > > Subject: Help with grid_stat on Neighborhood
>> statistics
>> > > > > >> > > Owner: Nobody
>> > > > > >> > > Requestors: christopher.melick at us.af.mil
>> > > > > >> > > Status: new
>> > > > > >> > > Ticket <URL:
>> > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > Hi John,
>> > > > > >> > >
>> > > > > >> > > I am trying to run grid_stat in MET to calculate
>> Neighborhood
>> > > > > >> > > statistics for probability of precipitation from
some
>> ensemble
>> > > > > >> > > data with the observations
>> > > > > >> > > coming from Stage IV analyses. Unfortunately, I am
>> getting
>> > no
>> > > > > output
>> > > > > >> > > (just
>> > > > > >> > > headers) in many of the ASCII files from the program
when I
>> > run
>> > > > it.
>> > > > > >> > Could
>> > > > > >> > > you help me figure out what I am doing wrong? I
have
>> provided
>> > > > > >> > > the command line specifics I use as well as the
output log
>> > file
>> > > > > >> > > that is
>> > > > > >> > created when
>> > > > > >> > > running that command. I also have attached the
>> configuration
>> > > > file.
>> > > > > >> I
>> > > > > >> > am
>> > > > > >> > > having trouble using ftp to send the raw data from
any of
>> the
>> > > > > clients
>> > > > > >> > > available to us. I believe this restriction
occurred
>> > recently
>> > > > > since
>> > > > > >> Bob
>> > > > > >> > > Craig had mentioned that he had been able to
connect
>> > externally
>> > > > > >> > > in the past.
>> > > > > >> > >
>> > > > > >> > > I should mention that I have pre-processed the Stage
IV
>> > analyses
>> > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
>> > since
>> > > > > >> > > I was unclear as to whether MET was able to do that
on the
>> fly
>> > > > > >> > > when running
>> > > > > >> > grid_stat.
>> > > > > >> > >
>> > > > > >> > > Thanks,
>> > > > > >> > > Chris
>> > > > > >> > >
>> > > > > >> > > Python command:
>> > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
>> > > > > >> > > 'precip/post-processed-obs.nc', \
>> > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp', '-v',
>> > '5]
>> > > > > >> > > Call(metcall)
>> > > > > >> > >
>> > > > > >> > > //SIGNED//
>> > > > > >> > >
>> > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
>> > > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
>> > > WS/WXEV)
>> > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm (402)
>> > > > > >> > > 294-3693
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 21 14:46:56 2018
Hi John,
You are correct --- I never received the message below. Are the
average coverage values what is used to compute the AFSS? I guess I
am still a little confused - I was just using the first example to
compute the O_RATE/F_RATE for AFSS. I'm not so sure exploring this
option would help me in the example case I sent you because most
events occurred in the interior of the domain. Also, wouldn't edge
effects be eliminated if the vld_thresh is set to one in grid_stat
since the entire neighborhood must contain valid data?
If we leave the whole neighborhood averaging aside, and just focus on
the computing AFSS at the grid scale (neighborhood = 1 grid point),
then my FSS is still greater than AFSS. Missing data points on the
analysis grid only result from different domain areas between that
dataset and the ensemble forecast system.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 21, 2018 2:59 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
Tressa sent an email on this issue, but she didn't include
met_help at ucar.edu.
So I don't think you received it. I've cut-and-pasted it below:
John and Chris,
It is not generally true that the average of the coverage values is
the base rate. I think it may sometimes be true, if there are no
events in edge neighborhoods without full coverage.
I worked up the following example with a 4x4 neighborhood:
1 0 1 1
0 0 0 0
1 0 1 1
0 1 0 0
The rate here is 7/16 = .4375
Fractional coverage values for a 3x3 neighborhood are:
1/4 2/6 2/6 2/4
2/6 4/9 4/9 4/6
2/6 3/9 3/9 2/6
2/4 3/6 3/6 2/4
The average of these values is 0.4149306
I suppose if each fraction had the same denominator (e.g. no edges, or
all non events there) then we would add each event up for 9 different
neighborhoods and then divide by 9. In that case, it should work.
Tressa
On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> I don't get negative values for AFSS but I do get values that are
less
> than what I calculate for FSS. I approach the procedure for
computing all
> the metrics such that everything is on the same "playing ground".
Thus,
> F_RATE and O_RATE are only computed by the mask of defined grid
points on
> *BOTH* the observation and forecast grids. When I process the
sample time
> period I sent you, F_RATE >> O_RATE which causes AFSS to be a small
value.
> Is that not the approach that I should follow? Were you able to
> examine my script and raw data file I sent you?
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 20, 2018 8:20 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Your email got me thinking. Thinking through some toy examples in
my
> mind, I can see that probably yes, the mean of the fractional
coverage field
> *should* be the same as the frequency of the event.
>
> I ran MET in the debugger and was able to stop at the point in the
logic
> where this is computed. At that point, we're passing around arrays
of
> fractional coverage field values as well as arrays of 0's and 1's
> indicating whether the event did or did not occur at each grid
point.
> Comparing the mean of these arrays, I find that they are close but
not
> identical. Sometimes one is larger, sometimes the other.
>
> I can think of 2 reasons why they wouldn't be the same, but these
may not
> fully explain it:
>
> (1) In Grid-Stat, the fractional coverage field is computed once
over the
> full verification domain. Then, the masking regions are processed
as
> subsets of these grid points. For a grid point that falls just
barely
> inside that mask, around 1/2 of the grid points used to compute the
> fractional coverage value came from *OUTSIDE* that masking region.
The 0/1
> arrays are only for points inside the current mask. So the mean of
the 0/1
> array represents only the points inside the mask while the mean of
the
> fractional coverage includes info from outside the mask.
>
> (2) When data values are missing in the forecast or observation
fields,
> depending on how the user has set the vld_thresh config file option,
the
> "denominator" used to compute the fractional coverage field value
could
> change from grid point to grid point. That would likely throw off
the mean
> of the fractional coverage field.
>
> Now what is the *CORRECT* way of computing the forecast and
observation
> rates? In papers about this, I see references to the "base_rate"
but no
> explicit equation.
>
> One very nice feature about the way MET is currently doing it is
that
> F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> Therefore, UFSS and AFSS remain constant as well. Using the other
method,
> O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood
size
> since the number of points included from outside the mask would
change.
>
> You mentioned that you had issues with negative UFSS and AFSS
values.
> Perhaps it's related to how you're computing the F_RATE and O_RATE?
>
> Lots of little details to consider here!
>
> Thanks,
> John
>
>
> On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > can't be calculated in the cases below? Can't the gridded data
still
> > be read in to determine the relative frequency of the events,
regardless
> > whether grid_stat is used to compute fractional coverage fields?
I am
> > basically doing that right now in some Python code I developed
when
> > calculating Fractions Skill Score.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 14, 2018 3:20 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I'm working on the interp.field option we discussed.
> >
> > I have a quick question for you. The NBRCNT line type that is
computed
> by
> > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
> > grid points over a masking region at which the "event" is
occurring in
> the
> > forecast and observation fields respectively.
> >
> > When grid_stat computes the fractional coverage field, we can
compute
> > that. But if grid_stat does *NOT* compute the fractional coverage
> fields,
> > then we don't know what those ratio's are.
> >
> > - If interp.field = FCST, then O_RATE = NA.
> > - If interp.field = OBS, then F_RATE = NA.
> > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > John,
> > >
> > > That sounds like an excellent plan. Thanks for your assistance
and
> > > trying to accommodate and make changes to the MET software so
quickly.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Just wanted to close the loop on this ticket. I'll try to get
it into
> > the
> > > next release due out next in March.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > Yes, thanks for the explanations. That clears up any possible
> > > > ambiguity and confusion. I kind of thought that was the
procedure.
> > > >
> > > > When do you think that the next MET would be released with the
> > > > enhancement that we discussed for the neighborhood statistics?
Is it
> > > > an easy or difficult update to the software?
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > It sounds like you understand how the fractional coverage
field is
> > > > computed.
> > > >
> > > > - Read the raw field... probability values between 0 and 1 in
your
> > case.
> > > > - Apply the categorical threshold to that data to generate a
field of
> > 0's
> > > > and 1's... >0.5 in your case.
> > > > - For each grid point, replace the value at that grid point
with the
> > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
> > event
> > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
> > current
> > > > point, so fractional coverage = 4/9.
> > > > - Compare the forecast fractional coverage values to the
observed
> > > > fractional coverage values to compute FSS.
> > > >
> > > > Yes, I do think it would be a good idea to talk to Craig about
how he
> > > > configured MET for his work.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Getting back to your initial response, I was able to run
grid_stat
> > > > > with your changes in the attached configuration file to get
FSS
> > output
> > > > > (as well as the other statistics for the neighborhood type
> results).
> > > > >
> > > > > However, I do have a question about the forecast categorical
> > threshold
> > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > > probabilities from the ensemble probabilities: What is the
actual
> > > > > formula that MET uses to create the final field to calculate
FSS?
> Is
> > > > > it a strict neighborhood fractional coverage of ensemble
> > probabilities
> > > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > > operation or does the threshold represent something else? I
know
> in
> > > > > Craig's paper, they were calling the last step a
neighborhood
> average
> > > > > of the ensemble probabilities to calculate NEP (since the
event had
> > > > > already been defined at the grid point in determining the
ensemble
> > > > probability [i.e. likelihood of exceeding
> > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig was
> > > using
> > > > > MET for his paper.
> > > > >
> > > > > Thanks again,
> > > > > Chris
> > > > >
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > I was thinking about what changes would be needed to compute
FSS on
> > > input
> > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > section,
> > > > > just like we have in the “interp” section. It would be set
to
> FCST,
> > > OBS,
> > > > > or BOTH (default) and would control the logic as to which
fields
> > should
> > > > be
> > > > > passed through the logic for deriving fractional coverage
fields.
> In
> > > > your
> > > > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > > > probability values.
> > > > >
> > > > > Would that logic make sense to you?
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > > > Chris,
> > > > > >
> > > > > > In fact, SPC's 40-km radius probabilities were the
motivation for
> > > > > > adding the shape = circle option to MET.
> > > > > >
> > > > > > For your first question, no, there is not currently a
Gaussian
> > > > > > smoother option for the "interp" section in Grid-Stat.
> Currently,
> > > the
> > > > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > > > unweighted mean).
> > > > > > There is a distance-weighted mean option for point data
where the
> > > > > > weight is 1/distance squared. But that doesn't work for
> Grid-Stat
> > > > > > because we'd end up dividing by 0.
> > > > > >
> > > > > > I can see how adding a gaussian smoother would be useful.
> > > > > >
> > > > > > Here's what we've done in the past with this SPC 40-km
> > probabilities:
> > > > > >
> > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
> > > > > > TRUE;" in the config file).
> > > > > > (2) Use the "interp" section to define a smoothing
operation on
> the
> > > > > > observations:
> > > > > > interp = {
> > > > > > field = OBS;
> > > > > > vld_thresh = 1.0;
> > > > > > shape = CIRCLE;
> > > > > > type = [
> > > > > > { method = MAX; width = 11; }
> > > > > > ];
> > > > > > }
> > > > > > Or set width to some circle diameter in grid units that
works
> > out
> > > > > > to 40km in the real world.
> > > > > >
> > > > > > In this example, we post-process the observations to make
them
> > > > > > comparable the event for which the probability was
defined.
> > > > > >
> > > > > > But I think I understand what you'd like to do...
interpret as
> the
> > > > > > probability value at each grid point as if it was a
fractional
> > > > > > coverage value. So don't apply a threshold and
neighborhood size
> > to
> > > > > > the forecast data at all. Just pass it directly to the
routine
> > that
> > > > > computes FSS.
> > > > > >
> > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > Grid-Stat
> > > > > > to do that in met-6.1. If you think this logic would be
> generally
> > > > > > useful, we could consider adding a flag to the config file
to
> > enable
> > > > > that logic.
> > > > > >
> > > > > > Sorry I don't have better answers for you.
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >>
> > > > > >> Hi John,
> > > > > >>
> > > > > >> Your suggestion for accomplishing #2 sounds promising. I
need
> to
> > > > think
> > > > > >> the details through some. On a somewhat related note,
is
> there a
> > > > > Gaussian
> > > > > >> smoother flag for smoothing the probabilistic fields? I
know
> > that
> > > > was
> > > > > >> mentioned in Craig's paper and I have pursued that option
in the
> > > past
> > > > > >> (but just not using MET).
> > > > > >>
> > > > > >> I guess my basic question is can you apply a neighborhood
width
> > that
> > > > > >> just includes the grid point maybe because you already
have
> > > > > >> inherently a probabilistic field with spatial
uncertainty?
> Would
> > > that
> > > > > be a width of 1
> > > > > >> or does that also include the nearest grid point? A
good
> > example
> > > of
> > > > > this
> > > > > >> would be a probabilistic forecast issued from the Storm
> Prediction
> > > > > >> Center where the probabilities implicitly assume a 40-km
radius
> of
> > > > > >> influence (so there is no need to compute a fractional
coverage
> of
> > > the
> > > > > event).
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> Chris
> > > > > >>
> > > > > >>
> > > > > >> //SIGNED//
> > > > > >>
> > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> Weather
> > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> christopher.melick at us.af.mil>
> > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC 16
> > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986] Help
> > > with
> > > > > >> grid_stat on Neighborhood statistics
> > > > > >>
> > > > > >> Chris,
> > > > > >>
> > > > > >> (1) The default shape is a square since that's what MET
> previously
> > > > did.
> > > > > >> The circle is the *new* option.
> > > > > >>
> > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
> MET
> > > by
> > > > > >> doing the following...
> > > > > >> - For each field separately (i.e. fcst and obs), apply
the
> > > > > >> threshold
> > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > > >> - Apply the neighborhood width and shape to compute a
> fractional
> > > > > >> coverage value for each grid point.
> > > > > >> - Compare the fcst and obs fractional coverage fields
to
> compute
> > > > FSS.
> > > > > >>
> > > > > >> But there are some other options that you may find
useful. The
> > > > "interp"
> > > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > > operation.
> > > > > >> After reading the raw input data, you can specify if/how
to
> smooth
> > > > > >> that data prior to computing stats. For example, since
you
> > > mentioned
> > > > > >> maximums, you could replace the value at each grid point
with
> the
> > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > >>
> > > > > >> interp = {
> > > > > >> field = FCST;
> > > > > >> vld_thresh = 1.0;
> > > > > >> shape = SQUARE;
> > > > > >>
> > > > > >> type = [
> > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > > points
> > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
> > > > > >> closest points
> > > > > >> ];
> > > > > >> }
> > > > > >>
> > > > > >> Using the example above, you'll get 3 times the amount of
output
> > > from
> > > > > MET.
> > > > > >> The first smoother really is no smoothing at all. It
would run
> on
> > > > > >> the raw input data. The second smoother replaces the
value at
> > each
> > > > > >> grid point with the maximum of the 25 surrounding points.
The
> > third
> > > > > >> smoother replaces the value at each grid point with the
mean of
> > > those
> > > > > 25.
> > > > > >>
> > > > > >> I worked with Craig Schwartz when he was running MET for
that
> > paper.
> > > > > >> My recollection is that he was able to use the
configuration
> > options
> > > > > >> of Grid-Stat to accomplish the types of processing he was
after.
> > > But
> > > > > >> I don't recall all the details.
> > > > > >>
> > > > > >> If you explain exactly what you're trying to do, it may
be
> > possible
> > > > > >> to configure Grid-Stat to accomplish that.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
> via
> > > RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > >> >
> > > > > >> > Hi John,
> > > > > >> >
> > > > > >> > Your explanation and suggestions are very helpful and
do
> explain
> > > my
> > > > > >> > problems quite substantially. I will have to give it a
try to
> > see
> > > > > >> > what happens.
> > > > > >> >
> > > > > >> > I do have a couple of other minor questions dealing
with how
> the
> > > > > >> > neighborhood verification is performed:
> > > > > >> > 1.) I noticed in the documentation that there is a
flag for
> > > > > >> > defining the shape of the neighborhood area (i.e.
circular or
> > > > > >> > square). What is the default if you don't specify?
> > > > > >> > 2.) In the literature, there are a couple of different
ways
> to
> > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
> and
> > > > > >> > Sobash 2017;
> > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1
> ).
> > > I
> > > > > >> > was curious is the only way to get the Fractions Skill
Score
> > (FSS)
> > > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > > >> > fractional coverage of the threshold event? I have
previously
> > > > > >> > explored computing the neighborhood maximum value
within a
> > radius
> > > of
> > > > > each grid point before calculating a probability.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Chris
> > > > > >> >
> > > > > >> >
> > > > > >> > //SIGNED//
> > > > > >> >
> > > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > -----Original Message-----
> > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> > christopher.melick at us.af.mil>
> > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > robert.craig.2 at us.af.mil>
> > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > >> > grid_stat on Neighborhood statistics
> > > > > >> >
> > > > > >> > Hi Chris,
> > > > > >> >
> > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > >> > statistics in Grid-Stat.
> > > > > >> >
> > > > > >> > First, Grid-Stat definitely does have the ability to
threshold
> > the
> > > > > >> StageIV
> > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
> not
> > > > > needed.
> > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
> just
> > > set:
> > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > >> >
> > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
> > > >=0.1
> > > > > >> inches.
> > > > > >> >
> > > > > >> > If for some reason you have data that's already 0's and
1's,
> > you'd
> > > > > >> > just use a different threshold in the config file:
> > > > > >> > cat_thresh = [ ==1 ];
> > > > > >> >
> > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
> event
> > is
> > > > > >> > occurring. Otherwise, it's not.
> > > > > >> >
> > > > > >> > Now on the to real issue. Looking at your config file,
I see
> > that
> > > > > >> you're
> > > > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > > > >> > computes
> > > > > >> the
> > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
> why
> > > > > >> > you're getting no output in the NBRCNT line type, which
is the
> > one
> > > > > >> > that
> > > > > >> contains
> > > > > >> > Fractions Skill Score (FSS).
> > > > > >> >
> > > > > >> > In the attached config file, I've made a few changes:
> > > > > >> > - Set the obtype to indicate that you're verifying
against
> > > StageIV
> > > > > >> > - Defined 2 fcst.field entries.
> > > > > >> > - In the first we process QP010 as a probability
field.
> > > Here
> > > > > >> > cat_thresh is set to ==0.10 which is shorthand notation
for
> > > > > >> > defining
> > > > > >> bins
> > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > >> > - In the second we process it as a field of
scalars.
> Here
> > > > > >> > cat_thresh is set to >=0.5. So that's the probability
> threshold
> > > > > >> > I'm
> > > > > >> using
> > > > > >> > for defining events for fractions skill score.
> > > > > >> > - Defined 2 corresponding obs.field entries to verify
the
> > > forecast
> > > > > >> data.
> > > > > >> > In both the verifying threshold is >=2.54. This assumes
that
> you
> > > > > >> > pass
> > > > > >> the
> > > > > >> > raw StageIV data as input.
> > > > > >> > - In the output_flag section, I turned off the line
types
> that
> > > > > >> > don't apply.
> > > > > >> >
> > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
> > > > > >> > grabbing record number 19 from the input file, it's not
> > > > > >> > recommended. Instead,
> > > > > >> it'd
> > > > > >> > be better to figure out how to set the level string so
that
> MET
> > > > > >> > always grabs the same record, regardless if it's in
spot
> number
> > 19
> > > > > >> > or 20 or any other location in the GRIB file.
> > > > > >> >
> > > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > John Halley Gotway
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
> > via
> > > RT
> > > > > >> > < met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > > > >> > > Queue: met_help
> > > > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > > > >> > > Owner: Nobody
> > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > >> > > Status: new
> > > > > >> > > Ticket <URL:
> > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Hi John,
> > > > > >> > >
> > > > > >> > > I am trying to run grid_stat in MET to calculate
> Neighborhood
> > > > > >> > > statistics for probability of precipitation from some
> ensemble
> > > > > >> > > data with the observations
> > > > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting
> > no
> > > > > output
> > > > > >> > > (just
> > > > > >> > > headers) in many of the ASCII files from the program
when I
> > run
> > > > it.
> > > > > >> > Could
> > > > > >> > > you help me figure out what I am doing wrong? I have
> provided
> > > > > >> > > the command line specifics I use as well as the
output log
> > file
> > > > > >> > > that is
> > > > > >> > created when
> > > > > >> > > running that command. I also have attached the
> configuration
> > > > file.
> > > > > >> I
> > > > > >> > am
> > > > > >> > > having trouble using ftp to send the raw data from
any of
> the
> > > > > clients
> > > > > >> > > available to us. I believe this restriction
occurred
> > recently
> > > > > since
> > > > > >> Bob
> > > > > >> > > Craig had mentioned that he had been able to connect
> > externally
> > > > > >> > > in the past.
> > > > > >> > >
> > > > > >> > > I should mention that I have pre-processed the Stage
IV
> > analyses
> > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> > since
> > > > > >> > > I was unclear as to whether MET was able to do that
on the
> fly
> > > > > >> > > when running
> > > > > >> > grid_stat.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Chris
> > > > > >> > >
> > > > > >> > > Python command:
> > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp',
'-v',
> > '5]
> > > > > >> > > Call(metcall)
> > > > > >> > >
> > > > > >> > > //SIGNED//
> > > > > >> > >
> > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
> > > WS/WXEV)
> > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > > > >> > > 294-3693
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Tue Feb 27 08:21:53 2018
Hi John,
Did you ever get the chance to take a look at my Python program and
see if you notice anything that needs to be corrected? I'm still not
sure what to make of the values I obtained for AFSS and why they are
lower than FSS, especially at the grid-scale. Is this even possible?
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
Sent: Wednesday, February 21, 2018 3:47 PM
To: 'met_help at ucar.edu' <met_help at ucar.edu>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi John,
You are correct --- I never received the message below. Are the
average coverage values what is used to compute the AFSS? I guess I
am still a little confused - I was just using the first example to
compute the O_RATE/F_RATE for AFSS. I'm not so sure exploring this
option would help me in the example case I sent you because most
events occurred in the interior of the domain. Also, wouldn't edge
effects be eliminated if the vld_thresh is set to one in grid_stat
since the entire neighborhood must contain valid data?
If we leave the whole neighborhood averaging aside, and just focus on
the computing AFSS at the grid scale (neighborhood = 1 grid point),
then my FSS is still greater than AFSS. Missing data points on the
analysis grid only result from different domain areas between that
dataset and the ensemble forecast system.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 21, 2018 2:59 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
Tressa sent an email on this issue, but she didn't include
met_help at ucar.edu.
So I don't think you received it. I've cut-and-pasted it below:
John and Chris,
It is not generally true that the average of the coverage values is
the base rate. I think it may sometimes be true, if there are no
events in edge neighborhoods without full coverage.
I worked up the following example with a 4x4 neighborhood:
1 0 1 1
0 0 0 0
1 0 1 1
0 1 0 0
The rate here is 7/16 = .4375
Fractional coverage values for a 3x3 neighborhood are:
1/4 2/6 2/6 2/4
2/6 4/9 4/9 4/6
2/6 3/9 3/9 2/6
2/4 3/6 3/6 2/4
The average of these values is 0.4149306
I suppose if each fraction had the same denominator (e.g. no edges, or
all non events there) then we would add each event up for 9 different
neighborhoods and then divide by 9. In that case, it should work.
Tressa
On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> I don't get negative values for AFSS but I do get values that are
less
> than what I calculate for FSS. I approach the procedure for
computing all
> the metrics such that everything is on the same "playing ground".
Thus,
> F_RATE and O_RATE are only computed by the mask of defined grid
points on
> *BOTH* the observation and forecast grids. When I process the
sample time
> period I sent you, F_RATE >> O_RATE which causes AFSS to be a small
value.
> Is that not the approach that I should follow? Were you able to
> examine my script and raw data file I sent you?
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 20, 2018 8:20 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> Your email got me thinking. Thinking through some toy examples in
my
> mind, I can see that probably yes, the mean of the fractional
coverage field
> *should* be the same as the frequency of the event.
>
> I ran MET in the debugger and was able to stop at the point in the
logic
> where this is computed. At that point, we're passing around arrays
of
> fractional coverage field values as well as arrays of 0's and 1's
> indicating whether the event did or did not occur at each grid
point.
> Comparing the mean of these arrays, I find that they are close but
not
> identical. Sometimes one is larger, sometimes the other.
>
> I can think of 2 reasons why they wouldn't be the same, but these
may not
> fully explain it:
>
> (1) In Grid-Stat, the fractional coverage field is computed once
over the
> full verification domain. Then, the masking regions are processed
as
> subsets of these grid points. For a grid point that falls just
barely
> inside that mask, around 1/2 of the grid points used to compute the
> fractional coverage value came from *OUTSIDE* that masking region.
The 0/1
> arrays are only for points inside the current mask. So the mean of
the 0/1
> array represents only the points inside the mask while the mean of
the
> fractional coverage includes info from outside the mask.
>
> (2) When data values are missing in the forecast or observation
fields,
> depending on how the user has set the vld_thresh config file option,
the
> "denominator" used to compute the fractional coverage field value
could
> change from grid point to grid point. That would likely throw off
the mean
> of the fractional coverage field.
>
> Now what is the *CORRECT* way of computing the forecast and
observation
> rates? In papers about this, I see references to the "base_rate"
but no
> explicit equation.
>
> One very nice feature about the way MET is currently doing it is
that
> F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> Therefore, UFSS and AFSS remain constant as well. Using the other
method,
> O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood
size
> since the number of points included from outside the mask would
change.
>
> You mentioned that you had issues with negative UFSS and AFSS
values.
> Perhaps it's related to how you're computing the F_RATE and O_RATE?
>
> Lots of little details to consider here!
>
> Thanks,
> John
>
>
> On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > can't be calculated in the cases below? Can't the gridded data
still
> > be read in to determine the relative frequency of the events,
regardless
> > whether grid_stat is used to compute fractional coverage fields?
I am
> > basically doing that right now in some Python code I developed
when
> > calculating Fractions Skill Score.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 14, 2018 3:20 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I'm working on the interp.field option we discussed.
> >
> > I have a quick question for you. The NBRCNT line type that is
computed
> by
> > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
> > grid points over a masking region at which the "event" is
occurring in
> the
> > forecast and observation fields respectively.
> >
> > When grid_stat computes the fractional coverage field, we can
compute
> > that. But if grid_stat does *NOT* compute the fractional coverage
> fields,
> > then we don't know what those ratio's are.
> >
> > - If interp.field = FCST, then O_RATE = NA.
> > - If interp.field = OBS, then F_RATE = NA.
> > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > John,
> > >
> > > That sounds like an excellent plan. Thanks for your assistance
and
> > > trying to accommodate and make changes to the MET software so
quickly.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Just wanted to close the loop on this ticket. I'll try to get
it into
> > the
> > > next release due out next in March.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > Yes, thanks for the explanations. That clears up any possible
> > > > ambiguity and confusion. I kind of thought that was the
procedure.
> > > >
> > > > When do you think that the next MET would be released with the
> > > > enhancement that we discussed for the neighborhood statistics?
Is it
> > > > an easy or difficult update to the software?
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > It sounds like you understand how the fractional coverage
field is
> > > > computed.
> > > >
> > > > - Read the raw field... probability values between 0 and 1 in
your
> > case.
> > > > - Apply the categorical threshold to that data to generate a
field of
> > 0's
> > > > and 1's... >0.5 in your case.
> > > > - For each grid point, replace the value at that grid point
with the
> > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
> > event
> > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
> > current
> > > > point, so fractional coverage = 4/9.
> > > > - Compare the forecast fractional coverage values to the
observed
> > > > fractional coverage values to compute FSS.
> > > >
> > > > Yes, I do think it would be a good idea to talk to Craig about
how he
> > > > configured MET for his work.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Getting back to your initial response, I was able to run
grid_stat
> > > > > with your changes in the attached configuration file to get
FSS
> > output
> > > > > (as well as the other statistics for the neighborhood type
> results).
> > > > >
> > > > > However, I do have a question about the forecast categorical
> > threshold
> > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > > probabilities from the ensemble probabilities: What is the
actual
> > > > > formula that MET uses to create the final field to calculate
FSS?
> Is
> > > > > it a strict neighborhood fractional coverage of ensemble
> > probabilities
> > > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > > operation or does the threshold represent something else? I
know
> in
> > > > > Craig's paper, they were calling the last step a
neighborhood
> average
> > > > > of the ensemble probabilities to calculate NEP (since the
event had
> > > > > already been defined at the grid point in determining the
ensemble
> > > > probability [i.e. likelihood of exceeding
> > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig was
> > > using
> > > > > MET for his paper.
> > > > >
> > > > > Thanks again,
> > > > > Chris
> > > > >
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > I was thinking about what changes would be needed to compute
FSS on
> > > input
> > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > section,
> > > > > just like we have in the “interp” section. It would be set
to
> FCST,
> > > OBS,
> > > > > or BOTH (default) and would control the logic as to which
fields
> > should
> > > > be
> > > > > passed through the logic for deriving fractional coverage
fields.
> In
> > > > your
> > > > > case, you’d set it to OBS and we’d just use the raw input
forecast
> > > > > probability values.
> > > > >
> > > > > Would that logic make sense to you?
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > > > Chris,
> > > > > >
> > > > > > In fact, SPC's 40-km radius probabilities were the
motivation for
> > > > > > adding the shape = circle option to MET.
> > > > > >
> > > > > > For your first question, no, there is not currently a
Gaussian
> > > > > > smoother option for the "interp" section in Grid-Stat.
> Currently,
> > > the
> > > > > > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN
(for
> > > > > unweighted mean).
> > > > > > There is a distance-weighted mean option for point data
where the
> > > > > > weight is 1/distance squared. But that doesn't work for
> Grid-Stat
> > > > > > because we'd end up dividing by 0.
> > > > > >
> > > > > > I can see how adding a gaussian smoother would be useful.
> > > > > >
> > > > > > Here's what we've done in the past with this SPC 40-km
> > probabilities:
> > > > > >
> > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob =
> > > > > > TRUE;" in the config file).
> > > > > > (2) Use the "interp" section to define a smoothing
operation on
> the
> > > > > > observations:
> > > > > > interp = {
> > > > > > field = OBS;
> > > > > > vld_thresh = 1.0;
> > > > > > shape = CIRCLE;
> > > > > > type = [
> > > > > > { method = MAX; width = 11; }
> > > > > > ];
> > > > > > }
> > > > > > Or set width to some circle diameter in grid units that
works
> > out
> > > > > > to 40km in the real world.
> > > > > >
> > > > > > In this example, we post-process the observations to make
them
> > > > > > comparable the event for which the probability was
defined.
> > > > > >
> > > > > > But I think I understand what you'd like to do...
interpret as
> the
> > > > > > probability value at each grid point as if it was a
fractional
> > > > > > coverage value. So don't apply a threshold and
neighborhood size
> > to
> > > > > > the forecast data at all. Just pass it directly to the
routine
> > that
> > > > > computes FSS.
> > > > > >
> > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > Grid-Stat
> > > > > > to do that in met-6.1. If you think this logic would be
> generally
> > > > > > useful, we could consider adding a flag to the config file
to
> > enable
> > > > > that logic.
> > > > > >
> > > > > > Sorry I don't have better answers for you.
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >>
> > > > > >> Hi John,
> > > > > >>
> > > > > >> Your suggestion for accomplishing #2 sounds promising. I
need
> to
> > > > think
> > > > > >> the details through some. On a somewhat related note,
is
> there a
> > > > > Gaussian
> > > > > >> smoother flag for smoothing the probabilistic fields? I
know
> > that
> > > > was
> > > > > >> mentioned in Craig's paper and I have pursued that option
in the
> > > past
> > > > > >> (but just not using MET).
> > > > > >>
> > > > > >> I guess my basic question is can you apply a neighborhood
width
> > that
> > > > > >> just includes the grid point maybe because you already
have
> > > > > >> inherently a probabilistic field with spatial
uncertainty?
> Would
> > > that
> > > > > be a width of 1
> > > > > >> or does that also include the nearest grid point? A
good
> > example
> > > of
> > > > > this
> > > > > >> would be a probabilistic forecast issued from the Storm
> Prediction
> > > > > >> Center where the probabilities implicitly assume a 40-km
radius
> of
> > > > > >> influence (so there is no need to compute a fractional
coverage
> of
> > > the
> > > > > event).
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> Chris
> > > > > >>
> > > > > >>
> > > > > >> //SIGNED//
> > > > > >>
> > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> Weather
> > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> christopher.melick at us.af.mil>
> > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF
ACC 16
> > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986] Help
> > > with
> > > > > >> grid_stat on Neighborhood statistics
> > > > > >>
> > > > > >> Chris,
> > > > > >>
> > > > > >> (1) The default shape is a square since that's what MET
> previously
> > > > did.
> > > > > >> The circle is the *new* option.
> > > > > >>
> > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
> MET
> > > by
> > > > > >> doing the following...
> > > > > >> - For each field separately (i.e. fcst and obs), apply
the
> > > > > >> threshold
> > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > > >> - Apply the neighborhood width and shape to compute a
> fractional
> > > > > >> coverage value for each grid point.
> > > > > >> - Compare the fcst and obs fractional coverage fields
to
> compute
> > > > FSS.
> > > > > >>
> > > > > >> But there are some other options that you may find
useful. The
> > > > "interp"
> > > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > > operation.
> > > > > >> After reading the raw input data, you can specify if/how
to
> smooth
> > > > > >> that data prior to computing stats. For example, since
you
> > > mentioned
> > > > > >> maximums, you could replace the value at each grid point
with
> the
> > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > >>
> > > > > >> interp = {
> > > > > >> field = FCST;
> > > > > >> vld_thresh = 1.0;
> > > > > >> shape = SQUARE;
> > > > > >>
> > > > > >> type = [
> > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > > points
> > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean of
the 25
> > > > > >> closest points
> > > > > >> ];
> > > > > >> }
> > > > > >>
> > > > > >> Using the example above, you'll get 3 times the amount of
output
> > > from
> > > > > MET.
> > > > > >> The first smoother really is no smoothing at all. It
would run
> on
> > > > > >> the raw input data. The second smoother replaces the
value at
> > each
> > > > > >> grid point with the maximum of the 25 surrounding points.
The
> > third
> > > > > >> smoother replaces the value at each grid point with the
mean of
> > > those
> > > > > 25.
> > > > > >>
> > > > > >> I worked with Craig Schwartz when he was running MET for
that
> > paper.
> > > > > >> My recollection is that he was able to use the
configuration
> > options
> > > > > >> of Grid-Stat to accomplish the types of processing he was
after.
> > > But
> > > > > >> I don't recall all the details.
> > > > > >>
> > > > > >> If you explain exactly what you're trying to do, it may
be
> > possible
> > > > > >> to configure Grid-Stat to accomplish that.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
> via
> > > RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > >> >
> > > > > >> > Hi John,
> > > > > >> >
> > > > > >> > Your explanation and suggestions are very helpful and
do
> explain
> > > my
> > > > > >> > problems quite substantially. I will have to give it a
try to
> > see
> > > > > >> > what happens.
> > > > > >> >
> > > > > >> > I do have a couple of other minor questions dealing
with how
> the
> > > > > >> > neighborhood verification is performed:
> > > > > >> > 1.) I noticed in the documentation that there is a
flag for
> > > > > >> > defining the shape of the neighborhood area (i.e.
circular or
> > > > > >> > square). What is the default if you don't specify?
> > > > > >> > 2.) In the literature, there are a couple of different
ways
> to
> > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
> and
> > > > > >> > Sobash 2017;
> > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
0400.1
> ).
> > > I
> > > > > >> > was curious is the only way to get the Fractions Skill
Score
> > (FSS)
> > > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > > >> > fractional coverage of the threshold event? I have
previously
> > > > > >> > explored computing the neighborhood maximum value
within a
> > radius
> > > of
> > > > > each grid point before calculating a probability.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Chris
> > > > > >> >
> > > > > >> >
> > > > > >> > //SIGNED//
> > > > > >> >
> > > > > >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > -----Original Message-----
> > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > >> > christopher.melick at us.af.mil>
> > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > robert.craig.2 at us.af.mil>
> > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > >> > grid_stat on Neighborhood statistics
> > > > > >> >
> > > > > >> > Hi Chris,
> > > > > >> >
> > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > >> > statistics in Grid-Stat.
> > > > > >> >
> > > > > >> > First, Grid-Stat definitely does have the ability to
threshold
> > the
> > > > > >> StageIV
> > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
> not
> > > > > needed.
> > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
> just
> > > set:
> > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > >> >
> > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same as
> > > >=0.1
> > > > > >> inches.
> > > > > >> >
> > > > > >> > If for some reason you have data that's already 0's and
1's,
> > you'd
> > > > > >> > just use a different threshold in the config file:
> > > > > >> > cat_thresh = [ ==1 ];
> > > > > >> >
> > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
> event
> > is
> > > > > >> > occurring. Otherwise, it's not.
> > > > > >> >
> > > > > >> > Now on the to real issue. Looking at your config file,
I see
> > that
> > > > > >> you're
> > > > > >> > processing probabilistic data. When "prob = TRUE", MET
only
> > > > > >> > computes
> > > > > >> the
> > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
> why
> > > > > >> > you're getting no output in the NBRCNT line type, which
is the
> > one
> > > > > >> > that
> > > > > >> contains
> > > > > >> > Fractions Skill Score (FSS).
> > > > > >> >
> > > > > >> > In the attached config file, I've made a few changes:
> > > > > >> > - Set the obtype to indicate that you're verifying
against
> > > StageIV
> > > > > >> > - Defined 2 fcst.field entries.
> > > > > >> > - In the first we process QP010 as a probability
field.
> > > Here
> > > > > >> > cat_thresh is set to ==0.10 which is shorthand notation
for
> > > > > >> > defining
> > > > > >> bins
> > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > >> > - In the second we process it as a field of
scalars.
> Here
> > > > > >> > cat_thresh is set to >=0.5. So that's the probability
> threshold
> > > > > >> > I'm
> > > > > >> using
> > > > > >> > for defining events for fractions skill score.
> > > > > >> > - Defined 2 corresponding obs.field entries to verify
the
> > > forecast
> > > > > >> data.
> > > > > >> > In both the verifying threshold is >=2.54. This assumes
that
> you
> > > > > >> > pass
> > > > > >> the
> > > > > >> > raw StageIV data as input.
> > > > > >> > - In the output_flag section, I turned off the line
types
> that
> > > > > >> > don't apply.
> > > > > >> >
> > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
> > > > > >> > grabbing record number 19 from the input file, it's not
> > > > > >> > recommended. Instead,
> > > > > >> it'd
> > > > > >> > be better to figure out how to set the level string so
that
> MET
> > > > > >> > always grabs the same record, regardless if it's in
spot
> number
> > 19
> > > > > >> > or 20 or any other location in the GRIB file.
> > > > > >> >
> > > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > John Halley Gotway
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
christopher.melick at us.af.mil
> > via
> > > RT
> > > > > >> > < met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > >> > > Transaction: Ticket created by
christopher.melick at us.af.mil
> > > > > >> > > Queue: met_help
> > > > > >> > > Subject: Help with grid_stat on Neighborhood
statistics
> > > > > >> > > Owner: Nobody
> > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > >> > > Status: new
> > > > > >> > > Ticket <URL:
> > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Hi John,
> > > > > >> > >
> > > > > >> > > I am trying to run grid_stat in MET to calculate
> Neighborhood
> > > > > >> > > statistics for probability of precipitation from some
> ensemble
> > > > > >> > > data with the observations
> > > > > >> > > coming from Stage IV analyses. Unfortunately, I am
getting
> > no
> > > > > output
> > > > > >> > > (just
> > > > > >> > > headers) in many of the ASCII files from the program
when I
> > run
> > > > it.
> > > > > >> > Could
> > > > > >> > > you help me figure out what I am doing wrong? I have
> provided
> > > > > >> > > the command line specifics I use as well as the
output log
> > file
> > > > > >> > > that is
> > > > > >> > created when
> > > > > >> > > running that command. I also have attached the
> configuration
> > > > file.
> > > > > >> I
> > > > > >> > am
> > > > > >> > > having trouble using ftp to send the raw data from
any of
> the
> > > > > clients
> > > > > >> > > available to us. I believe this restriction
occurred
> > recently
> > > > > since
> > > > > >> Bob
> > > > > >> > > Craig had mentioned that he had been able to connect
> > externally
> > > > > >> > > in the past.
> > > > > >> > >
> > > > > >> > > I should mention that I have pre-processed the Stage
IV
> > analyses
> > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> > since
> > > > > >> > > I was unclear as to whether MET was able to do that
on the
> fly
> > > > > >> > > when running
> > > > > >> > grid_stat.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Chris
> > > > > >> > >
> > > > > >> > > Python command:
> > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > >> > > 'GridStatConfig_POP', '-outdir', 'tmp',
'-v',
> > '5]
> > > > > >> > > Call(metcall)
> > > > > >> > >
> > > > > >> > > //SIGNED//
> > > > > >> > >
> > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > >> > > Verification Exploitation Team 16th Weather Squadron
(16
> > > WS/WXEV)
> > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm
(402)
> > > > > >> > > 294-3693
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Tue Feb 27 09:26:40 2018
Chris,
I looked at your python code and don't see anything obviously wrong in
the
equations you've used. However, your computation of AFSS won't match
what
MET is computing for the reasons we've already discussed. You're
computing
fcst_rate and obs_rate as an average of the fractional coverage
values.
MET computes them as a ratio of events occurring in the input fields.
As
Tressa demonstrated, those will not necessarily result in the same
values.
Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute
will not match the AFSS that MET computes.
Thanks,
John
On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Did you ever get the chance to take a look at my Python program and
see if
> you notice anything that needs to be corrected? I'm still not sure
what to
> make of the values I obtained for AFSS and why they are lower than
FSS,
> especially at the grid-scale. Is this even possible?
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> Sent: Wednesday, February 21, 2018 3:47 PM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi John,
>
> You are correct --- I never received the message below. Are the
average
> coverage values what is used to compute the AFSS? I guess I am
still a
> little confused - I was just using the first example to compute the
> O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would help
> me in the example case I sent you because most events occurred in
the
> interior of the domain. Also, wouldn't edge effects be eliminated
if the
> vld_thresh is set to one in grid_stat since the entire neighborhood
must
> contain valid data?
>
> If we leave the whole neighborhood averaging aside, and just focus
on the
> computing AFSS at the grid scale (neighborhood = 1 grid point), then
my FSS
> is still greater than AFSS. Missing data points on the analysis
grid only
> result from different domain areas between that dataset and the
ensemble
> forecast system.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing,
> Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 21, 2018 2:59 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> Tressa sent an email on this issue, but she didn't include
> met_help at ucar.edu.
> So I don't think you received it. I've cut-and-pasted it below:
>
> John and Chris,
>
> It is not generally true that the average of the coverage values is
the
> base rate. I think it may sometimes be true, if there are no events
in edge
> neighborhoods without full coverage.
>
> I worked up the following example with a 4x4 neighborhood:
>
> 1 0 1 1
> 0 0 0 0
> 1 0 1 1
> 0 1 0 0
>
> The rate here is 7/16 = .4375
>
> Fractional coverage values for a 3x3 neighborhood are:
>
> 1/4 2/6 2/6 2/4
> 2/6 4/9 4/9 4/6
> 2/6 3/9 3/9 2/6
> 2/4 3/6 3/6 2/4
>
> The average of these values is 0.4149306
>
> I suppose if each fraction had the same denominator (e.g. no edges,
or all
> non events there) then we would add each event up for 9 different
> neighborhoods and then divide by 9. In that case, it should work.
>
> Tressa
>
>
> On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > I don't get negative values for AFSS but I do get values that are
less
> > than what I calculate for FSS. I approach the procedure for
computing
> all
> > the metrics such that everything is on the same "playing ground".
Thus,
> > F_RATE and O_RATE are only computed by the mask of defined grid
points on
> > *BOTH* the observation and forecast grids. When I process the
sample
> time
> > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> value.
> > Is that not the approach that I should follow? Were you able to
> > examine my script and raw data file I sent you?
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, February 20, 2018 8:20 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > Your email got me thinking. Thinking through some toy examples in
my
> > mind, I can see that probably yes, the mean of the fractional
coverage
> field
> > *should* be the same as the frequency of the event.
> >
> > I ran MET in the debugger and was able to stop at the point in the
logic
> > where this is computed. At that point, we're passing around
arrays of
> > fractional coverage field values as well as arrays of 0's and 1's
> > indicating whether the event did or did not occur at each grid
point.
> > Comparing the mean of these arrays, I find that they are close but
not
> > identical. Sometimes one is larger, sometimes the other.
> >
> > I can think of 2 reasons why they wouldn't be the same, but these
may not
> > fully explain it:
> >
> > (1) In Grid-Stat, the fractional coverage field is computed once
over the
> > full verification domain. Then, the masking regions are processed
as
> > subsets of these grid points. For a grid point that falls just
barely
> > inside that mask, around 1/2 of the grid points used to compute
the
> > fractional coverage value came from *OUTSIDE* that masking region.
The
> 0/1
> > arrays are only for points inside the current mask. So the mean
of the
> 0/1
> > array represents only the points inside the mask while the mean of
the
> > fractional coverage includes info from outside the mask.
> >
> > (2) When data values are missing in the forecast or observation
fields,
> > depending on how the user has set the vld_thresh config file
option, the
> > "denominator" used to compute the fractional coverage field value
could
> > change from grid point to grid point. That would likely throw off
the
> mean
> > of the fractional coverage field.
> >
> > Now what is the *CORRECT* way of computing the forecast and
observation
> > rates? In papers about this, I see references to the "base_rate"
but no
> > explicit equation.
> >
> > One very nice feature about the way MET is currently doing it is
that
> > F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> > Therefore, UFSS and AFSS remain constant as well. Using the other
> method,
> > O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood
size
> > since the number of points included from outside the mask would
change.
> >
> > You mentioned that you had issues with negative UFSS and AFSS
values.
> > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> >
> > Lots of little details to consider here!
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > John,
> > >
> > > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > > can't be calculated in the cases below? Can't the gridded data
still
> > > be read in to determine the relative frequency of the events,
> regardless
> > > whether grid_stat is used to compute fractional coverage fields?
I am
> > > basically doing that right now in some Python code I developed
when
> > > calculating Fractions Skill Score.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > I'm working on the interp.field option we discussed.
> > >
> > > I have a quick question for you. The NBRCNT line type that is
> computed
> > by
> > > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
> > > grid points over a masking region at which the "event" is
occurring in
> > the
> > > forecast and observation fields respectively.
> > >
> > > When grid_stat computes the fractional coverage field, we can
compute
> > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > fields,
> > > then we don't know what those ratio's are.
> > >
> > > - If interp.field = FCST, then O_RATE = NA.
> > > - If interp.field = OBS, then F_RATE = NA.
> > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > >
> > > Make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > John,
> > > >
> > > > That sounds like an excellent plan. Thanks for your
assistance and
> > > > trying to accommodate and make changes to the MET software so
> quickly.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > Just wanted to close the loop on this ticket. I'll try to get
it
> into
> > > the
> > > > next release due out next in March.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > ambiguity and confusion. I kind of thought that was the
procedure.
> > > > >
> > > > > When do you think that the next MET would be released with
the
> > > > > enhancement that we discussed for the neighborhood
statistics? Is
> it
> > > > > an easy or difficult update to the software?
> > > > >
> > > > > Thanks again,
> > > > > Chris
> > > > >
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Chris,
> > > > >
> > > > > It sounds like you understand how the fractional coverage
field is
> > > > > computed.
> > > > >
> > > > > - Read the raw field... probability values between 0 and 1
in your
> > > case.
> > > > > - Apply the categorical threshold to that data to generate a
field
> of
> > > 0's
> > > > > and 1's... >0.5 in your case.
> > > > > - For each grid point, replace the value at that grid point
with
> the
> > > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
> > > event
> > > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
> > > current
> > > > > point, so fractional coverage = 4/9.
> > > > > - Compare the forecast fractional coverage values to the
observed
> > > > > fractional coverage values to compute FSS.
> > > > >
> > > > > Yes, I do think it would be a good idea to talk to Craig
about how
> he
> > > > > configured MET for his work.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Getting back to your initial response, I was able to run
> grid_stat
> > > > > > with your changes in the attached configuration file to
get FSS
> > > output
> > > > > > (as well as the other statistics for the neighborhood type
> > results).
> > > > > >
> > > > > > However, I do have a question about the forecast
categorical
> > > threshold
> > > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > > > probabilities from the ensemble probabilities: What is the
actual
> > > > > > formula that MET uses to create the final field to
calculate FSS?
> > Is
> > > > > > it a strict neighborhood fractional coverage of ensemble
> > > probabilities
> > > > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > > > operation or does the threshold represent something else?
I know
> > in
> > > > > > Craig's paper, they were calling the last step a
neighborhood
> > average
> > > > > > of the ensemble probabilities to calculate NEP (since the
event
> had
> > > > > > already been defined at the grid point in determining the
> ensemble
> > > > > probability [i.e. likelihood of exceeding
> > > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig
> was
> > > > using
> > > > > > MET for his paper.
> > > > > >
> > > > > > Thanks again,
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > I was thinking about what changes would be needed to
compute FSS
> on
> > > > input
> > > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > > section,
> > > > > > just like we have in the “interp” section. It would be
set to
> > FCST,
> > > > OBS,
> > > > > > or BOTH (default) and would control the logic as to which
fields
> > > should
> > > > > be
> > > > > > passed through the logic for deriving fractional coverage
fields.
> > In
> > > > > your
> > > > > > case, you’d set it to OBS and we’d just use the raw input
> forecast
> > > > > > probability values.
> > > > > >
> > > > > > Would that logic make sense to you?
> > > > > >
> > > > > > Thanks
> > > > > > John
> > > > > >
> > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
> for
> > > > > > > adding the shape = circle option to MET.
> > > > > > >
> > > > > > > For your first question, no, there is not currently a
Gaussian
> > > > > > > smoother option for the "interp" section in Grid-Stat.
> > Currently,
> > > > the
> > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN (for
> > > > > > unweighted mean).
> > > > > > > There is a distance-weighted mean option for point data
where
> the
> > > > > > > weight is 1/distance squared. But that doesn't work for
> > Grid-Stat
> > > > > > > because we'd end up dividing by 0.
> > > > > > >
> > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > >
> > > > > > > Here's what we've done in the past with this SPC 40-km
> > > probabilities:
> > > > > > >
> > > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob
> =
> > > > > > > TRUE;" in the config file).
> > > > > > > (2) Use the "interp" section to define a smoothing
operation on
> > the
> > > > > > > observations:
> > > > > > > interp = {
> > > > > > > field = OBS;
> > > > > > > vld_thresh = 1.0;
> > > > > > > shape = CIRCLE;
> > > > > > > type = [
> > > > > > > { method = MAX; width = 11; }
> > > > > > > ];
> > > > > > > }
> > > > > > > Or set width to some circle diameter in grid units
that
> works
> > > out
> > > > > > > to 40km in the real world.
> > > > > > >
> > > > > > > In this example, we post-process the observations to
make them
> > > > > > > comparable the event for which the probability was
defined.
> > > > > > >
> > > > > > > But I think I understand what you'd like to do...
interpret as
> > the
> > > > > > > probability value at each grid point as if it was a
fractional
> > > > > > > coverage value. So don't apply a threshold and
neighborhood
> size
> > > to
> > > > > > > the forecast data at all. Just pass it directly to the
routine
> > > that
> > > > > > computes FSS.
> > > > > > >
> > > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > > Grid-Stat
> > > > > > > to do that in met-6.1. If you think this logic would be
> > generally
> > > > > > > useful, we could consider adding a flag to the config
file to
> > > enable
> > > > > > that logic.
> > > > > > >
> > > > > > > Sorry I don't have better answers for you.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >>
> > > > > > >> Hi John,
> > > > > > >>
> > > > > > >> Your suggestion for accomplishing #2 sounds promising.
I need
> > to
> > > > > think
> > > > > > >> the details through some. On a somewhat related note,
is
> > there a
> > > > > > Gaussian
> > > > > > >> smoother flag for smoothing the probabilistic fields?
I know
> > > that
> > > > > was
> > > > > > >> mentioned in Craig's paper and I have pursued that
option in
> the
> > > > past
> > > > > > >> (but just not using MET).
> > > > > > >>
> > > > > > >> I guess my basic question is can you apply a
neighborhood
> width
> > > that
> > > > > > >> just includes the grid point maybe because you already
have
> > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > Would
> > > > that
> > > > > > be a width of 1
> > > > > > >> or does that also include the nearest grid point? A
good
> > > example
> > > > of
> > > > > > this
> > > > > > >> would be a probabilistic forecast issued from the Storm
> > Prediction
> > > > > > >> Center where the probabilities implicitly assume a 40-
km
> radius
> > of
> > > > > > >> influence (so there is no need to compute a fractional
> coverage
> > of
> > > > the
> > > > > > event).
> > > > > > >>
> > > > > > >> Thanks again,
> > > > > > >> Chris
> > > > > > >>
> > > > > > >>
> > > > > > >> //SIGNED//
> > > > > > >>
> > > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > >> christopher.melick at us.af.mil>
> > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF ACC
> 16
> > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > > with
> > > > > > >> grid_stat on Neighborhood statistics
> > > > > > >>
> > > > > > >> Chris,
> > > > > > >>
> > > > > > >> (1) The default shape is a square since that's what MET
> > previously
> > > > > did.
> > > > > > >> The circle is the *new* option.
> > > > > > >>
> > > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
> > MET
> > > > by
> > > > > > >> doing the following...
> > > > > > >> - For each field separately (i.e. fcst and obs),
apply the
> > > > > > >> threshold
> > > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > > > >> - Apply the neighborhood width and shape to compute a
> > fractional
> > > > > > >> coverage value for each grid point.
> > > > > > >> - Compare the fcst and obs fractional coverage fields
to
> > compute
> > > > > FSS.
> > > > > > >>
> > > > > > >> But there are some other options that you may find
useful.
> The
> > > > > "interp"
> > > > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > > > operation.
> > > > > > >> After reading the raw input data, you can specify
if/how to
> > smooth
> > > > > > >> that data prior to computing stats. For example, since
you
> > > > mentioned
> > > > > > >> maximums, you could replace the value at each grid
point with
> > the
> > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > >>
> > > > > > >> interp = {
> > > > > > >> field = FCST;
> > > > > > >> vld_thresh = 1.0;
> > > > > > >> shape = SQUARE;
> > > > > > >>
> > > > > > >> type = [
> > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > > > points
> > > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean
of the
> 25
> > > > > > >> closest points
> > > > > > >> ];
> > > > > > >> }
> > > > > > >>
> > > > > > >> Using the example above, you'll get 3 times the amount
of
> output
> > > > from
> > > > > > MET.
> > > > > > >> The first smoother really is no smoothing at all. It
would
> run
> > on
> > > > > > >> the raw input data. The second smoother replaces the
value at
> > > each
> > > > > > >> grid point with the maximum of the 25 surrounding
points. The
> > > third
> > > > > > >> smoother replaces the value at each grid point with the
mean
> of
> > > > those
> > > > > > 25.
> > > > > > >>
> > > > > > >> I worked with Craig Schwartz when he was running MET
for that
> > > paper.
> > > > > > >> My recollection is that he was able to use the
configuration
> > > options
> > > > > > >> of Grid-Stat to accomplish the types of processing he
was
> after.
> > > > But
> > > > > > >> I don't recall all the details.
> > > > > > >>
> > > > > > >> If you explain exactly what you're trying to do, it may
be
> > > possible
> > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
> > via
> > > > RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83986
> > >
> > > > > > >> >
> > > > > > >> > Hi John,
> > > > > > >> >
> > > > > > >> > Your explanation and suggestions are very helpful and
do
> > explain
> > > > my
> > > > > > >> > problems quite substantially. I will have to give it
a try
> to
> > > see
> > > > > > >> > what happens.
> > > > > > >> >
> > > > > > >> > I do have a couple of other minor questions dealing
with how
> > the
> > > > > > >> > neighborhood verification is performed:
> > > > > > >> > 1.) I noticed in the documentation that there is a
flag for
> > > > > > >> > defining the shape of the neighborhood area (i.e.
circular
> or
> > > > > > >> > square). What is the default if you don't specify?
> > > > > > >> > 2.) In the literature, there are a couple of
different ways
> > to
> > > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
> > and
> > > > > > >> > Sobash 2017;
> > > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-
16-
> 0400.1
> > ).
> > > > I
> > > > > > >> > was curious is the only way to get the Fractions
Skill Score
> > > (FSS)
> > > > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > > > >> > fractional coverage of the threshold event? I have
> previously
> > > > > > >> > explored computing the neighborhood maximum value
within a
> > > radius
> > > > of
> > > > > > each grid point before calculating a probability.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Chris
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > //SIGNED//
> > > > > > >> >
> > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > Verification
> > > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > -----Original Message-----
> > > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > >> > christopher.melick at us.af.mil>
> > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > robert.craig.2 at us.af.mil>
> > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986] Help
> > > with
> > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > >> >
> > > > > > >> > Hi Chris,
> > > > > > >> >
> > > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > > >> > statistics in Grid-Stat.
> > > > > > >> >
> > > > > > >> > First, Grid-Stat definitely does have the ability to
> threshold
> > > the
> > > > > > >> StageIV
> > > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
> > not
> > > > > > needed.
> > > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
> > just
> > > > set:
> > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > >> >
> > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same
> as
> > > > >=0.1
> > > > > > >> inches.
> > > > > > >> >
> > > > > > >> > If for some reason you have data that's already 0's
and 1's,
> > > you'd
> > > > > > >> > just use a different threshold in the config file:
> > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > >> >
> > > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
> > event
> > > is
> > > > > > >> > occurring. Otherwise, it's not.
> > > > > > >> >
> > > > > > >> > Now on the to real issue. Looking at your config
file, I
> see
> > > that
> > > > > > >> you're
> > > > > > >> > processing probabilistic data. When "prob = TRUE",
MET only
> > > > > > >> > computes
> > > > > > >> the
> > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
> > why
> > > > > > >> > you're getting no output in the NBRCNT line type,
which is
> the
> > > one
> > > > > > >> > that
> > > > > > >> contains
> > > > > > >> > Fractions Skill Score (FSS).
> > > > > > >> >
> > > > > > >> > In the attached config file, I've made a few changes:
> > > > > > >> > - Set the obtype to indicate that you're verifying
against
> > > > StageIV
> > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > >> > - In the first we process QP010 as a
probability
> field.
> > > > Here
> > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation for
> > > > > > >> > defining
> > > > > > >> bins
> > > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > > >> > - In the second we process it as a field of
scalars.
> > Here
> > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > threshold
> > > > > > >> > I'm
> > > > > > >> using
> > > > > > >> > for defining events for fractions skill score.
> > > > > > >> > - Defined 2 corresponding obs.field entries to
verify the
> > > > forecast
> > > > > > >> data.
> > > > > > >> > In both the verifying threshold is >=2.54. This
assumes that
> > you
> > > > > > >> > pass
> > > > > > >> the
> > > > > > >> > raw StageIV data as input.
> > > > > > >> > - In the output_flag section, I turned off the line
types
> > that
> > > > > > >> > don't apply.
> > > > > > >> >
> > > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
> > > > > > >> > grabbing record number 19 from the input file, it's
not
> > > > > > >> > recommended. Instead,
> > > > > > >> it'd
> > > > > > >> > be better to figure out how to set the level string
so that
> > MET
> > > > > > >> > always grabs the same record, regardless if it's in
spot
> > number
> > > 19
> > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > >> >
> > > > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > John Halley Gotway
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> christopher.melick at us.af.mil
> > > via
> > > > RT
> > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > > >> > > Transaction: Ticket created by
> christopher.melick at us.af.mil
> > > > > > >> > > Queue: met_help
> > > > > > >> > > Subject: Help with grid_stat on Neighborhood
> statistics
> > > > > > >> > > Owner: Nobody
> > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > >> > > Status: new
> > > > > > >> > > Ticket <URL:
> > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Hi John,
> > > > > > >> > >
> > > > > > >> > > I am trying to run grid_stat in MET to calculate
> > Neighborhood
> > > > > > >> > > statistics for probability of precipitation from
some
> > ensemble
> > > > > > >> > > data with the observations
> > > > > > >> > > coming from Stage IV analyses. Unfortunately, I
am
> getting
> > > no
> > > > > > output
> > > > > > >> > > (just
> > > > > > >> > > headers) in many of the ASCII files from the
program when
> I
> > > run
> > > > > it.
> > > > > > >> > Could
> > > > > > >> > > you help me figure out what I am doing wrong? I
have
> > provided
> > > > > > >> > > the command line specifics I use as well as the
output log
> > > file
> > > > > > >> > > that is
> > > > > > >> > created when
> > > > > > >> > > running that command. I also have attached the
> > configuration
> > > > > file.
> > > > > > >> I
> > > > > > >> > am
> > > > > > >> > > having trouble using ftp to send the raw data from
any of
> > the
> > > > > > clients
> > > > > > >> > > available to us. I believe this restriction
occurred
> > > recently
> > > > > > since
> > > > > > >> Bob
> > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > externally
> > > > > > >> > > in the past.
> > > > > > >> > >
> > > > > > >> > > I should mention that I have pre-processed the
Stage IV
> > > analyses
> > > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> > > since
> > > > > > >> > > I was unclear as to whether MET was able to do that
on the
> > fly
> > > > > > >> > > when running
> > > > > > >> > grid_stat.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Chris
> > > > > > >> > >
> > > > > > >> > > Python command:
> > > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> '-v',
> > > '5]
> > > > > > >> > > Call(metcall)
> > > > > > >> > >
> > > > > > >> > > //SIGNED//
> > > > > > >> > >
> > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron (16
> > > > WS/WXEV)
> > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm
> (402)
> > > > > > >> > > 294-3693
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Tue Feb 27 11:58:56 2018
John,
I believe I have an idea of what some of the problem is when computing
AFSS in my particular case: When the original work on Fractions
Skill Score was published by Roberts and Lean (2008), it was designed
around the concept of a single deterministic model. Since then, that
statistic has been expanded to ensembles in a variety of ways to
determine neighborhood probabilities as well as expanded to other
probabilistic forecasts which implicitly incorporate spatial
uncertainty. Since the original binary forecast fields are sometimes
not available and/or possible, I don't know how the F_RATE fits into
these new paradigms for computing an upper level benchmark for FSS
(i.e. AFSS).
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, February 27, 2018 10:27 AM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with grid_stat on Neighborhood statistics
Chris,
I looked at your python code and don't see anything obviously wrong in
the equations you've used. However, your computation of AFSS won't
match what MET is computing for the reasons we've already discussed.
You're computing fcst_rate and obs_rate as an average of the
fractional coverage values.
MET computes them as a ratio of events occurring in the input fields.
As Tressa demonstrated, those will not necessarily result in the same
values.
Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute will not match the AFSS that MET computes.
Thanks,
John
On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Did you ever get the chance to take a look at my Python program and
> see if you notice anything that needs to be corrected? I'm still
not
> sure what to make of the values I obtained for AFSS and why they are
lower than FSS,
> especially at the grid-scale. Is this even possible?
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> Sent: Wednesday, February 21, 2018 3:47 PM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi John,
>
> You are correct --- I never received the message below. Are the
average
> coverage values what is used to compute the AFSS? I guess I am
still a
> little confused - I was just using the first example to compute the
> O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would help
> me in the example case I sent you because most events occurred in
the
> interior of the domain. Also, wouldn't edge effects be eliminated
if the
> vld_thresh is set to one in grid_stat since the entire neighborhood
must
> contain valid data?
>
> If we leave the whole neighborhood averaging aside, and just focus
on the
> computing AFSS at the grid scale (neighborhood = 1 grid point), then
my FSS
> is still greater than AFSS. Missing data points on the analysis
grid only
> result from different domain areas between that dataset and the
ensemble
> forecast system.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing,
> Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 21, 2018 2:59 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> Tressa sent an email on this issue, but she didn't include
> met_help at ucar.edu.
> So I don't think you received it. I've cut-and-pasted it below:
>
> John and Chris,
>
> It is not generally true that the average of the coverage values is
the
> base rate. I think it may sometimes be true, if there are no events
in edge
> neighborhoods without full coverage.
>
> I worked up the following example with a 4x4 neighborhood:
>
> 1 0 1 1
> 0 0 0 0
> 1 0 1 1
> 0 1 0 0
>
> The rate here is 7/16 = .4375
>
> Fractional coverage values for a 3x3 neighborhood are:
>
> 1/4 2/6 2/6 2/4
> 2/6 4/9 4/9 4/6
> 2/6 3/9 3/9 2/6
> 2/4 3/6 3/6 2/4
>
> The average of these values is 0.4149306
>
> I suppose if each fraction had the same denominator (e.g. no edges,
or all
> non events there) then we would add each event up for 9 different
> neighborhoods and then divide by 9. In that case, it should work.
>
> Tressa
>
>
> On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > I don't get negative values for AFSS but I do get values that are
less
> > than what I calculate for FSS. I approach the procedure for
computing
> all
> > the metrics such that everything is on the same "playing ground".
Thus,
> > F_RATE and O_RATE are only computed by the mask of defined grid
points on
> > *BOTH* the observation and forecast grids. When I process the
sample
> time
> > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> value.
> > Is that not the approach that I should follow? Were you able to
> > examine my script and raw data file I sent you?
> >
> > Thanks again,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, February 20, 2018 8:20 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > Your email got me thinking. Thinking through some toy examples in
my
> > mind, I can see that probably yes, the mean of the fractional
coverage
> field
> > *should* be the same as the frequency of the event.
> >
> > I ran MET in the debugger and was able to stop at the point in the
logic
> > where this is computed. At that point, we're passing around
arrays of
> > fractional coverage field values as well as arrays of 0's and 1's
> > indicating whether the event did or did not occur at each grid
point.
> > Comparing the mean of these arrays, I find that they are close but
not
> > identical. Sometimes one is larger, sometimes the other.
> >
> > I can think of 2 reasons why they wouldn't be the same, but these
may not
> > fully explain it:
> >
> > (1) In Grid-Stat, the fractional coverage field is computed once
over the
> > full verification domain. Then, the masking regions are processed
as
> > subsets of these grid points. For a grid point that falls just
barely
> > inside that mask, around 1/2 of the grid points used to compute
the
> > fractional coverage value came from *OUTSIDE* that masking region.
The
> 0/1
> > arrays are only for points inside the current mask. So the mean
of the
> 0/1
> > array represents only the points inside the mask while the mean of
the
> > fractional coverage includes info from outside the mask.
> >
> > (2) When data values are missing in the forecast or observation
fields,
> > depending on how the user has set the vld_thresh config file
option, the
> > "denominator" used to compute the fractional coverage field value
could
> > change from grid point to grid point. That would likely throw off
the
> mean
> > of the fractional coverage field.
> >
> > Now what is the *CORRECT* way of computing the forecast and
observation
> > rates? In papers about this, I see references to the "base_rate"
but no
> > explicit equation.
> >
> > One very nice feature about the way MET is currently doing it is
that
> > F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> > Therefore, UFSS and AFSS remain constant as well. Using the other
> method,
> > O_RATE, F_RATE, UFSS, and AFSS would change for each neighborhood
size
> > since the number of points included from outside the mask would
change.
> >
> > You mentioned that you had issues with negative UFSS and AFSS
values.
> > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> >
> > Lots of little details to consider here!
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > John,
> > >
> > > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > > can't be calculated in the cases below? Can't the gridded data
still
> > > be read in to determine the relative frequency of the events,
> regardless
> > > whether grid_stat is used to compute fractional coverage fields?
I am
> > > basically doing that right now in some Python code I developed
when
> > > calculating Fractions Skill Score.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > I'm working on the interp.field option we discussed.
> > >
> > > I have a quick question for you. The NBRCNT line type that is
> computed
> > by
> > > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio of
> > > grid points over a masking region at which the "event" is
occurring in
> > the
> > > forecast and observation fields respectively.
> > >
> > > When grid_stat computes the fractional coverage field, we can
compute
> > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > fields,
> > > then we don't know what those ratio's are.
> > >
> > > - If interp.field = FCST, then O_RATE = NA.
> > > - If interp.field = OBS, then F_RATE = NA.
> > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > >
> > > Make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > John,
> > > >
> > > > That sounds like an excellent plan. Thanks for your
assistance and
> > > > trying to accommodate and make changes to the MET software so
> quickly.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > Just wanted to close the loop on this ticket. I'll try to get
it
> into
> > > the
> > > > next release due out next in March.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > ambiguity and confusion. I kind of thought that was the
procedure.
> > > > >
> > > > > When do you think that the next MET would be released with
the
> > > > > enhancement that we discussed for the neighborhood
statistics? Is
> it
> > > > > an easy or difficult update to the software?
> > > > >
> > > > > Thanks again,
> > > > > Chris
> > > > >
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Chris,
> > > > >
> > > > > It sounds like you understand how the fractional coverage
field is
> > > > > computed.
> > > > >
> > > > > - Read the raw field... probability values between 0 and 1
in your
> > > case.
> > > > > - Apply the categorical threshold to that data to generate a
field
> of
> > > 0's
> > > > > and 1's... >0.5 in your case.
> > > > > - For each grid point, replace the value at that grid point
with
> the
> > > > > "fraction" of 1's in the neighborhood around that point...
e.g. the
> > > event
> > > > > occurs at 4 of the 9 grid points in a 3x3 square surrounding
the
> > > current
> > > > > point, so fractional coverage = 4/9.
> > > > > - Compare the forecast fractional coverage values to the
observed
> > > > > fractional coverage values to compute FSS.
> > > > >
> > > > > Yes, I do think it would be a good idea to talk to Craig
about how
> he
> > > > > configured MET for his work.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Getting back to your initial response, I was able to run
> grid_stat
> > > > > > with your changes in the attached configuration file to
get FSS
> > > output
> > > > > > (as well as the other statistics for the neighborhood type
> > results).
> > > > > >
> > > > > > However, I do have a question about the forecast
categorical
> > > threshold
> > > > > > (cat_thresh is set to >=0.5) used in determining the
neighborhood
> > > > > > probabilities from the ensemble probabilities: What is the
actual
> > > > > > formula that MET uses to create the final field to
calculate FSS?
> > Is
> > > > > > it a strict neighborhood fractional coverage of ensemble
> > > probabilities
> > > > > > above 0.5 in some set radius of influence? Or is it some
other
> > > > > > operation or does the threshold represent something else?
I know
> > in
> > > > > > Craig's paper, they were calling the last step a
neighborhood
> > average
> > > > > > of the ensemble probabilities to calculate NEP (since the
event
> had
> > > > > > already been defined at the grid point in determining the
> ensemble
> > > > > probability [i.e. likelihood of exceeding
> > > > > > 0.1" of precipitation]). Maybe, I should understand how
Craig
> was
> > > > using
> > > > > > MET for his paper.
> > > > > >
> > > > > > Thanks again,
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > I was thinking about what changes would be needed to
compute FSS
> on
> > > > input
> > > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > > section,
> > > > > > just like we have in the “interp” section. It would be
set to
> > FCST,
> > > > OBS,
> > > > > > or BOTH (default) and would control the logic as to which
fields
> > > should
> > > > > be
> > > > > > passed through the logic for deriving fractional coverage
fields.
> > In
> > > > > your
> > > > > > case, you’d set it to OBS and we’d just use the raw input
> forecast
> > > > > > probability values.
> > > > > >
> > > > > > Would that logic make sense to you?
> > > > > >
> > > > > > Thanks
> > > > > > John
> > > > > >
> > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
> for
> > > > > > > adding the shape = circle option to MET.
> > > > > > >
> > > > > > > For your first question, no, there is not currently a
Gaussian
> > > > > > > smoother option for the "interp" section in Grid-Stat.
> > Currently,
> > > > the
> > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN (for
> > > > > > unweighted mean).
> > > > > > > There is a distance-weighted mean option for point data
where
> the
> > > > > > > weight is 1/distance squared. But that doesn't work for
> > Grid-Stat
> > > > > > > because we'd end up dividing by 0.
> > > > > > >
> > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > >
> > > > > > > Here's what we've done in the past with this SPC 40-km
> > > probabilities:
> > > > > > >
> > > > > > > (1) Evaluate that field as a probability field (i.e. set
"prob
> =
> > > > > > > TRUE;" in the config file).
> > > > > > > (2) Use the "interp" section to define a smoothing
operation on
> > the
> > > > > > > observations:
> > > > > > > interp = {
> > > > > > > field = OBS;
> > > > > > > vld_thresh = 1.0;
> > > > > > > shape = CIRCLE;
> > > > > > > type = [
> > > > > > > { method = MAX; width = 11; }
> > > > > > > ];
> > > > > > > }
> > > > > > > Or set width to some circle diameter in grid units
that
> works
> > > out
> > > > > > > to 40km in the real world.
> > > > > > >
> > > > > > > In this example, we post-process the observations to
make them
> > > > > > > comparable the event for which the probability was
defined.
> > > > > > >
> > > > > > > But I think I understand what you'd like to do...
interpret as
> > the
> > > > > > > probability value at each grid point as if it was a
fractional
> > > > > > > coverage value. So don't apply a threshold and
neighborhood
> size
> > > to
> > > > > > > the forecast data at all. Just pass it directly to the
routine
> > > that
> > > > > > computes FSS.
> > > > > > >
> > > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > > Grid-Stat
> > > > > > > to do that in met-6.1. If you think this logic would be
> > generally
> > > > > > > useful, we could consider adding a flag to the config
file to
> > > enable
> > > > > > that logic.
> > > > > > >
> > > > > > > Sorry I don't have better answers for you.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >>
> > > > > > >> Hi John,
> > > > > > >>
> > > > > > >> Your suggestion for accomplishing #2 sounds promising.
I need
> > to
> > > > > think
> > > > > > >> the details through some. On a somewhat related note,
is
> > there a
> > > > > > Gaussian
> > > > > > >> smoother flag for smoothing the probabilistic fields?
I know
> > > that
> > > > > was
> > > > > > >> mentioned in Craig's paper and I have pursued that
option in
> the
> > > > past
> > > > > > >> (but just not using MET).
> > > > > > >>
> > > > > > >> I guess my basic question is can you apply a
neighborhood
> width
> > > that
> > > > > > >> just includes the grid point maybe because you already
have
> > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > Would
> > > > that
> > > > > > be a width of 1
> > > > > > >> or does that also include the nearest grid point? A
good
> > > example
> > > > of
> > > > > > this
> > > > > > >> would be a probabilistic forecast issued from the Storm
> > Prediction
> > > > > > >> Center where the probabilities implicitly assume a 40-
km
> radius
> > of
> > > > > > >> influence (so there is no need to compute a fractional
> coverage
> > of
> > > > the
> > > > > > event).
> > > > > > >>
> > > > > > >> Thanks again,
> > > > > > >> Chris
> > > > > > >>
> > > > > > >>
> > > > > > >> //SIGNED//
> > > > > > >>
> > > > > > >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > >> christopher.melick at us.af.mil>
> > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF ACC
> 16
> > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > > with
> > > > > > >> grid_stat on Neighborhood statistics
> > > > > > >>
> > > > > > >> Chris,
> > > > > > >>
> > > > > > >> (1) The default shape is a square since that's what MET
> > previously
> > > > > did.
> > > > > > >> The circle is the *new* option.
> > > > > > >>
> > > > > > >> (2) For this one, the short answer is yes. FSS is
computed in
> > MET
> > > > by
> > > > > > >> doing the following...
> > > > > > >> - For each field separately (i.e. fcst and obs),
apply the
> > > > > > >> threshold
> > > > > > >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> > > > > > >> - Apply the neighborhood width and shape to compute a
> > fractional
> > > > > > >> coverage value for each grid point.
> > > > > > >> - Compare the fcst and obs fractional coverage fields
to
> > compute
> > > > > FSS.
> > > > > > >>
> > > > > > >> But there are some other options that you may find
useful.
> The
> > > > > "interp"
> > > > > > >> section in the config file is used in Grid-Stat as a
smoothing
> > > > > > operation.
> > > > > > >> After reading the raw input data, you can specify
if/how to
> > smooth
> > > > > > >> that data prior to computing stats. For example, since
you
> > > > mentioned
> > > > > > >> maximums, you could replace the value at each grid
point with
> > the
> > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > >>
> > > > > > >> interp = {
> > > > > > >> field = FCST;
> > > > > > >> vld_thresh = 1.0;
> > > > > > >> shape = SQUARE;
> > > > > > >>
> > > > > > >> type = [
> > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
smoothing
> > > > > > >> { method = MAX; width = 5; }, // i.e. max of 25
closest
> > > > points
> > > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean
of the
> 25
> > > > > > >> closest points
> > > > > > >> ];
> > > > > > >> }
> > > > > > >>
> > > > > > >> Using the example above, you'll get 3 times the amount
of
> output
> > > > from
> > > > > > MET.
> > > > > > >> The first smoother really is no smoothing at all. It
would
> run
> > on
> > > > > > >> the raw input data. The second smoother replaces the
value at
> > > each
> > > > > > >> grid point with the maximum of the 25 surrounding
points. The
> > > third
> > > > > > >> smoother replaces the value at each grid point with the
mean
> of
> > > > those
> > > > > > 25.
> > > > > > >>
> > > > > > >> I worked with Craig Schwartz when he was running MET
for that
> > > paper.
> > > > > > >> My recollection is that he was able to use the
configuration
> > > options
> > > > > > >> of Grid-Stat to accomplish the types of processing he
was
> after.
> > > > But
> > > > > > >> I don't recall all the details.
> > > > > > >>
> > > > > > >> If you explain exactly what you're trying to do, it may
be
> > > possible
> > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
christopher.melick at us.af.mil
> > via
> > > > RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83986
> > >
> > > > > > >> >
> > > > > > >> > Hi John,
> > > > > > >> >
> > > > > > >> > Your explanation and suggestions are very helpful and
do
> > explain
> > > > my
> > > > > > >> > problems quite substantially. I will have to give it
a try
> to
> > > see
> > > > > > >> > what happens.
> > > > > > >> >
> > > > > > >> > I do have a couple of other minor questions dealing
with how
> > the
> > > > > > >> > neighborhood verification is performed:
> > > > > > >> > 1.) I noticed in the documentation that there is a
flag for
> > > > > > >> > defining the shape of the neighborhood area (i.e.
circular
> or
> > > > > > >> > square). What is the default if you don't specify?
> > > > > > >> > 2.) In the literature, there are a couple of
different ways
> > to
> > > > > > >> > handle generating neighborhood probabilities (e.g.,
Schwartz
> > and
> > > > > > >> > Sobash 2017;
> > > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-
16-
> 0400.1
> > ).
> > > > I
> > > > > > >> > was curious is the only way to get the Fractions
Skill Score
> > > (FSS)
> > > > > > >> > output is to define a neighborhood in which MET
calculates a
> > > > > > >> > fractional coverage of the threshold event? I have
> previously
> > > > > > >> > explored computing the neighborhood maximum value
within a
> > > radius
> > > > of
> > > > > > each grid point before calculating a probability.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Chris
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > //SIGNED//
> > > > > > >> >
> > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > Verification
> > > > > > >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > -----Original Message-----
> > > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > >> > christopher.melick at us.af.mil>
> > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > robert.craig.2 at us.af.mil>
> > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986] Help
> > > with
> > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > >> >
> > > > > > >> > Hi Chris,
> > > > > > >> >
> > > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > > >> > statistics in Grid-Stat.
> > > > > > >> >
> > > > > > >> > First, Grid-Stat definitely does have the ability to
> threshold
> > > the
> > > > > > >> StageIV
> > > > > > >> > data on the fly. So pre-processing to 0's and 1's
first is
> > not
> > > > > > needed.
> > > > > > >> > Instead, in the "obs" dictionary of the config file,
you'd
> > just
> > > > set:
> > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > >> >
> > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is the
same
> as
> > > > >=0.1
> > > > > > >> inches.
> > > > > > >> >
> > > > > > >> > If for some reason you have data that's already 0's
and 1's,
> > > you'd
> > > > > > >> > just use a different threshold in the config file:
> > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > >> >
> > > > > > >> > This tells MET that anywhere you see a 1 in the data,
the
> > event
> > > is
> > > > > > >> > occurring. Otherwise, it's not.
> > > > > > >> >
> > > > > > >> > Now on the to real issue. Looking at your config
file, I
> see
> > > that
> > > > > > >> you're
> > > > > > >> > processing probabilistic data. When "prob = TRUE",
MET only
> > > > > > >> > computes
> > > > > > >> the
> > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
That's
> > why
> > > > > > >> > you're getting no output in the NBRCNT line type,
which is
> the
> > > one
> > > > > > >> > that
> > > > > > >> contains
> > > > > > >> > Fractions Skill Score (FSS).
> > > > > > >> >
> > > > > > >> > In the attached config file, I've made a few changes:
> > > > > > >> > - Set the obtype to indicate that you're verifying
against
> > > > StageIV
> > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > >> > - In the first we process QP010 as a
probability
> field.
> > > > Here
> > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation for
> > > > > > >> > defining
> > > > > > >> bins
> > > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > > >> > - In the second we process it as a field of
scalars.
> > Here
> > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > threshold
> > > > > > >> > I'm
> > > > > > >> using
> > > > > > >> > for defining events for fractions skill score.
> > > > > > >> > - Defined 2 corresponding obs.field entries to
verify the
> > > > forecast
> > > > > > >> data.
> > > > > > >> > In both the verifying threshold is >=2.54. This
assumes that
> > you
> > > > > > >> > pass
> > > > > > >> the
> > > > > > >> > raw StageIV data as input.
> > > > > > >> > - In the output_flag section, I turned off the line
types
> > that
> > > > > > >> > don't apply.
> > > > > > >> >
> > > > > > >> > Lastly, while setting "level = [ "R19" ];" works as a
way of
> > > > > > >> > grabbing record number 19 from the input file, it's
not
> > > > > > >> > recommended. Instead,
> > > > > > >> it'd
> > > > > > >> > be better to figure out how to set the level string
so that
> > MET
> > > > > > >> > always grabs the same record, regardless if it's in
spot
> > number
> > > 19
> > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > >> >
> > > > > > >> > Hope this helps clarify. What other questions do you
have?
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > John Halley Gotway
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> christopher.melick at us.af.mil
> > > via
> > > > RT
> > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > > >> > > Transaction: Ticket created by
> christopher.melick at us.af.mil
> > > > > > >> > > Queue: met_help
> > > > > > >> > > Subject: Help with grid_stat on Neighborhood
> statistics
> > > > > > >> > > Owner: Nobody
> > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > >> > > Status: new
> > > > > > >> > > Ticket <URL:
> > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Hi John,
> > > > > > >> > >
> > > > > > >> > > I am trying to run grid_stat in MET to calculate
> > Neighborhood
> > > > > > >> > > statistics for probability of precipitation from
some
> > ensemble
> > > > > > >> > > data with the observations
> > > > > > >> > > coming from Stage IV analyses. Unfortunately, I
am
> getting
> > > no
> > > > > > output
> > > > > > >> > > (just
> > > > > > >> > > headers) in many of the ASCII files from the
program when
> I
> > > run
> > > > > it.
> > > > > > >> > Could
> > > > > > >> > > you help me figure out what I am doing wrong? I
have
> > provided
> > > > > > >> > > the command line specifics I use as well as the
output log
> > > file
> > > > > > >> > > that is
> > > > > > >> > created when
> > > > > > >> > > running that command. I also have attached the
> > configuration
> > > > > file.
> > > > > > >> I
> > > > > > >> > am
> > > > > > >> > > having trouble using ftp to send the raw data from
any of
> > the
> > > > > > clients
> > > > > > >> > > available to us. I believe this restriction
occurred
> > > recently
> > > > > > since
> > > > > > >> Bob
> > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > externally
> > > > > > >> > > in the past.
> > > > > > >> > >
> > > > > > >> > > I should mention that I have pre-processed the
Stage IV
> > > analyses
> > > > > > >> > > already to be either 0's or 1's for the threshold
(>=0.1")
> > > since
> > > > > > >> > > I was unclear as to whether MET was able to do that
on the
> > fly
> > > > > > >> > > when running
> > > > > > >> > grid_stat.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Chris
> > > > > > >> > >
> > > > > > >> > > Python command:
> > > > > > >> > > Metcall = [metpath + 'grid_stat',
'grib.2017091422.0006',
> > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> '-v',
> > > '5]
> > > > > > >> > > Call(metcall)
> > > > > > >> > >
> > > > > > >> > > //SIGNED//
> > > > > > >> > >
> > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron (16
> > > > WS/WXEV)
> > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm
> (402)
> > > > > > >> > > 294-3693
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Jun 06 11:31:30 2018
Hi Chris,
I went hunting through the met-help tickets and found this one that we
discussed during the poster session yesterday.
I've attached a PDF of the development ticket from JiRA which
describes the
functionality we added in met-7.0 as a result of this question.
Unfortunately though, I understand that you won't be able to make use
of
met-7.0 at the AF until we've addressed the security issues through
Fortify.
Thanks,
John
On Tue, Feb 27, 2018 at 11:58 AM, christopher.melick at us.af.mil via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> I believe I have an idea of what some of the problem is when
computing
> AFSS in my particular case: When the original work on Fractions
Skill
> Score was published by Roberts and Lean (2008), it was designed
around the
> concept of a single deterministic model. Since then, that
statistic has
> been expanded to ensembles in a variety of ways to determine
neighborhood
> probabilities as well as expanded to other probabilistic forecasts
which
> implicitly incorporate spatial uncertainty. Since the original
binary
> forecast fields are sometimes not available and/or possible, I don't
know
> how the F_RATE fits into these new paradigms for computing an upper
level
> benchmark for FSS (i.e. AFSS).
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 27, 2018 10:27 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> I looked at your python code and don't see anything obviously wrong
in the
> equations you've used. However, your computation of AFSS won't
match what
> MET is computing for the reasons we've already discussed. You're
computing
> fcst_rate and obs_rate as an average of the fractional coverage
values.
> MET computes them as a ratio of events occurring in the input
fields. As
> Tressa demonstrated, those will not necessarily result in the same
values.
> Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute
> will not match the AFSS that MET computes.
>
> Thanks,
> John
>
> On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Did you ever get the chance to take a look at my Python program
and
> > see if you notice anything that needs to be corrected? I'm still
not
> > sure what to make of the values I obtained for AFSS and why they
are
> lower than FSS,
> > especially at the grid-scale. Is this even possible?
> >
> > Thanks,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> > Sent: Wednesday, February 21, 2018 3:47 PM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi John,
> >
> > You are correct --- I never received the message below. Are the
average
> > coverage values what is used to compute the AFSS? I guess I am
still a
> > little confused - I was just using the first example to compute
the
> > O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would
> help
> > me in the example case I sent you because most events occurred in
the
> > interior of the domain. Also, wouldn't edge effects be eliminated
if the
> > vld_thresh is set to one in grid_stat since the entire
neighborhood must
> > contain valid data?
> >
> > If we leave the whole neighborhood averaging aside, and just focus
on the
> > computing AFSS at the grid scale (neighborhood = 1 grid point),
then my
> FSS
> > is still greater than AFSS. Missing data points on the analysis
grid
> only
> > result from different domain areas between that dataset and the
ensemble
> > forecast system.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing,
> > Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 21, 2018 2:59 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > Tressa sent an email on this issue, but she didn't include
> > met_help at ucar.edu.
> > So I don't think you received it. I've cut-and-pasted it below:
> >
> > John and Chris,
> >
> > It is not generally true that the average of the coverage values
is the
> > base rate. I think it may sometimes be true, if there are no
events in
> edge
> > neighborhoods without full coverage.
> >
> > I worked up the following example with a 4x4 neighborhood:
> >
> > 1 0 1 1
> > 0 0 0 0
> > 1 0 1 1
> > 0 1 0 0
> >
> > The rate here is 7/16 = .4375
> >
> > Fractional coverage values for a 3x3 neighborhood are:
> >
> > 1/4 2/6 2/6 2/4
> > 2/6 4/9 4/9 4/6
> > 2/6 3/9 3/9 2/6
> > 2/4 3/6 3/6 2/4
> >
> > The average of these values is 0.4149306
> >
> > I suppose if each fraction had the same denominator (e.g. no
edges, or
> all
> > non events there) then we would add each event up for 9 different
> > neighborhoods and then divide by 9. In that case, it should work.
> >
> > Tressa
> >
> >
> > On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > I don't get negative values for AFSS but I do get values that
are less
> > > than what I calculate for FSS. I approach the procedure for
computing
> > all
> > > the metrics such that everything is on the same "playing
ground".
> Thus,
> > > F_RATE and O_RATE are only computed by the mask of defined grid
points
> on
> > > *BOTH* the observation and forecast grids. When I process the
sample
> > time
> > > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> > value.
> > > Is that not the approach that I should follow? Were you able
to
> > > examine my script and raw data file I sent you?
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, February 20, 2018 8:20 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Your email got me thinking. Thinking through some toy examples
in my
> > > mind, I can see that probably yes, the mean of the fractional
coverage
> > field
> > > *should* be the same as the frequency of the event.
> > >
> > > I ran MET in the debugger and was able to stop at the point in
the
> logic
> > > where this is computed. At that point, we're passing around
arrays of
> > > fractional coverage field values as well as arrays of 0's and
1's
> > > indicating whether the event did or did not occur at each grid
point.
> > > Comparing the mean of these arrays, I find that they are close
but not
> > > identical. Sometimes one is larger, sometimes the other.
> > >
> > > I can think of 2 reasons why they wouldn't be the same, but
these may
> not
> > > fully explain it:
> > >
> > > (1) In Grid-Stat, the fractional coverage field is computed once
over
> the
> > > full verification domain. Then, the masking regions are
processed as
> > > subsets of these grid points. For a grid point that falls just
barely
> > > inside that mask, around 1/2 of the grid points used to compute
the
> > > fractional coverage value came from *OUTSIDE* that masking
region. The
> > 0/1
> > > arrays are only for points inside the current mask. So the mean
of the
> > 0/1
> > > array represents only the points inside the mask while the mean
of the
> > > fractional coverage includes info from outside the mask.
> > >
> > > (2) When data values are missing in the forecast or observation
fields,
> > > depending on how the user has set the vld_thresh config file
option,
> the
> > > "denominator" used to compute the fractional coverage field
value could
> > > change from grid point to grid point. That would likely throw
off the
> > mean
> > > of the fractional coverage field.
> > >
> > > Now what is the *CORRECT* way of computing the forecast and
observation
> > > rates? In papers about this, I see references to the
"base_rate" but
> no
> > > explicit equation.
> > >
> > > One very nice feature about the way MET is currently doing it is
that
> > > F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> > > Therefore, UFSS and AFSS remain constant as well. Using the
other
> > method,
> > > O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood size
> > > since the number of points included from outside the mask would
change.
> > >
> > > You mentioned that you had issues with negative UFSS and AFSS
values.
> > > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> > >
> > > Lots of little details to consider here!
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > John,
> > > >
> > > > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > > > can't be calculated in the cases below? Can't the gridded
data still
> > > > be read in to determine the relative frequency of the events,
> > regardless
> > > > whether grid_stat is used to compute fractional coverage
fields? I
> am
> > > > basically doing that right now in some Python code I developed
when
> > > > calculating Fractions Skill Score.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Hi Chris,
> > > >
> > > > I'm working on the interp.field option we discussed.
> > > >
> > > > I have a quick question for you. The NBRCNT line type that
is
> > computed
> > > by
> > > > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio
> of
> > > > grid points over a masking region at which the "event" is
occurring
> in
> > > the
> > > > forecast and observation fields respectively.
> > > >
> > > > When grid_stat computes the fractional coverage field, we can
compute
> > > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > > fields,
> > > > then we don't know what those ratio's are.
> > > >
> > > > - If interp.field = FCST, then O_RATE = NA.
> > > > - If interp.field = OBS, then F_RATE = NA.
> > > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > > >
> > > > Make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > John,
> > > > >
> > > > > That sounds like an excellent plan. Thanks for your
assistance and
> > > > > trying to accommodate and make changes to the MET software
so
> > quickly.
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Chris,
> > > > >
> > > > > Just wanted to close the loop on this ticket. I'll try to
get it
> > into
> > > > the
> > > > > next release due out next in March.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > > ambiguity and confusion. I kind of thought that was the
> procedure.
> > > > > >
> > > > > > When do you think that the next MET would be released with
the
> > > > > > enhancement that we discussed for the neighborhood
statistics?
> Is
> > it
> > > > > > an easy or difficult update to the software?
> > > > > >
> > > > > > Thanks again,
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Chris,
> > > > > >
> > > > > > It sounds like you understand how the fractional coverage
field
> is
> > > > > > computed.
> > > > > >
> > > > > > - Read the raw field... probability values between 0 and 1
in
> your
> > > > case.
> > > > > > - Apply the categorical threshold to that data to generate
a
> field
> > of
> > > > 0's
> > > > > > and 1's... >0.5 in your case.
> > > > > > - For each grid point, replace the value at that grid
point with
> > the
> > > > > > "fraction" of 1's in the neighborhood around that point...
e.g.
> the
> > > > event
> > > > > > occurs at 4 of the 9 grid points in a 3x3 square
surrounding the
> > > > current
> > > > > > point, so fractional coverage = 4/9.
> > > > > > - Compare the forecast fractional coverage values to the
observed
> > > > > > fractional coverage values to compute FSS.
> > > > > >
> > > > > > Yes, I do think it would be a good idea to talk to Craig
about
> how
> > he
> > > > > > configured MET for his work.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 8, 2018 at 2:09 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Getting back to your initial response, I was able to run
> > grid_stat
> > > > > > > with your changes in the attached configuration file to
get FSS
> > > > output
> > > > > > > (as well as the other statistics for the neighborhood
type
> > > results).
> > > > > > >
> > > > > > > However, I do have a question about the forecast
categorical
> > > > threshold
> > > > > > > (cat_thresh is set to >=0.5) used in determining the
> neighborhood
> > > > > > > probabilities from the ensemble probabilities: What is
the
> actual
> > > > > > > formula that MET uses to create the final field to
calculate
> FSS?
> > > Is
> > > > > > > it a strict neighborhood fractional coverage of ensemble
> > > > probabilities
> > > > > > > above 0.5 in some set radius of influence? Or is it
some other
> > > > > > > operation or does the threshold represent something
else? I
> know
> > > in
> > > > > > > Craig's paper, they were calling the last step a
neighborhood
> > > average
> > > > > > > of the ensemble probabilities to calculate NEP (since
the event
> > had
> > > > > > > already been defined at the grid point in determining
the
> > ensemble
> > > > > > probability [i.e. likelihood of exceeding
> > > > > > > 0.1" of precipitation]). Maybe, I should understand
how Craig
> > was
> > > > > using
> > > > > > > MET for his paper.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > > //SIGNED//
> > > > > > >
> > > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > christopher.melick at us.af.mil>
> > > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil
> > > >
> > > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > with
> > > > > > > grid_stat on Neighborhood statistics
> > > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > I was thinking about what changes would be needed to
compute
> FSS
> > on
> > > > > input
> > > > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > > > section,
> > > > > > > just like we have in the “interp” section. It would be
set to
> > > FCST,
> > > > > OBS,
> > > > > > > or BOTH (default) and would control the logic as to
which
> fields
> > > > should
> > > > > > be
> > > > > > > passed through the logic for deriving fractional
coverage
> fields.
> > > In
> > > > > > your
> > > > > > > case, you’d set it to OBS and we’d just use the raw
input
> > forecast
> > > > > > > probability values.
> > > > > > >
> > > > > > > Would that logic make sense to you?
> > > > > > >
> > > > > > > Thanks
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
> > for
> > > > > > > > adding the shape = circle option to MET.
> > > > > > > >
> > > > > > > > For your first question, no, there is not currently a
> Gaussian
> > > > > > > > smoother option for the "interp" section in Grid-Stat.
> > > Currently,
> > > > > the
> > > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN
> (for
> > > > > > > unweighted mean).
> > > > > > > > There is a distance-weighted mean option for point
data where
> > the
> > > > > > > > weight is 1/distance squared. But that doesn't work
for
> > > Grid-Stat
> > > > > > > > because we'd end up dividing by 0.
> > > > > > > >
> > > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > > >
> > > > > > > > Here's what we've done in the past with this SPC 40-km
> > > > probabilities:
> > > > > > > >
> > > > > > > > (1) Evaluate that field as a probability field (i.e.
set
> "prob
> > =
> > > > > > > > TRUE;" in the config file).
> > > > > > > > (2) Use the "interp" section to define a smoothing
operation
> on
> > > the
> > > > > > > > observations:
> > > > > > > > interp = {
> > > > > > > > field = OBS;
> > > > > > > > vld_thresh = 1.0;
> > > > > > > > shape = CIRCLE;
> > > > > > > > type = [
> > > > > > > > { method = MAX; width = 11; }
> > > > > > > > ];
> > > > > > > > }
> > > > > > > > Or set width to some circle diameter in grid units
that
> > works
> > > > out
> > > > > > > > to 40km in the real world.
> > > > > > > >
> > > > > > > > In this example, we post-process the observations to
make
> them
> > > > > > > > comparable the event for which the probability was
defined.
> > > > > > > >
> > > > > > > > But I think I understand what you'd like to do...
interpret
> as
> > > the
> > > > > > > > probability value at each grid point as if it was a
> fractional
> > > > > > > > coverage value. So don't apply a threshold and
neighborhood
> > size
> > > > to
> > > > > > > > the forecast data at all. Just pass it directly to
the
> routine
> > > > that
> > > > > > > computes FSS.
> > > > > > > >
> > > > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > > > Grid-Stat
> > > > > > > > to do that in met-6.1. If you think this logic would
be
> > > generally
> > > > > > > > useful, we could consider adding a flag to the config
file to
> > > > enable
> > > > > > > that logic.
> > > > > > > >
> > > > > > > > Sorry I don't have better answers for you.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
> > via
> > > > RT
> > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >>
> > > > > > > >> <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83986
> > >
> > > > > > > >>
> > > > > > > >> Hi John,
> > > > > > > >>
> > > > > > > >> Your suggestion for accomplishing #2 sounds
promising. I
> need
> > > to
> > > > > > think
> > > > > > > >> the details through some. On a somewhat related
note, is
> > > there a
> > > > > > > Gaussian
> > > > > > > >> smoother flag for smoothing the probabilistic fields?
I
> know
> > > > that
> > > > > > was
> > > > > > > >> mentioned in Craig's paper and I have pursued that
option in
> > the
> > > > > past
> > > > > > > >> (but just not using MET).
> > > > > > > >>
> > > > > > > >> I guess my basic question is can you apply a
neighborhood
> > width
> > > > that
> > > > > > > >> just includes the grid point maybe because you
already have
> > > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > > Would
> > > > > that
> > > > > > > be a width of 1
> > > > > > > >> or does that also include the nearest grid point?
A good
> > > > example
> > > > > of
> > > > > > > this
> > > > > > > >> would be a probabilistic forecast issued from the
Storm
> > > Prediction
> > > > > > > >> Center where the probabilities implicitly assume a
40-km
> > radius
> > > of
> > > > > > > >> influence (so there is no need to compute a
fractional
> > coverage
> > > of
> > > > > the
> > > > > > > event).
> > > > > > > >>
> > > > > > > >> Thanks again,
> > > > > > > >> Chris
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> //SIGNED//
> > > > > > > >>
> > > > > > > >> Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > Verification
> > > > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> -----Original Message-----
> > > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > >> christopher.melick at us.af.mil>
> > > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF
> ACC
> > 16
> > > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> > Help
> > > > > with
> > > > > > > >> grid_stat on Neighborhood statistics
> > > > > > > >>
> > > > > > > >> Chris,
> > > > > > > >>
> > > > > > > >> (1) The default shape is a square since that's what
MET
> > > previously
> > > > > > did.
> > > > > > > >> The circle is the *new* option.
> > > > > > > >>
> > > > > > > >> (2) For this one, the short answer is yes. FSS is
computed
> in
> > > MET
> > > > > by
> > > > > > > >> doing the following...
> > > > > > > >> - For each field separately (i.e. fcst and obs),
apply the
> > > > > > > >> threshold
> > > > > > > >> (cat_thresh) to convert the input data to a 0/1
bitmap.
> > > > > > > >> - Apply the neighborhood width and shape to compute
a
> > > fractional
> > > > > > > >> coverage value for each grid point.
> > > > > > > >> - Compare the fcst and obs fractional coverage
fields to
> > > compute
> > > > > > FSS.
> > > > > > > >>
> > > > > > > >> But there are some other options that you may find
useful.
> > The
> > > > > > "interp"
> > > > > > > >> section in the config file is used in Grid-Stat as a
> smoothing
> > > > > > > operation.
> > > > > > > >> After reading the raw input data, you can specify
if/how to
> > > smooth
> > > > > > > >> that data prior to computing stats. For example,
since you
> > > > > mentioned
> > > > > > > >> maximums, you could replace the value at each grid
point
> with
> > > the
> > > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > > >>
> > > > > > > >> interp = {
> > > > > > > >> field = FCST;
> > > > > > > >> vld_thresh = 1.0;
> > > > > > > >> shape = SQUARE;
> > > > > > > >>
> > > > > > > >> type = [
> > > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
> smoothing
> > > > > > > >> { method = MAX; width = 5; }, // i.e. max of
25
> closest
> > > > > points
> > > > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean
of the
> > 25
> > > > > > > >> closest points
> > > > > > > >> ];
> > > > > > > >> }
> > > > > > > >>
> > > > > > > >> Using the example above, you'll get 3 times the
amount of
> > output
> > > > > from
> > > > > > > MET.
> > > > > > > >> The first smoother really is no smoothing at all. It
would
> > run
> > > on
> > > > > > > >> the raw input data. The second smoother replaces the
value
> at
> > > > each
> > > > > > > >> grid point with the maximum of the 25 surrounding
points.
> The
> > > > third
> > > > > > > >> smoother replaces the value at each grid point with
the mean
> > of
> > > > > those
> > > > > > > 25.
> > > > > > > >>
> > > > > > > >> I worked with Craig Schwartz when he was running MET
for
> that
> > > > paper.
> > > > > > > >> My recollection is that he was able to use the
configuration
> > > > options
> > > > > > > >> of Grid-Stat to accomplish the types of processing he
was
> > after.
> > > > > But
> > > > > > > >> I don't recall all the details.
> > > > > > > >>
> > > > > > > >> If you explain exactly what you're trying to do, it
may be
> > > > possible
> > > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
> christopher.melick at us.af.mil
> > > via
> > > > > RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=83986
> > > >
> > > > > > > >> >
> > > > > > > >> > Hi John,
> > > > > > > >> >
> > > > > > > >> > Your explanation and suggestions are very helpful
and do
> > > explain
> > > > > my
> > > > > > > >> > problems quite substantially. I will have to give
it a
> try
> > to
> > > > see
> > > > > > > >> > what happens.
> > > > > > > >> >
> > > > > > > >> > I do have a couple of other minor questions dealing
with
> how
> > > the
> > > > > > > >> > neighborhood verification is performed:
> > > > > > > >> > 1.) I noticed in the documentation that there is a
flag
> for
> > > > > > > >> > defining the shape of the neighborhood area (i.e.
circular
> > or
> > > > > > > >> > square). What is the default if you don't specify?
> > > > > > > >> > 2.) In the literature, there are a couple of
different
> ways
> > > to
> > > > > > > >> > handle generating neighborhood probabilities (e.g.,
> Schwartz
> > > and
> > > > > > > >> > Sobash 2017;
> > > > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-
D-16-
> > 0400.1
> > > ).
> > > > > I
> > > > > > > >> > was curious is the only way to get the Fractions
Skill
> Score
> > > > (FSS)
> > > > > > > >> > output is to define a neighborhood in which MET
> calculates a
> > > > > > > >> > fractional coverage of the threshold event? I have
> > previously
> > > > > > > >> > explored computing the neighborhood maximum value
within a
> > > > radius
> > > > > of
> > > > > > > each grid point before calculating a probability.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chris
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > //SIGNED//
> > > > > > > >> >
> > > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > Verification
> > > > > > > >> > Exploitation Team 16th Weather Squadron (16
WS/WXEV) 557th
> > > > Weather
> > > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > -----Original Message-----
> > > > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> ]
> > > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<
> > > > > > > >> > christopher.melick at us.af.mil>
> > > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > > robert.craig.2 at us.af.mil>
> > > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > > with
> > > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > > >> >
> > > > > > > >> > Hi Chris,
> > > > > > > >> >
> > > > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > > > >> > statistics in Grid-Stat.
> > > > > > > >> >
> > > > > > > >> > First, Grid-Stat definitely does have the ability
to
> > threshold
> > > > the
> > > > > > > >> StageIV
> > > > > > > >> > data on the fly. So pre-processing to 0's and 1's
first
> is
> > > not
> > > > > > > needed.
> > > > > > > >> > Instead, in the "obs" dictionary of the config
file, you'd
> > > just
> > > > > set:
> > > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > > >> >
> > > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is
the same
> > as
> > > > > >=0.1
> > > > > > > >> inches.
> > > > > > > >> >
> > > > > > > >> > If for some reason you have data that's already 0's
and
> 1's,
> > > > you'd
> > > > > > > >> > just use a different threshold in the config file:
> > > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > > >> >
> > > > > > > >> > This tells MET that anywhere you see a 1 in the
data, the
> > > event
> > > > is
> > > > > > > >> > occurring. Otherwise, it's not.
> > > > > > > >> >
> > > > > > > >> > Now on the to real issue. Looking at your config
file, I
> > see
> > > > that
> > > > > > > >> you're
> > > > > > > >> > processing probabilistic data. When "prob = TRUE",
MET
> only
> > > > > > > >> > computes
> > > > > > > >> the
> > > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
> That's
> > > why
> > > > > > > >> > you're getting no output in the NBRCNT line type,
which is
> > the
> > > > one
> > > > > > > >> > that
> > > > > > > >> contains
> > > > > > > >> > Fractions Skill Score (FSS).
> > > > > > > >> >
> > > > > > > >> > In the attached config file, I've made a few
changes:
> > > > > > > >> > - Set the obtype to indicate that you're verifying
> against
> > > > > StageIV
> > > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > > >> > - In the first we process QP010 as a
probability
> > field.
> > > > > Here
> > > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation
> for
> > > > > > > >> > defining
> > > > > > > >> bins
> > > > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > > > >> > - In the second we process it as a field of
scalars.
> > > Here
> > > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > > threshold
> > > > > > > >> > I'm
> > > > > > > >> using
> > > > > > > >> > for defining events for fractions skill score.
> > > > > > > >> > - Defined 2 corresponding obs.field entries to
verify the
> > > > > forecast
> > > > > > > >> data.
> > > > > > > >> > In both the verifying threshold is >=2.54. This
assumes
> that
> > > you
> > > > > > > >> > pass
> > > > > > > >> the
> > > > > > > >> > raw StageIV data as input.
> > > > > > > >> > - In the output_flag section, I turned off the
line types
> > > that
> > > > > > > >> > don't apply.
> > > > > > > >> >
> > > > > > > >> > Lastly, while setting "level = [ "R19" ];" works as
a way
> of
> > > > > > > >> > grabbing record number 19 from the input file, it's
not
> > > > > > > >> > recommended. Instead,
> > > > > > > >> it'd
> > > > > > > >> > be better to figure out how to set the level string
so
> that
> > > MET
> > > > > > > >> > always grabs the same record, regardless if it's in
spot
> > > number
> > > > 19
> > > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > > >> >
> > > > > > > >> > Hope this helps clarify. What other questions do
you
> have?
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> > John Halley Gotway
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> > christopher.melick at us.af.mil
> > > > via
> > > > > RT
> > > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > > > >> > > Transaction: Ticket created by
> > christopher.melick at us.af.mil
> > > > > > > >> > > Queue: met_help
> > > > > > > >> > > Subject: Help with grid_stat on Neighborhood
> > statistics
> > > > > > > >> > > Owner: Nobody
> > > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > > >> > > Status: new
> > > > > > > >> > > Ticket <URL:
> > > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Hi John,
> > > > > > > >> > >
> > > > > > > >> > > I am trying to run grid_stat in MET to calculate
> > > Neighborhood
> > > > > > > >> > > statistics for probability of precipitation from
some
> > > ensemble
> > > > > > > >> > > data with the observations
> > > > > > > >> > > coming from Stage IV analyses. Unfortunately, I
am
> > getting
> > > > no
> > > > > > > output
> > > > > > > >> > > (just
> > > > > > > >> > > headers) in many of the ASCII files from the
program
> when
> > I
> > > > run
> > > > > > it.
> > > > > > > >> > Could
> > > > > > > >> > > you help me figure out what I am doing wrong? I
have
> > > provided
> > > > > > > >> > > the command line specifics I use as well as the
output
> log
> > > > file
> > > > > > > >> > > that is
> > > > > > > >> > created when
> > > > > > > >> > > running that command. I also have attached the
> > > configuration
> > > > > > file.
> > > > > > > >> I
> > > > > > > >> > am
> > > > > > > >> > > having trouble using ftp to send the raw data
from any
> of
> > > the
> > > > > > > clients
> > > > > > > >> > > available to us. I believe this restriction
occurred
> > > > recently
> > > > > > > since
> > > > > > > >> Bob
> > > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > > externally
> > > > > > > >> > > in the past.
> > > > > > > >> > >
> > > > > > > >> > > I should mention that I have pre-processed the
Stage IV
> > > > analyses
> > > > > > > >> > > already to be either 0's or 1's for the threshold
> (>=0.1")
> > > > since
> > > > > > > >> > > I was unclear as to whether MET was able to do
that on
> the
> > > fly
> > > > > > > >> > > when running
> > > > > > > >> > grid_stat.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Chris
> > > > > > > >> > >
> > > > > > > >> > > Python command:
> > > > > > > >> > > Metcall = [metpath + 'grid_stat',
> 'grib.2017091422.0006',
> > > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> > '-v',
> > > > '5]
> > > > > > > >> > > Call(metcall)
> > > > > > > >> > >
> > > > > > > >> > > //SIGNED//
> > > > > > > >> > >
> > > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron (16
> > > > > WS/WXEV)
> > > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm
> > (402)
> > > > > > > >> > > 294-3693
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: robert.craig.2 at us.af.mil
Time: Thu Jun 07 07:44:55 2018
John, do you know if NCAR has received funds to work the MET security
issues yet?
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, June 6, 2018 12:32 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with grid_stat on Neighborhood statistics
Hi Chris,
I went hunting through the met-help tickets and found this one that we
discussed during the poster session yesterday.
I've attached a PDF of the development ticket from JiRA which
describes the functionality we added in met-7.0 as a result of this
question.
Unfortunately though, I understand that you won't be able to make use
of
met-7.0 at the AF until we've addressed the security issues through
Fortify.
Thanks,
John
On Tue, Feb 27, 2018 at 11:58 AM, christopher.melick at us.af.mil via RT
< met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> I believe I have an idea of what some of the problem is when
computing
> AFSS in my particular case: When the original work on Fractions
Skill
> Score was published by Roberts and Lean (2008), it was designed
around the
> concept of a single deterministic model. Since then, that
statistic has
> been expanded to ensembles in a variety of ways to determine
> neighborhood probabilities as well as expanded to other
probabilistic forecasts which
> implicitly incorporate spatial uncertainty. Since the original
binary
> forecast fields are sometimes not available and/or possible, I don't
> know how the F_RATE fits into these new paradigms for computing an
> upper level benchmark for FSS (i.e. AFSS).
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 27, 2018 10:27 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> I looked at your python code and don't see anything obviously wrong
in the
> equations you've used. However, your computation of AFSS won't
match what
> MET is computing for the reasons we've already discussed. You're
computing
> fcst_rate and obs_rate as an average of the fractional coverage
values.
> MET computes them as a ratio of events occurring in the input
fields. As
> Tressa demonstrated, those will not necessarily result in the same
values.
> Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute
> will not match the AFSS that MET computes.
>
> Thanks,
> John
>
> On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Did you ever get the chance to take a look at my Python program
and
> > see if you notice anything that needs to be corrected? I'm still
not
> > sure what to make of the values I obtained for AFSS and why they
are
> lower than FSS,
> > especially at the grid-scale. Is this even possible?
> >
> > Thanks,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> > Sent: Wednesday, February 21, 2018 3:47 PM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi John,
> >
> > You are correct --- I never received the message below. Are the
average
> > coverage values what is used to compute the AFSS? I guess I am
still a
> > little confused - I was just using the first example to compute
the
> > O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would
> help
> > me in the example case I sent you because most events occurred in
the
> > interior of the domain. Also, wouldn't edge effects be eliminated
if the
> > vld_thresh is set to one in grid_stat since the entire
neighborhood must
> > contain valid data?
> >
> > If we leave the whole neighborhood averaging aside, and just focus
on the
> > computing AFSS at the grid scale (neighborhood = 1 grid point),
then my
> FSS
> > is still greater than AFSS. Missing data points on the analysis
grid
> only
> > result from different domain areas between that dataset and the
ensemble
> > forecast system.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing,
> > Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 21, 2018 2:59 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > Tressa sent an email on this issue, but she didn't include
> > met_help at ucar.edu.
> > So I don't think you received it. I've cut-and-pasted it below:
> >
> > John and Chris,
> >
> > It is not generally true that the average of the coverage values
is the
> > base rate. I think it may sometimes be true, if there are no
events in
> edge
> > neighborhoods without full coverage.
> >
> > I worked up the following example with a 4x4 neighborhood:
> >
> > 1 0 1 1
> > 0 0 0 0
> > 1 0 1 1
> > 0 1 0 0
> >
> > The rate here is 7/16 = .4375
> >
> > Fractional coverage values for a 3x3 neighborhood are:
> >
> > 1/4 2/6 2/6 2/4
> > 2/6 4/9 4/9 4/6
> > 2/6 3/9 3/9 2/6
> > 2/4 3/6 3/6 2/4
> >
> > The average of these values is 0.4149306
> >
> > I suppose if each fraction had the same denominator (e.g. no
edges, or
> all
> > non events there) then we would add each event up for 9 different
> > neighborhoods and then divide by 9. In that case, it should work.
> >
> > Tressa
> >
> >
> > On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > I don't get negative values for AFSS but I do get values that
are less
> > > than what I calculate for FSS. I approach the procedure for
computing
> > all
> > > the metrics such that everything is on the same "playing
ground".
> Thus,
> > > F_RATE and O_RATE are only computed by the mask of defined grid
points
> on
> > > *BOTH* the observation and forecast grids. When I process the
sample
> > time
> > > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> > value.
> > > Is that not the approach that I should follow? Were you able
to
> > > examine my script and raw data file I sent you?
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, February 20, 2018 8:20 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Your email got me thinking. Thinking through some toy examples
in my
> > > mind, I can see that probably yes, the mean of the fractional
coverage
> > field
> > > *should* be the same as the frequency of the event.
> > >
> > > I ran MET in the debugger and was able to stop at the point in
the
> logic
> > > where this is computed. At that point, we're passing around
arrays of
> > > fractional coverage field values as well as arrays of 0's and
1's
> > > indicating whether the event did or did not occur at each grid
point.
> > > Comparing the mean of these arrays, I find that they are close
but not
> > > identical. Sometimes one is larger, sometimes the other.
> > >
> > > I can think of 2 reasons why they wouldn't be the same, but
these may
> not
> > > fully explain it:
> > >
> > > (1) In Grid-Stat, the fractional coverage field is computed once
over
> the
> > > full verification domain. Then, the masking regions are
processed as
> > > subsets of these grid points. For a grid point that falls just
barely
> > > inside that mask, around 1/2 of the grid points used to compute
the
> > > fractional coverage value came from *OUTSIDE* that masking
region. The
> > 0/1
> > > arrays are only for points inside the current mask. So the mean
of the
> > 0/1
> > > array represents only the points inside the mask while the mean
of the
> > > fractional coverage includes info from outside the mask.
> > >
> > > (2) When data values are missing in the forecast or observation
fields,
> > > depending on how the user has set the vld_thresh config file
option,
> the
> > > "denominator" used to compute the fractional coverage field
value could
> > > change from grid point to grid point. That would likely throw
off the
> > mean
> > > of the fractional coverage field.
> > >
> > > Now what is the *CORRECT* way of computing the forecast and
observation
> > > rates? In papers about this, I see references to the
"base_rate" but
> no
> > > explicit equation.
> > >
> > > One very nice feature about the way MET is currently doing it is
that
> > > F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> > > Therefore, UFSS and AFSS remain constant as well. Using the
other
> > method,
> > > O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood size
> > > since the number of points included from outside the mask would
change.
> > >
> > > You mentioned that you had issues with negative UFSS and AFSS
values.
> > > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> > >
> > > Lots of little details to consider here!
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > John,
> > > >
> > > > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > > > can't be calculated in the cases below? Can't the gridded
data still
> > > > be read in to determine the relative frequency of the events,
> > regardless
> > > > whether grid_stat is used to compute fractional coverage
fields? I
> am
> > > > basically doing that right now in some Python code I developed
when
> > > > calculating Fractions Skill Score.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Hi Chris,
> > > >
> > > > I'm working on the interp.field option we discussed.
> > > >
> > > > I have a quick question for you. The NBRCNT line type that
is
> > computed
> > > by
> > > > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio
> of
> > > > grid points over a masking region at which the "event" is
occurring
> in
> > > the
> > > > forecast and observation fields respectively.
> > > >
> > > > When grid_stat computes the fractional coverage field, we can
compute
> > > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > > fields,
> > > > then we don't know what those ratio's are.
> > > >
> > > > - If interp.field = FCST, then O_RATE = NA.
> > > > - If interp.field = OBS, then F_RATE = NA.
> > > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > > >
> > > > Make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > John,
> > > > >
> > > > > That sounds like an excellent plan. Thanks for your
assistance and
> > > > > trying to accommodate and make changes to the MET software
so
> > quickly.
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Chris,
> > > > >
> > > > > Just wanted to close the loop on this ticket. I'll try to
get it
> > into
> > > > the
> > > > > next release due out next in March.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > > ambiguity and confusion. I kind of thought that was the
> procedure.
> > > > > >
> > > > > > When do you think that the next MET would be released with
the
> > > > > > enhancement that we discussed for the neighborhood
statistics?
> Is
> > it
> > > > > > an easy or difficult update to the software?
> > > > > >
> > > > > > Thanks again,
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Chris,
> > > > > >
> > > > > > It sounds like you understand how the fractional coverage
field
> is
> > > > > > computed.
> > > > > >
> > > > > > - Read the raw field... probability values between 0 and 1
in
> your
> > > > case.
> > > > > > - Apply the categorical threshold to that data to generate
a
> field
> > of
> > > > 0's
> > > > > > and 1's... >0.5 in your case.
> > > > > > - For each grid point, replace the value at that grid
point with
> > the
> > > > > > "fraction" of 1's in the neighborhood around that point...
e.g.
> the
> > > > event
> > > > > > occurs at 4 of the 9 grid points in a 3x3 square
surrounding the
> > > > current
> > > > > > point, so fractional coverage = 4/9.
> > > > > > - Compare the forecast fractional coverage values to the
observed
> > > > > > fractional coverage values to compute FSS.
> > > > > >
> > > > > > Yes, I do think it would be a good idea to talk to Craig
about
> how
> > he
> > > > > > configured MET for his work.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 8, 2018 at 2:09 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Getting back to your initial response, I was able to run
> > grid_stat
> > > > > > > with your changes in the attached configuration file to
get FSS
> > > > output
> > > > > > > (as well as the other statistics for the neighborhood
type
> > > results).
> > > > > > >
> > > > > > > However, I do have a question about the forecast
categorical
> > > > threshold
> > > > > > > (cat_thresh is set to >=0.5) used in determining the
> neighborhood
> > > > > > > probabilities from the ensemble probabilities: What is
the
> actual
> > > > > > > formula that MET uses to create the final field to
calculate
> FSS?
> > > Is
> > > > > > > it a strict neighborhood fractional coverage of ensemble
> > > > probabilities
> > > > > > > above 0.5 in some set radius of influence? Or is it
some other
> > > > > > > operation or does the threshold represent something
else? I
> know
> > > in
> > > > > > > Craig's paper, they were calling the last step a
neighborhood
> > > average
> > > > > > > of the ensemble probabilities to calculate NEP (since
the event
> > had
> > > > > > > already been defined at the grid point in determining
the
> > ensemble
> > > > > > probability [i.e. likelihood of exceeding
> > > > > > > 0.1" of precipitation]). Maybe, I should understand
how Craig
> > was
> > > > > using
> > > > > > > MET for his paper.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > > //SIGNED//
> > > > > > >
> > > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > christopher.melick at us.af.mil>
> > > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil
> > > >
> > > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > with
> > > > > > > grid_stat on Neighborhood statistics
> > > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > I was thinking about what changes would be needed to
compute
> FSS
> > on
> > > > > input
> > > > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > > > section,
> > > > > > > just like we have in the “interp” section. It would be
set to
> > > FCST,
> > > > > OBS,
> > > > > > > or BOTH (default) and would control the logic as to
which
> fields
> > > > should
> > > > > > be
> > > > > > > passed through the logic for deriving fractional
coverage
> fields.
> > > In
> > > > > > your
> > > > > > > case, you’d set it to OBS and we’d just use the raw
input
> > forecast
> > > > > > > probability values.
> > > > > > >
> > > > > > > Would that logic make sense to you?
> > > > > > >
> > > > > > > Thanks
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
> > for
> > > > > > > > adding the shape = circle option to MET.
> > > > > > > >
> > > > > > > > For your first question, no, there is not currently a
> Gaussian
> > > > > > > > smoother option for the "interp" section in Grid-Stat.
> > > Currently,
> > > > > the
> > > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN
> (for
> > > > > > > unweighted mean).
> > > > > > > > There is a distance-weighted mean option for point
data where
> > the
> > > > > > > > weight is 1/distance squared. But that doesn't work
for
> > > Grid-Stat
> > > > > > > > because we'd end up dividing by 0.
> > > > > > > >
> > > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > > >
> > > > > > > > Here's what we've done in the past with this SPC 40-km
> > > > probabilities:
> > > > > > > >
> > > > > > > > (1) Evaluate that field as a probability field (i.e.
set
> "prob
> > =
> > > > > > > > TRUE;" in the config file).
> > > > > > > > (2) Use the "interp" section to define a smoothing
operation
> on
> > > the
> > > > > > > > observations:
> > > > > > > > interp = {
> > > > > > > > field = OBS;
> > > > > > > > vld_thresh = 1.0;
> > > > > > > > shape = CIRCLE;
> > > > > > > > type = [
> > > > > > > > { method = MAX; width = 11; }
> > > > > > > > ];
> > > > > > > > }
> > > > > > > > Or set width to some circle diameter in grid units
that
> > works
> > > > out
> > > > > > > > to 40km in the real world.
> > > > > > > >
> > > > > > > > In this example, we post-process the observations to
make
> them
> > > > > > > > comparable the event for which the probability was
defined.
> > > > > > > >
> > > > > > > > But I think I understand what you'd like to do...
interpret
> as
> > > the
> > > > > > > > probability value at each grid point as if it was a
> fractional
> > > > > > > > coverage value. So don't apply a threshold and
neighborhood
> > size
> > > > to
> > > > > > > > the forecast data at all. Just pass it directly to
the
> routine
> > > > that
> > > > > > > computes FSS.
> > > > > > > >
> > > > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > > > Grid-Stat
> > > > > > > > to do that in met-6.1. If you think this logic would
be
> > > generally
> > > > > > > > useful, we could consider adding a flag to the config
file to
> > > > enable
> > > > > > > that logic.
> > > > > > > >
> > > > > > > > Sorry I don't have better answers for you.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
> > via
> > > > RT
> > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >>
> > > > > > > >> <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83986
> > >
> > > > > > > >>
> > > > > > > >> Hi John,
> > > > > > > >>
> > > > > > > >> Your suggestion for accomplishing #2 sounds
promising. I
> need
> > > to
> > > > > > think
> > > > > > > >> the details through some. On a somewhat related
note, is
> > > there a
> > > > > > > Gaussian
> > > > > > > >> smoother flag for smoothing the probabilistic fields?
I
> know
> > > > that
> > > > > > was
> > > > > > > >> mentioned in Craig's paper and I have pursued that
option in
> > the
> > > > > past
> > > > > > > >> (but just not using MET).
> > > > > > > >>
> > > > > > > >> I guess my basic question is can you apply a
neighborhood
> > width
> > > > that
> > > > > > > >> just includes the grid point maybe because you
already have
> > > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > > Would
> > > > > that
> > > > > > > be a width of 1
> > > > > > > >> or does that also include the nearest grid point?
A good
> > > > example
> > > > > of
> > > > > > > this
> > > > > > > >> would be a probabilistic forecast issued from the
Storm
> > > Prediction
> > > > > > > >> Center where the probabilities implicitly assume a
40-km
> > radius
> > > of
> > > > > > > >> influence (so there is no need to compute a
fractional
> > coverage
> > > of
> > > > > the
> > > > > > > event).
> > > > > > > >>
> > > > > > > >> Thanks again,
> > > > > > > >> Chris
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> //SIGNED//
> > > > > > > >>
> > > > > > > >> Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > Verification
> > > > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> -----Original Message-----
> > > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > >> christopher.melick at us.af.mil>
> > > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF
> ACC
> > 16
> > > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> > Help
> > > > > with
> > > > > > > >> grid_stat on Neighborhood statistics
> > > > > > > >>
> > > > > > > >> Chris,
> > > > > > > >>
> > > > > > > >> (1) The default shape is a square since that's what
MET
> > > previously
> > > > > > did.
> > > > > > > >> The circle is the *new* option.
> > > > > > > >>
> > > > > > > >> (2) For this one, the short answer is yes. FSS is
computed
> in
> > > MET
> > > > > by
> > > > > > > >> doing the following...
> > > > > > > >> - For each field separately (i.e. fcst and obs),
apply the
> > > > > > > >> threshold
> > > > > > > >> (cat_thresh) to convert the input data to a 0/1
bitmap.
> > > > > > > >> - Apply the neighborhood width and shape to compute
a
> > > fractional
> > > > > > > >> coverage value for each grid point.
> > > > > > > >> - Compare the fcst and obs fractional coverage
fields to
> > > compute
> > > > > > FSS.
> > > > > > > >>
> > > > > > > >> But there are some other options that you may find
useful.
> > The
> > > > > > "interp"
> > > > > > > >> section in the config file is used in Grid-Stat as a
> smoothing
> > > > > > > operation.
> > > > > > > >> After reading the raw input data, you can specify
if/how to
> > > smooth
> > > > > > > >> that data prior to computing stats. For example,
since you
> > > > > mentioned
> > > > > > > >> maximums, you could replace the value at each grid
point
> with
> > > the
> > > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > > >>
> > > > > > > >> interp = {
> > > > > > > >> field = FCST;
> > > > > > > >> vld_thresh = 1.0;
> > > > > > > >> shape = SQUARE;
> > > > > > > >>
> > > > > > > >> type = [
> > > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
> smoothing
> > > > > > > >> { method = MAX; width = 5; }, // i.e. max of
25
> closest
> > > > > points
> > > > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean
of the
> > 25
> > > > > > > >> closest points
> > > > > > > >> ];
> > > > > > > >> }
> > > > > > > >>
> > > > > > > >> Using the example above, you'll get 3 times the
amount of
> > output
> > > > > from
> > > > > > > MET.
> > > > > > > >> The first smoother really is no smoothing at all. It
would
> > run
> > > on
> > > > > > > >> the raw input data. The second smoother replaces the
value
> at
> > > > each
> > > > > > > >> grid point with the maximum of the 25 surrounding
points.
> The
> > > > third
> > > > > > > >> smoother replaces the value at each grid point with
the mean
> > of
> > > > > those
> > > > > > > 25.
> > > > > > > >>
> > > > > > > >> I worked with Craig Schwartz when he was running MET
for
> that
> > > > paper.
> > > > > > > >> My recollection is that he was able to use the
configuration
> > > > options
> > > > > > > >> of Grid-Stat to accomplish the types of processing he
was
> > after.
> > > > > But
> > > > > > > >> I don't recall all the details.
> > > > > > > >>
> > > > > > > >> If you explain exactly what you're trying to do, it
may be
> > > > possible
> > > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
> christopher.melick at us.af.mil
> > > via
> > > > > RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=83986
> > > >
> > > > > > > >> >
> > > > > > > >> > Hi John,
> > > > > > > >> >
> > > > > > > >> > Your explanation and suggestions are very helpful
and do
> > > explain
> > > > > my
> > > > > > > >> > problems quite substantially. I will have to give
it a
> try
> > to
> > > > see
> > > > > > > >> > what happens.
> > > > > > > >> >
> > > > > > > >> > I do have a couple of other minor questions dealing
with
> how
> > > the
> > > > > > > >> > neighborhood verification is performed:
> > > > > > > >> > 1.) I noticed in the documentation that there is a
flag
> for
> > > > > > > >> > defining the shape of the neighborhood area (i.e.
circular
> > or
> > > > > > > >> > square). What is the default if you don't specify?
> > > > > > > >> > 2.) In the literature, there are a couple of
different
> ways
> > > to
> > > > > > > >> > handle generating neighborhood probabilities (e.g.,
> Schwartz
> > > and
> > > > > > > >> > Sobash 2017;
> > > > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-
D-16-
> > 0400.1
> > > ).
> > > > > I
> > > > > > > >> > was curious is the only way to get the Fractions
Skill
> Score
> > > > (FSS)
> > > > > > > >> > output is to define a neighborhood in which MET
> calculates a
> > > > > > > >> > fractional coverage of the threshold event? I have
> > previously
> > > > > > > >> > explored computing the neighborhood maximum value
within a
> > > > radius
> > > > > of
> > > > > > > each grid point before calculating a probability.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chris
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > //SIGNED//
> > > > > > > >> >
> > > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > Verification
> > > > > > > >> > Exploitation Team 16th Weather Squadron (16
WS/WXEV) 557th
> > > > Weather
> > > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > -----Original Message-----
> > > > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> ]
> > > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<
> > > > > > > >> > christopher.melick at us.af.mil>
> > > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > > robert.craig.2 at us.af.mil>
> > > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > > with
> > > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > > >> >
> > > > > > > >> > Hi Chris,
> > > > > > > >> >
> > > > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > > > >> > statistics in Grid-Stat.
> > > > > > > >> >
> > > > > > > >> > First, Grid-Stat definitely does have the ability
to
> > threshold
> > > > the
> > > > > > > >> StageIV
> > > > > > > >> > data on the fly. So pre-processing to 0's and 1's
first
> is
> > > not
> > > > > > > needed.
> > > > > > > >> > Instead, in the "obs" dictionary of the config
file, you'd
> > > just
> > > > > set:
> > > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > > >> >
> > > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is
the same
> > as
> > > > > >=0.1
> > > > > > > >> inches.
> > > > > > > >> >
> > > > > > > >> > If for some reason you have data that's already 0's
and
> 1's,
> > > > you'd
> > > > > > > >> > just use a different threshold in the config file:
> > > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > > >> >
> > > > > > > >> > This tells MET that anywhere you see a 1 in the
data, the
> > > event
> > > > is
> > > > > > > >> > occurring. Otherwise, it's not.
> > > > > > > >> >
> > > > > > > >> > Now on the to real issue. Looking at your config
file, I
> > see
> > > > that
> > > > > > > >> you're
> > > > > > > >> > processing probabilistic data. When "prob = TRUE",
MET
> only
> > > > > > > >> > computes
> > > > > > > >> the
> > > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
> That's
> > > why
> > > > > > > >> > you're getting no output in the NBRCNT line type,
which is
> > the
> > > > one
> > > > > > > >> > that
> > > > > > > >> contains
> > > > > > > >> > Fractions Skill Score (FSS).
> > > > > > > >> >
> > > > > > > >> > In the attached config file, I've made a few
changes:
> > > > > > > >> > - Set the obtype to indicate that you're verifying
> against
> > > > > StageIV
> > > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > > >> > - In the first we process QP010 as a
probability
> > field.
> > > > > Here
> > > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation
> for
> > > > > > > >> > defining
> > > > > > > >> bins
> > > > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > > > >> > - In the second we process it as a field of
scalars.
> > > Here
> > > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > > threshold
> > > > > > > >> > I'm
> > > > > > > >> using
> > > > > > > >> > for defining events for fractions skill score.
> > > > > > > >> > - Defined 2 corresponding obs.field entries to
verify the
> > > > > forecast
> > > > > > > >> data.
> > > > > > > >> > In both the verifying threshold is >=2.54. This
assumes
> that
> > > you
> > > > > > > >> > pass
> > > > > > > >> the
> > > > > > > >> > raw StageIV data as input.
> > > > > > > >> > - In the output_flag section, I turned off the
line types
> > > that
> > > > > > > >> > don't apply.
> > > > > > > >> >
> > > > > > > >> > Lastly, while setting "level = [ "R19" ];" works as
a way
> of
> > > > > > > >> > grabbing record number 19 from the input file, it's
not
> > > > > > > >> > recommended. Instead,
> > > > > > > >> it'd
> > > > > > > >> > be better to figure out how to set the level string
so
> that
> > > MET
> > > > > > > >> > always grabs the same record, regardless if it's in
spot
> > > number
> > > > 19
> > > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > > >> >
> > > > > > > >> > Hope this helps clarify. What other questions do
you
> have?
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> > John Halley Gotway
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> > christopher.melick at us.af.mil
> > > > via
> > > > > RT
> > > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > > > >> > > Transaction: Ticket created by
> > christopher.melick at us.af.mil
> > > > > > > >> > > Queue: met_help
> > > > > > > >> > > Subject: Help with grid_stat on Neighborhood
> > statistics
> > > > > > > >> > > Owner: Nobody
> > > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > > >> > > Status: new
> > > > > > > >> > > Ticket <URL:
> > > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Hi John,
> > > > > > > >> > >
> > > > > > > >> > > I am trying to run grid_stat in MET to calculate
> > > Neighborhood
> > > > > > > >> > > statistics for probability of precipitation from
some
> > > ensemble
> > > > > > > >> > > data with the observations
> > > > > > > >> > > coming from Stage IV analyses. Unfortunately, I
am
> > getting
> > > > no
> > > > > > > output
> > > > > > > >> > > (just
> > > > > > > >> > > headers) in many of the ASCII files from the
program
> when
> > I
> > > > run
> > > > > > it.
> > > > > > > >> > Could
> > > > > > > >> > > you help me figure out what I am doing wrong? I
have
> > > provided
> > > > > > > >> > > the command line specifics I use as well as the
output
> log
> > > > file
> > > > > > > >> > > that is
> > > > > > > >> > created when
> > > > > > > >> > > running that command. I also have attached the
> > > configuration
> > > > > > file.
> > > > > > > >> I
> > > > > > > >> > am
> > > > > > > >> > > having trouble using ftp to send the raw data
from any
> of
> > > the
> > > > > > > clients
> > > > > > > >> > > available to us. I believe this restriction
occurred
> > > > recently
> > > > > > > since
> > > > > > > >> Bob
> > > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > > externally
> > > > > > > >> > > in the past.
> > > > > > > >> > >
> > > > > > > >> > > I should mention that I have pre-processed the
Stage IV
> > > > analyses
> > > > > > > >> > > already to be either 0's or 1's for the threshold
> (>=0.1")
> > > > since
> > > > > > > >> > > I was unclear as to whether MET was able to do
that on
> the
> > > fly
> > > > > > > >> > > when running
> > > > > > > >> > grid_stat.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Chris
> > > > > > > >> > >
> > > > > > > >> > > Python command:
> > > > > > > >> > > Metcall = [metpath + 'grid_stat',
> 'grib.2017091422.0006',
> > > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> > '-v',
> > > > '5]
> > > > > > > >> > > Call(metcall)
> > > > > > > >> > >
> > > > > > > >> > > //SIGNED//
> > > > > > > >> > >
> > > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron (16
> > > > > WS/WXEV)
> > > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm
> > (402)
> > > > > > > >> > > 294-3693
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Thu Jun 07 10:29:06 2018
Hi Bob,
Tara tells me that you two discussed this in a separate email chain.
I
don’t have any more info than she does. Our expectation is that the
funding will be in place at the end of June, and we’ll start working
on it
in July.
Thanks
John
On Thu, Jun 7, 2018 at 7:45 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John, do you know if NCAR has received funds to work the MET
security
> issues yet?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, June 6, 2018 12:32 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I went hunting through the met-help tickets and found this one that
we
> discussed during the poster session yesterday.
>
> I've attached a PDF of the development ticket from JiRA which
describes
> the functionality we added in met-7.0 as a result of this question.
> Unfortunately though, I understand that you won't be able to make
use of
> met-7.0 at the AF until we've addressed the security issues through
> Fortify.
>
> Thanks,
> John
>
>
> On Tue, Feb 27, 2018 at 11:58 AM, christopher.melick at us.af.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > John,
> >
> > I believe I have an idea of what some of the problem is when
computing
> > AFSS in my particular case: When the original work on Fractions
Skill
> > Score was published by Roberts and Lean (2008), it was designed
around
> the
> > concept of a single deterministic model. Since then, that
statistic has
> > been expanded to ensembles in a variety of ways to determine
> > neighborhood probabilities as well as expanded to other
probabilistic
> forecasts which
> > implicitly incorporate spatial uncertainty. Since the original
binary
> > forecast fields are sometimes not available and/or possible, I
don't
> > know how the F_RATE fits into these new paradigms for computing an
> > upper level benchmark for FSS (i.e. AFSS).
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, February 27, 2018 10:27 AM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > grid_stat on Neighborhood statistics
> >
> > Chris,
> >
> > I looked at your python code and don't see anything obviously
wrong in
> the
> > equations you've used. However, your computation of AFSS won't
match
> what
> > MET is computing for the reasons we've already discussed. You're
> computing
> > fcst_rate and obs_rate as an average of the fractional coverage
values.
> > MET computes them as a ratio of events occurring in the input
fields. As
> > Tressa demonstrated, those will not necessarily result in the same
> values.
> > Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute
> > will not match the AFSS that MET computes.
> >
> > Thanks,
> > John
> >
> > On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > Did you ever get the chance to take a look at my Python program
and
> > > see if you notice anything that needs to be corrected? I'm
still not
> > > sure what to make of the values I obtained for AFSS and why they
are
> > lower than FSS,
> > > especially at the grid-scale. Is this even possible?
> > >
> > > Thanks,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> > > Sent: Wednesday, February 21, 2018 3:47 PM
> > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi John,
> > >
> > > You are correct --- I never received the message below. Are the
> average
> > > coverage values what is used to compute the AFSS? I guess I am
> still a
> > > little confused - I was just using the first example to compute
the
> > > O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would
> > help
> > > me in the example case I sent you because most events occurred
in the
> > > interior of the domain. Also, wouldn't edge effects be
eliminated if
> the
> > > vld_thresh is set to one in grid_stat since the entire
neighborhood
> must
> > > contain valid data?
> > >
> > > If we leave the whole neighborhood averaging aside, and just
focus on
> the
> > > computing AFSS at the grid scale (neighborhood = 1 grid point),
then my
> > FSS
> > > is still greater than AFSS. Missing data points on the analysis
grid
> > only
> > > result from different domain areas between that dataset and the
> ensemble
> > > forecast system.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> Wing,
> > > Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, February 21, 2018 2:59 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Hi Chris,
> > >
> > > Tressa sent an email on this issue, but she didn't include
> > > met_help at ucar.edu.
> > > So I don't think you received it. I've cut-and-pasted it below:
> > >
> > > John and Chris,
> > >
> > > It is not generally true that the average of the coverage values
is the
> > > base rate. I think it may sometimes be true, if there are no
events in
> > edge
> > > neighborhoods without full coverage.
> > >
> > > I worked up the following example with a 4x4 neighborhood:
> > >
> > > 1 0 1 1
> > > 0 0 0 0
> > > 1 0 1 1
> > > 0 1 0 0
> > >
> > > The rate here is 7/16 = .4375
> > >
> > > Fractional coverage values for a 3x3 neighborhood are:
> > >
> > > 1/4 2/6 2/6 2/4
> > > 2/6 4/9 4/9 4/6
> > > 2/6 3/9 3/9 2/6
> > > 2/4 3/6 3/6 2/4
> > >
> > > The average of these values is 0.4149306
> > >
> > > I suppose if each fraction had the same denominator (e.g. no
edges, or
> > all
> > > non events there) then we would add each event up for 9
different
> > > neighborhoods and then divide by 9. In that case, it should
work.
> > >
> > > Tressa
> > >
> > >
> > > On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > Hi John,
> > > >
> > > > I don't get negative values for AFSS but I do get values that
are
> less
> > > > than what I calculate for FSS. I approach the procedure for
> computing
> > > all
> > > > the metrics such that everything is on the same "playing
ground".
> > Thus,
> > > > F_RATE and O_RATE are only computed by the mask of defined
grid
> points
> > on
> > > > *BOTH* the observation and forecast grids. When I process
the
> sample
> > > time
> > > > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> > > value.
> > > > Is that not the approach that I should follow? Were you able
to
> > > > examine my script and raw data file I sent you?
> > > >
> > > > Thanks again,
> > > > Chris
> > > >
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Tuesday, February 20, 2018 8:20 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Chris,
> > > >
> > > > Your email got me thinking. Thinking through some toy
examples in my
> > > > mind, I can see that probably yes, the mean of the fractional
> coverage
> > > field
> > > > *should* be the same as the frequency of the event.
> > > >
> > > > I ran MET in the debugger and was able to stop at the point in
the
> > logic
> > > > where this is computed. At that point, we're passing around
arrays
> of
> > > > fractional coverage field values as well as arrays of 0's and
1's
> > > > indicating whether the event did or did not occur at each grid
point.
> > > > Comparing the mean of these arrays, I find that they are close
but
> not
> > > > identical. Sometimes one is larger, sometimes the other.
> > > >
> > > > I can think of 2 reasons why they wouldn't be the same, but
these may
> > not
> > > > fully explain it:
> > > >
> > > > (1) In Grid-Stat, the fractional coverage field is computed
once over
> > the
> > > > full verification domain. Then, the masking regions are
processed as
> > > > subsets of these grid points. For a grid point that falls
just
> barely
> > > > inside that mask, around 1/2 of the grid points used to
compute the
> > > > fractional coverage value came from *OUTSIDE* that masking
region.
> The
> > > 0/1
> > > > arrays are only for points inside the current mask. So the
mean of
> the
> > > 0/1
> > > > array represents only the points inside the mask while the
mean of
> the
> > > > fractional coverage includes info from outside the mask.
> > > >
> > > > (2) When data values are missing in the forecast or
observation
> fields,
> > > > depending on how the user has set the vld_thresh config file
option,
> > the
> > > > "denominator" used to compute the fractional coverage field
value
> could
> > > > change from grid point to grid point. That would likely throw
off
> the
> > > mean
> > > > of the fractional coverage field.
> > > >
> > > > Now what is the *CORRECT* way of computing the forecast and
> observation
> > > > rates? In papers about this, I see references to the
"base_rate" but
> > no
> > > > explicit equation.
> > > >
> > > > One very nice feature about the way MET is currently doing it
is that
> > > > F_RATE and O_RATE remain constant regardless of the
neighborhood
> size.
> > > > Therefore, UFSS and AFSS remain constant as well. Using the
other
> > > method,
> > > > O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood
> size
> > > > since the number of points included from outside the mask
would
> change.
> > > >
> > > > You mentioned that you had issues with negative UFSS and AFSS
values.
> > > > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> > > >
> > > > Lots of little details to consider here!
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > John,
> > > > >
> > > > > After thinking about it some, I was curious why the F_RATE
and
> O_RATE
> > > > > can't be calculated in the cases below? Can't the gridded
data
> still
> > > > > be read in to determine the relative frequency of the
events,
> > > regardless
> > > > > whether grid_stat is used to compute fractional coverage
fields?
> I
> > am
> > > > > basically doing that right now in some Python code I
developed when
> > > > > calculating Fractions Skill Score.
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > I'm working on the interp.field option we discussed.
> > > > >
> > > > > I have a quick question for you. The NBRCNT line type that
is
> > > computed
> > > > by
> > > > > grid_stat contains columns for F_RATE and O_RATE. That is
the
> ratio
> > of
> > > > > grid points over a masking region at which the "event" is
occurring
> > in
> > > > the
> > > > > forecast and observation fields respectively.
> > > > >
> > > > > When grid_stat computes the fractional coverage field, we
can
> compute
> > > > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > > > fields,
> > > > > then we don't know what those ratio's are.
> > > > >
> > > > > - If interp.field = FCST, then O_RATE = NA.
> > > > > - If interp.field = OBS, then F_RATE = NA.
> > > > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > > > >
> > > > > Make sense?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Wed, Feb 14, 2018 at 9:39 AM,
christopher.melick at us.af.mil via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > That sounds like an excellent plan. Thanks for your
assistance
> and
> > > > > > trying to accommodate and make changes to the MET software
so
> > > quickly.
> > > > > >
> > > > > > Thanks,
> > > > > > Chris
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Chris,
> > > > > >
> > > > > > Just wanted to close the loop on this ticket. I'll try to
get it
> > > into
> > > > > the
> > > > > > next release due out next in March.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Feb 9, 2018 at 8:11 AM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > > > ambiguity and confusion. I kind of thought that was the
> > procedure.
> > > > > > >
> > > > > > > When do you think that the next MET would be released
with the
> > > > > > > enhancement that we discussed for the neighborhood
statistics?
> > Is
> > > it
> > > > > > > an easy or difficult update to the software?
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > > //SIGNED//
> > > > > > >
> > > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > christopher.melick at us.af.mil>
> > > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil
> > > >
> > > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > with
> > > > > > > grid_stat on Neighborhood statistics
> > > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > It sounds like you understand how the fractional
coverage field
> > is
> > > > > > > computed.
> > > > > > >
> > > > > > > - Read the raw field... probability values between 0 and
1 in
> > your
> > > > > case.
> > > > > > > - Apply the categorical threshold to that data to
generate a
> > field
> > > of
> > > > > 0's
> > > > > > > and 1's... >0.5 in your case.
> > > > > > > - For each grid point, replace the value at that grid
point
> with
> > > the
> > > > > > > "fraction" of 1's in the neighborhood around that
point... e.g.
> > the
> > > > > event
> > > > > > > occurs at 4 of the 9 grid points in a 3x3 square
surrounding
> the
> > > > > current
> > > > > > > point, so fractional coverage = 4/9.
> > > > > > > - Compare the forecast fractional coverage values to the
> observed
> > > > > > > fractional coverage values to compute FSS.
> > > > > > >
> > > > > > > Yes, I do think it would be a good idea to talk to Craig
about
> > how
> > > he
> > > > > > > configured MET for his work.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 8, 2018 at 2:09 PM,
christopher.melick at us.af.mil
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Getting back to your initial response, I was able to
run
> > > grid_stat
> > > > > > > > with your changes in the attached configuration file
to get
> FSS
> > > > > output
> > > > > > > > (as well as the other statistics for the neighborhood
type
> > > > results).
> > > > > > > >
> > > > > > > > However, I do have a question about the forecast
categorical
> > > > > threshold
> > > > > > > > (cat_thresh is set to >=0.5) used in determining the
> > neighborhood
> > > > > > > > probabilities from the ensemble probabilities: What is
the
> > actual
> > > > > > > > formula that MET uses to create the final field to
calculate
> > FSS?
> > > > Is
> > > > > > > > it a strict neighborhood fractional coverage of
ensemble
> > > > > probabilities
> > > > > > > > above 0.5 in some set radius of influence? Or is it
some
> other
> > > > > > > > operation or does the threshold represent something
else? I
> > know
> > > > in
> > > > > > > > Craig's paper, they were calling the last step a
neighborhood
> > > > average
> > > > > > > > of the ensemble probabilities to calculate NEP (since
the
> event
> > > had
> > > > > > > > already been defined at the grid point in determining
the
> > > ensemble
> > > > > > > probability [i.e. likelihood of exceeding
> > > > > > > > 0.1" of precipitation]). Maybe, I should understand
how
> Craig
> > > was
> > > > > > using
> > > > > > > > MET for his paper.
> > > > > > > >
> > > > > > > > Thanks again,
> > > > > > > > Chris
> > > > > > > >
> > > > > > > >
> > > > > > > > //SIGNED//
> > > > > > > >
> > > > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > > Verification
> > > > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > > christopher.melick at us.af.mil>
> > > > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > robert.craig.2 at us.af.mil
> > > > >
> > > > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> > Help
> > > > with
> > > > > > > > grid_stat on Neighborhood statistics
> > > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > I was thinking about what changes would be needed to
compute
> > FSS
> > > on
> > > > > > input
> > > > > > > > probability data. We could add a “field” option to
the
> “nbrhd”
> > > > > > section,
> > > > > > > > just like we have in the “interp” section. It would
be set
> to
> > > > FCST,
> > > > > > OBS,
> > > > > > > > or BOTH (default) and would control the logic as to
which
> > fields
> > > > > should
> > > > > > > be
> > > > > > > > passed through the logic for deriving fractional
coverage
> > fields.
> > > > In
> > > > > > > your
> > > > > > > > case, you’d set it to OBS and we’d just use the raw
input
> > > forecast
> > > > > > > > probability values.
> > > > > > > >
> > > > > > > > Would that logic make sense to you?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> > > johnhg at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > >
> > > > > > > > > In fact, SPC's 40-km radius probabilities were the
> motivation
> > > for
> > > > > > > > > adding the shape = circle option to MET.
> > > > > > > > >
> > > > > > > > > For your first question, no, there is not currently
a
> > Gaussian
> > > > > > > > > smoother option for the "interp" section in Grid-
Stat.
> > > > Currently,
> > > > > > the
> > > > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN
> > (for
> > > > > > > > unweighted mean).
> > > > > > > > > There is a distance-weighted mean option for point
data
> where
> > > the
> > > > > > > > > weight is 1/distance squared. But that doesn't work
for
> > > > Grid-Stat
> > > > > > > > > because we'd end up dividing by 0.
> > > > > > > > >
> > > > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > > > >
> > > > > > > > > Here's what we've done in the past with this SPC 40-
km
> > > > > probabilities:
> > > > > > > > >
> > > > > > > > > (1) Evaluate that field as a probability field (i.e.
set
> > "prob
> > > =
> > > > > > > > > TRUE;" in the config file).
> > > > > > > > > (2) Use the "interp" section to define a smoothing
> operation
> > on
> > > > the
> > > > > > > > > observations:
> > > > > > > > > interp = {
> > > > > > > > > field = OBS;
> > > > > > > > > vld_thresh = 1.0;
> > > > > > > > > shape = CIRCLE;
> > > > > > > > > type = [
> > > > > > > > > { method = MAX; width = 11; }
> > > > > > > > > ];
> > > > > > > > > }
> > > > > > > > > Or set width to some circle diameter in grid
units that
> > > works
> > > > > out
> > > > > > > > > to 40km in the real world.
> > > > > > > > >
> > > > > > > > > In this example, we post-process the observations to
make
> > them
> > > > > > > > > comparable the event for which the probability was
defined.
> > > > > > > > >
> > > > > > > > > But I think I understand what you'd like to do...
interpret
> > as
> > > > the
> > > > > > > > > probability value at each grid point as if it was a
> > fractional
> > > > > > > > > coverage value. So don't apply a threshold and
> neighborhood
> > > size
> > > > > to
> > > > > > > > > the forecast data at all. Just pass it directly to
the
> > routine
> > > > > that
> > > > > > > > computes FSS.
> > > > > > > > >
> > > > > > > > > Unfortunately no, I don't think there's a way of
> configuring
> > > > > > Grid-Stat
> > > > > > > > > to do that in met-6.1. If you think this logic
would be
> > > > generally
> > > > > > > > > useful, we could consider adding a flag to the
config file
> to
> > > > > enable
> > > > > > > > that logic.
> > > > > > > > >
> > > > > > > > > Sorry I don't have better answers for you.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
> christopher.melick at us.af.mil
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > >>
> > > > > > > > >> <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=83986
> > > >
> > > > > > > > >>
> > > > > > > > >> Hi John,
> > > > > > > > >>
> > > > > > > > >> Your suggestion for accomplishing #2 sounds
promising. I
> > need
> > > > to
> > > > > > > think
> > > > > > > > >> the details through some. On a somewhat related
note, is
> > > > there a
> > > > > > > > Gaussian
> > > > > > > > >> smoother flag for smoothing the probabilistic
fields? I
> > know
> > > > > that
> > > > > > > was
> > > > > > > > >> mentioned in Craig's paper and I have pursued that
option
> in
> > > the
> > > > > > past
> > > > > > > > >> (but just not using MET).
> > > > > > > > >>
> > > > > > > > >> I guess my basic question is can you apply a
neighborhood
> > > width
> > > > > that
> > > > > > > > >> just includes the grid point maybe because you
already
> have
> > > > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > > > Would
> > > > > > that
> > > > > > > > be a width of 1
> > > > > > > > >> or does that also include the nearest grid point?
A
> good
> > > > > example
> > > > > > of
> > > > > > > > this
> > > > > > > > >> would be a probabilistic forecast issued from the
Storm
> > > > Prediction
> > > > > > > > >> Center where the probabilities implicitly assume a
40-km
> > > radius
> > > > of
> > > > > > > > >> influence (so there is no need to compute a
fractional
> > > coverage
> > > > of
> > > > > > the
> > > > > > > > event).
> > > > > > > > >>
> > > > > > > > >> Thanks again,
> > > > > > > > >> Chris
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> //SIGNED//
> > > > > > > > >>
> > > > > > > > >> Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > Verification
> > > > > > > > >> Exploitation Team 16th Weather Squadron (16
WS/WXEV) 557th
> > > > Weather
> > > > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> -----Original Message-----
> > > > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> ]
> > > > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<
> > > > > > > > >> christopher.melick at us.af.mil>
> > > > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF
> > ACC
> > > 16
> > > > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
> #83986]
> > > Help
> > > > > > with
> > > > > > > > >> grid_stat on Neighborhood statistics
> > > > > > > > >>
> > > > > > > > >> Chris,
> > > > > > > > >>
> > > > > > > > >> (1) The default shape is a square since that's what
MET
> > > > previously
> > > > > > > did.
> > > > > > > > >> The circle is the *new* option.
> > > > > > > > >>
> > > > > > > > >> (2) For this one, the short answer is yes. FSS is
> computed
> > in
> > > > MET
> > > > > > by
> > > > > > > > >> doing the following...
> > > > > > > > >> - For each field separately (i.e. fcst and obs),
apply
> the
> > > > > > > > >> threshold
> > > > > > > > >> (cat_thresh) to convert the input data to a 0/1
bitmap.
> > > > > > > > >> - Apply the neighborhood width and shape to
compute a
> > > > fractional
> > > > > > > > >> coverage value for each grid point.
> > > > > > > > >> - Compare the fcst and obs fractional coverage
fields to
> > > > compute
> > > > > > > FSS.
> > > > > > > > >>
> > > > > > > > >> But there are some other options that you may find
useful.
> > > The
> > > > > > > "interp"
> > > > > > > > >> section in the config file is used in Grid-Stat as
a
> > smoothing
> > > > > > > > operation.
> > > > > > > > >> After reading the raw input data, you can specify
if/how
> to
> > > > smooth
> > > > > > > > >> that data prior to computing stats. For example,
since
> you
> > > > > > mentioned
> > > > > > > > >> maximums, you could replace the value at each grid
point
> > with
> > > > the
> > > > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > > > >>
> > > > > > > > >> interp = {
> > > > > > > > >> field = FCST;
> > > > > > > > >> vld_thresh = 1.0;
> > > > > > > > >> shape = SQUARE;
> > > > > > > > >>
> > > > > > > > >> type = [
> > > > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
> > smoothing
> > > > > > > > >> { method = MAX; width = 5; }, // i.e. max of
25
> > closest
> > > > > > points
> > > > > > > > >> { method = UW_MEAN; width = 5; }, // i.e.
mean of
> the
> > > 25
> > > > > > > > >> closest points
> > > > > > > > >> ];
> > > > > > > > >> }
> > > > > > > > >>
> > > > > > > > >> Using the example above, you'll get 3 times the
amount of
> > > output
> > > > > > from
> > > > > > > > MET.
> > > > > > > > >> The first smoother really is no smoothing at all.
It
> would
> > > run
> > > > on
> > > > > > > > >> the raw input data. The second smoother replaces
the
> value
> > at
> > > > > each
> > > > > > > > >> grid point with the maximum of the 25 surrounding
points.
> > The
> > > > > third
> > > > > > > > >> smoother replaces the value at each grid point with
the
> mean
> > > of
> > > > > > those
> > > > > > > > 25.
> > > > > > > > >>
> > > > > > > > >> I worked with Craig Schwartz when he was running
MET for
> > that
> > > > > paper.
> > > > > > > > >> My recollection is that he was able to use the
> configuration
> > > > > options
> > > > > > > > >> of Grid-Stat to accomplish the types of processing
he was
> > > after.
> > > > > > But
> > > > > > > > >> I don't recall all the details.
> > > > > > > > >>
> > > > > > > > >> If you explain exactly what you're trying to do, it
may be
> > > > > possible
> > > > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> John
> > > > > > > > >>
> > > > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
> > christopher.melick at us.af.mil
> > > > via
> > > > > > RT <
> > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > >>
> > > > > > > > >> >
> > > > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=83986
> > > > >
> > > > > > > > >> >
> > > > > > > > >> > Hi John,
> > > > > > > > >> >
> > > > > > > > >> > Your explanation and suggestions are very helpful
and do
> > > > explain
> > > > > > my
> > > > > > > > >> > problems quite substantially. I will have to
give it a
> > try
> > > to
> > > > > see
> > > > > > > > >> > what happens.
> > > > > > > > >> >
> > > > > > > > >> > I do have a couple of other minor questions
dealing with
> > how
> > > > the
> > > > > > > > >> > neighborhood verification is performed:
> > > > > > > > >> > 1.) I noticed in the documentation that there is
a flag
> > for
> > > > > > > > >> > defining the shape of the neighborhood area (i.e.
> circular
> > > or
> > > > > > > > >> > square). What is the default if you don't
specify?
> > > > > > > > >> > 2.) In the literature, there are a couple of
different
> > ways
> > > > to
> > > > > > > > >> > handle generating neighborhood probabilities
(e.g.,
> > Schwartz
> > > > and
> > > > > > > > >> > Sobash 2017;
> > > > > > > > >> >
https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-
> > > 0400.1
> > > > ).
> > > > > > I
> > > > > > > > >> > was curious is the only way to get the Fractions
Skill
> > Score
> > > > > (FSS)
> > > > > > > > >> > output is to define a neighborhood in which MET
> > calculates a
> > > > > > > > >> > fractional coverage of the threshold event? I
have
> > > previously
> > > > > > > > >> > explored computing the neighborhood maximum value
> within a
> > > > > radius
> > > > > > of
> > > > > > > > each grid point before calculating a probability.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chris
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > //SIGNED//
> > > > > > > > >> >
> > > > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > Verification
> > > > > > > > >> > Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> 557th
> > > > > Weather
> > > > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
294-3693
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > -----Original Message-----
> > > > > > > > >> > From: John Halley Gotway via RT [mailto:
> met_help at ucar.edu
> > ]
> > > > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16
WS/WXES <
> > > > > > > > >> > christopher.melick at us.af.mil>
> > > > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > > > robert.craig.2 at us.af.mil>
> > > > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> > Help
> > > > > with
> > > > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > > > >> >
> > > > > > > > >> > Hi Chris,
> > > > > > > > >> >
> > > > > > > > >> > I see that you're having difficulty computing
> neighborhood
> > > > > > > > >> > statistics in Grid-Stat.
> > > > > > > > >> >
> > > > > > > > >> > First, Grid-Stat definitely does have the ability
to
> > > threshold
> > > > > the
> > > > > > > > >> StageIV
> > > > > > > > >> > data on the fly. So pre-processing to 0's and
1's first
> > is
> > > > not
> > > > > > > > needed.
> > > > > > > > >> > Instead, in the "obs" dictionary of the config
file,
> you'd
> > > > just
> > > > > > set:
> > > > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > > > >> >
> > > > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is
the
> same
> > > as
> > > > > > >=0.1
> > > > > > > > >> inches.
> > > > > > > > >> >
> > > > > > > > >> > If for some reason you have data that's already
0's and
> > 1's,
> > > > > you'd
> > > > > > > > >> > just use a different threshold in the config
file:
> > > > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > > > >> >
> > > > > > > > >> > This tells MET that anywhere you see a 1 in the
data,
> the
> > > > event
> > > > > is
> > > > > > > > >> > occurring. Otherwise, it's not.
> > > > > > > > >> >
> > > > > > > > >> > Now on the to real issue. Looking at your config
file,
> I
> > > see
> > > > > that
> > > > > > > > >> you're
> > > > > > > > >> > processing probabilistic data. When "prob =
TRUE", MET
> > only
> > > > > > > > >> > computes
> > > > > > > > >> the
> > > > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and
PJC).
> > That's
> > > > why
> > > > > > > > >> > you're getting no output in the NBRCNT line type,
which
> is
> > > the
> > > > > one
> > > > > > > > >> > that
> > > > > > > > >> contains
> > > > > > > > >> > Fractions Skill Score (FSS).
> > > > > > > > >> >
> > > > > > > > >> > In the attached config file, I've made a few
changes:
> > > > > > > > >> > - Set the obtype to indicate that you're
verifying
> > against
> > > > > > StageIV
> > > > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > > > >> > - In the first we process QP010 as a
probability
> > > field.
> > > > > > Here
> > > > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation
> > for
> > > > > > > > >> > defining
> > > > > > > > >> bins
> > > > > > > > >> > for probabilistic verification from 0 to 1 of
width 0.1.
> > > > > > > > >> > - In the second we process it as a field of
> scalars.
> > > > Here
> > > > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > > > threshold
> > > > > > > > >> > I'm
> > > > > > > > >> using
> > > > > > > > >> > for defining events for fractions skill score.
> > > > > > > > >> > - Defined 2 corresponding obs.field entries to
verify
> the
> > > > > > forecast
> > > > > > > > >> data.
> > > > > > > > >> > In both the verifying threshold is >=2.54. This
assumes
> > that
> > > > you
> > > > > > > > >> > pass
> > > > > > > > >> the
> > > > > > > > >> > raw StageIV data as input.
> > > > > > > > >> > - In the output_flag section, I turned off the
line
> types
> > > > that
> > > > > > > > >> > don't apply.
> > > > > > > > >> >
> > > > > > > > >> > Lastly, while setting "level = [ "R19" ];" works
as a
> way
> > of
> > > > > > > > >> > grabbing record number 19 from the input file,
it's not
> > > > > > > > >> > recommended. Instead,
> > > > > > > > >> it'd
> > > > > > > > >> > be better to figure out how to set the level
string so
> > that
> > > > MET
> > > > > > > > >> > always grabs the same record, regardless if it's
in spot
> > > > number
> > > > > 19
> > > > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > > > >> >
> > > > > > > > >> > Hope this helps clarify. What other questions do
you
> > have?
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> >
> > > > > > > > >> > John Halley Gotway
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> > > christopher.melick at us.af.mil
> > > > > via
> > > > > > RT
> > > > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was
acted
> upon.
> > > > > > > > >> > > Transaction: Ticket created by
> > > christopher.melick at us.af.mil
> > > > > > > > >> > > Queue: met_help
> > > > > > > > >> > > Subject: Help with grid_stat on
Neighborhood
> > > statistics
> > > > > > > > >> > > Owner: Nobody
> > > > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > > > >> > > Status: new
> > > > > > > > >> > > Ticket <URL:
> > > > > > > > >> > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Hi John,
> > > > > > > > >> > >
> > > > > > > > >> > > I am trying to run grid_stat in MET to
calculate
> > > > Neighborhood
> > > > > > > > >> > > statistics for probability of precipitation
from some
> > > > ensemble
> > > > > > > > >> > > data with the observations
> > > > > > > > >> > > coming from Stage IV analyses. Unfortunately,
I am
> > > getting
> > > > > no
> > > > > > > > output
> > > > > > > > >> > > (just
> > > > > > > > >> > > headers) in many of the ASCII files from the
program
> > when
> > > I
> > > > > run
> > > > > > > it.
> > > > > > > > >> > Could
> > > > > > > > >> > > you help me figure out what I am doing wrong?
I have
> > > > provided
> > > > > > > > >> > > the command line specifics I use as well as the
output
> > log
> > > > > file
> > > > > > > > >> > > that is
> > > > > > > > >> > created when
> > > > > > > > >> > > running that command. I also have attached
the
> > > > configuration
> > > > > > > file.
> > > > > > > > >> I
> > > > > > > > >> > am
> > > > > > > > >> > > having trouble using ftp to send the raw data
from any
> > of
> > > > the
> > > > > > > > clients
> > > > > > > > >> > > available to us. I believe this restriction
occurred
> > > > > recently
> > > > > > > > since
> > > > > > > > >> Bob
> > > > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > > > externally
> > > > > > > > >> > > in the past.
> > > > > > > > >> > >
> > > > > > > > >> > > I should mention that I have pre-processed the
Stage
> IV
> > > > > analyses
> > > > > > > > >> > > already to be either 0's or 1's for the
threshold
> > (>=0.1")
> > > > > since
> > > > > > > > >> > > I was unclear as to whether MET was able to do
that on
> > the
> > > > fly
> > > > > > > > >> > > when running
> > > > > > > > >> > grid_stat.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Chris
> > > > > > > > >> > >
> > > > > > > > >> > > Python command:
> > > > > > > > >> > > Metcall = [metpath + 'grid_stat',
> > 'grib.2017091422.0006',
> > > > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> > > '-v',
> > > > > '5]
> > > > > > > > >> > > Call(metcall)
> > > > > > > > >> > >
> > > > > > > > >> > > //SIGNED//
> > > > > > > > >> > >
> > > > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron
> (16
> > > > > > WS/WXEV)
> > > > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-
3693; Comm
> > > (402)
> > > > > > > > >> > > 294-3693
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Mon Jun 11 09:46:20 2018
Hi John,
Thanks for the follow-up. You are correct about the security issues
and not being able to use met-7.0 right now. It was a pleasure to
meet you at the WAF NWP Conference.
Best regards,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, June 6, 2018 12:32 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with grid_stat on Neighborhood statistics
Hi Chris,
I went hunting through the met-help tickets and found this one that we
discussed during the poster session yesterday.
I've attached a PDF of the development ticket from JiRA which
describes the functionality we added in met-7.0 as a result of this
question.
Unfortunately though, I understand that you won't be able to make use
of
met-7.0 at the AF until we've addressed the security issues through
Fortify.
Thanks,
John
On Tue, Feb 27, 2018 at 11:58 AM, christopher.melick at us.af.mil via RT
< met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> John,
>
> I believe I have an idea of what some of the problem is when
computing
> AFSS in my particular case: When the original work on Fractions
Skill
> Score was published by Roberts and Lean (2008), it was designed
around the
> concept of a single deterministic model. Since then, that
statistic has
> been expanded to ensembles in a variety of ways to determine
> neighborhood probabilities as well as expanded to other
probabilistic forecasts which
> implicitly incorporate spatial uncertainty. Since the original
binary
> forecast fields are sometimes not available and/or possible, I don't
> know how the F_RATE fits into these new paradigms for computing an
> upper level benchmark for FSS (i.e. AFSS).
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, February 27, 2018 10:27 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> I looked at your python code and don't see anything obviously wrong
in the
> equations you've used. However, your computation of AFSS won't
match what
> MET is computing for the reasons we've already discussed. You're
computing
> fcst_rate and obs_rate as an average of the fractional coverage
values.
> MET computes them as a ratio of events occurring in the input
fields. As
> Tressa demonstrated, those will not necessarily result in the same
values.
> Since AFSS is computed using fcst_rate and obs_rate, the AFSS you
compute
> will not match the AFSS that MET computes.
>
> Thanks,
> John
>
> On Tue, Feb 27, 2018 at 8:21 AM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Did you ever get the chance to take a look at my Python program
and
> > see if you notice anything that needs to be corrected? I'm still
not
> > sure what to make of the values I obtained for AFSS and why they
are
> lower than FSS,
> > especially at the grid-scale. Is this even possible?
> >
> > Thanks,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
> > Sent: Wednesday, February 21, 2018 3:47 PM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi John,
> >
> > You are correct --- I never received the message below. Are the
average
> > coverage values what is used to compute the AFSS? I guess I am
still a
> > little confused - I was just using the first example to compute
the
> > O_RATE/F_RATE for AFSS. I'm not so sure exploring this option
would
> help
> > me in the example case I sent you because most events occurred in
the
> > interior of the domain. Also, wouldn't edge effects be eliminated
if the
> > vld_thresh is set to one in grid_stat since the entire
neighborhood must
> > contain valid data?
> >
> > If we leave the whole neighborhood averaging aside, and just focus
on the
> > computing AFSS at the grid scale (neighborhood = 1 grid point),
then my
> FSS
> > is still greater than AFSS. Missing data points on the analysis
grid
> only
> > result from different domain areas between that dataset and the
ensemble
> > forecast system.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
Wing,
> > Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 21, 2018 2:59 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > Tressa sent an email on this issue, but she didn't include
> > met_help at ucar.edu.
> > So I don't think you received it. I've cut-and-pasted it below:
> >
> > John and Chris,
> >
> > It is not generally true that the average of the coverage values
is the
> > base rate. I think it may sometimes be true, if there are no
events in
> edge
> > neighborhoods without full coverage.
> >
> > I worked up the following example with a 4x4 neighborhood:
> >
> > 1 0 1 1
> > 0 0 0 0
> > 1 0 1 1
> > 0 1 0 0
> >
> > The rate here is 7/16 = .4375
> >
> > Fractional coverage values for a 3x3 neighborhood are:
> >
> > 1/4 2/6 2/6 2/4
> > 2/6 4/9 4/9 4/6
> > 2/6 3/9 3/9 2/6
> > 2/4 3/6 3/6 2/4
> >
> > The average of these values is 0.4149306
> >
> > I suppose if each fraction had the same denominator (e.g. no
edges, or
> all
> > non events there) then we would add each event up for 9 different
> > neighborhoods and then divide by 9. In that case, it should work.
> >
> > Tressa
> >
> >
> > On Wed, Feb 21, 2018 at 8:36 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > >
> > > Hi John,
> > >
> > > I don't get negative values for AFSS but I do get values that
are less
> > > than what I calculate for FSS. I approach the procedure for
computing
> > all
> > > the metrics such that everything is on the same "playing
ground".
> Thus,
> > > F_RATE and O_RATE are only computed by the mask of defined grid
points
> on
> > > *BOTH* the observation and forecast grids. When I process the
sample
> > time
> > > period I sent you, F_RATE >> O_RATE which causes AFSS to be a
small
> > value.
> > > Is that not the approach that I should follow? Were you able
to
> > > examine my script and raw data file I sent you?
> > >
> > > Thanks again,
> > > Chris
> > >
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, February 20, 2018 8:20 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> > > grid_stat on Neighborhood statistics
> > >
> > > Chris,
> > >
> > > Your email got me thinking. Thinking through some toy examples
in my
> > > mind, I can see that probably yes, the mean of the fractional
coverage
> > field
> > > *should* be the same as the frequency of the event.
> > >
> > > I ran MET in the debugger and was able to stop at the point in
the
> logic
> > > where this is computed. At that point, we're passing around
arrays of
> > > fractional coverage field values as well as arrays of 0's and
1's
> > > indicating whether the event did or did not occur at each grid
point.
> > > Comparing the mean of these arrays, I find that they are close
but not
> > > identical. Sometimes one is larger, sometimes the other.
> > >
> > > I can think of 2 reasons why they wouldn't be the same, but
these may
> not
> > > fully explain it:
> > >
> > > (1) In Grid-Stat, the fractional coverage field is computed once
over
> the
> > > full verification domain. Then, the masking regions are
processed as
> > > subsets of these grid points. For a grid point that falls just
barely
> > > inside that mask, around 1/2 of the grid points used to compute
the
> > > fractional coverage value came from *OUTSIDE* that masking
region. The
> > 0/1
> > > arrays are only for points inside the current mask. So the mean
of the
> > 0/1
> > > array represents only the points inside the mask while the mean
of the
> > > fractional coverage includes info from outside the mask.
> > >
> > > (2) When data values are missing in the forecast or observation
fields,
> > > depending on how the user has set the vld_thresh config file
option,
> the
> > > "denominator" used to compute the fractional coverage field
value could
> > > change from grid point to grid point. That would likely throw
off the
> > mean
> > > of the fractional coverage field.
> > >
> > > Now what is the *CORRECT* way of computing the forecast and
observation
> > > rates? In papers about this, I see references to the
"base_rate" but
> no
> > > explicit equation.
> > >
> > > One very nice feature about the way MET is currently doing it is
that
> > > F_RATE and O_RATE remain constant regardless of the neighborhood
size.
> > > Therefore, UFSS and AFSS remain constant as well. Using the
other
> > method,
> > > O_RATE, F_RATE, UFSS, and AFSS would change for each
neighborhood size
> > > since the number of points included from outside the mask would
change.
> > >
> > > You mentioned that you had issues with negative UFSS and AFSS
values.
> > > Perhaps it's related to how you're computing the F_RATE and
O_RATE?
> > >
> > > Lots of little details to consider here!
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Tue, Feb 20, 2018 at 3:38 PM, christopher.melick at us.af.mil
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>
> > > >
> > > > John,
> > > >
> > > > After thinking about it some, I was curious why the F_RATE and
O_RATE
> > > > can't be calculated in the cases below? Can't the gridded
data still
> > > > be read in to determine the relative frequency of the events,
> > regardless
> > > > whether grid_stat is used to compute fractional coverage
fields? I
> am
> > > > basically doing that right now in some Python code I developed
when
> > > > calculating Fractions Skill Score.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, February 14, 2018 3:20 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help with
> > > > grid_stat on Neighborhood statistics
> > > >
> > > > Hi Chris,
> > > >
> > > > I'm working on the interp.field option we discussed.
> > > >
> > > > I have a quick question for you. The NBRCNT line type that
is
> > computed
> > > by
> > > > grid_stat contains columns for F_RATE and O_RATE. That is the
ratio
> of
> > > > grid points over a masking region at which the "event" is
occurring
> in
> > > the
> > > > forecast and observation fields respectively.
> > > >
> > > > When grid_stat computes the fractional coverage field, we can
compute
> > > > that. But if grid_stat does *NOT* compute the fractional
coverage
> > > fields,
> > > > then we don't know what those ratio's are.
> > > >
> > > > - If interp.field = FCST, then O_RATE = NA.
> > > > - If interp.field = OBS, then F_RATE = NA.
> > > > - If interp.field = NONE, then F_RATE = O_RATE = NA.
> > > >
> > > > Make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Feb 14, 2018 at 9:39 AM, christopher.melick at us.af.mil
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > >
> > > > > John,
> > > > >
> > > > > That sounds like an excellent plan. Thanks for your
assistance and
> > > > > trying to accommodate and make changes to the MET software
so
> > quickly.
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, February 14, 2018 10:36 AM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> with
> > > > > grid_stat on Neighborhood statistics
> > > > >
> > > > > Chris,
> > > > >
> > > > > Just wanted to close the loop on this ticket. I'll try to
get it
> > into
> > > > the
> > > > > next release due out next in March.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Feb 9, 2018 at 8:11 AM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yes, thanks for the explanations. That clears up any
possible
> > > > > > ambiguity and confusion. I kind of thought that was the
> procedure.
> > > > > >
> > > > > > When do you think that the next MET would be released with
the
> > > > > > enhancement that we discussed for the neighborhood
statistics?
> Is
> > it
> > > > > > an easy or difficult update to the software?
> > > > > >
> > > > > > Thanks again,
> > > > > > Chris
> > > > > >
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Thursday, February 8, 2018 3:24 PM
> > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > christopher.melick at us.af.mil>
> > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> robert.craig.2 at us.af.mil
> > >
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986]
Help
> > with
> > > > > > grid_stat on Neighborhood statistics
> > > > > >
> > > > > > Chris,
> > > > > >
> > > > > > It sounds like you understand how the fractional coverage
field
> is
> > > > > > computed.
> > > > > >
> > > > > > - Read the raw field... probability values between 0 and 1
in
> your
> > > > case.
> > > > > > - Apply the categorical threshold to that data to generate
a
> field
> > of
> > > > 0's
> > > > > > and 1's... >0.5 in your case.
> > > > > > - For each grid point, replace the value at that grid
point with
> > the
> > > > > > "fraction" of 1's in the neighborhood around that point...
e.g.
> the
> > > > event
> > > > > > occurs at 4 of the 9 grid points in a 3x3 square
surrounding the
> > > > current
> > > > > > point, so fractional coverage = 4/9.
> > > > > > - Compare the forecast fractional coverage values to the
observed
> > > > > > fractional coverage values to compute FSS.
> > > > > >
> > > > > > Yes, I do think it would be a good idea to talk to Craig
about
> how
> > he
> > > > > > configured MET for his work.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 8, 2018 at 2:09 PM,
christopher.melick at us.af.mil via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Getting back to your initial response, I was able to run
> > grid_stat
> > > > > > > with your changes in the attached configuration file to
get FSS
> > > > output
> > > > > > > (as well as the other statistics for the neighborhood
type
> > > results).
> > > > > > >
> > > > > > > However, I do have a question about the forecast
categorical
> > > > threshold
> > > > > > > (cat_thresh is set to >=0.5) used in determining the
> neighborhood
> > > > > > > probabilities from the ensemble probabilities: What is
the
> actual
> > > > > > > formula that MET uses to create the final field to
calculate
> FSS?
> > > Is
> > > > > > > it a strict neighborhood fractional coverage of ensemble
> > > > probabilities
> > > > > > > above 0.5 in some set radius of influence? Or is it
some other
> > > > > > > operation or does the threshold represent something
else? I
> know
> > > in
> > > > > > > Craig's paper, they were calling the last step a
neighborhood
> > > average
> > > > > > > of the ensemble probabilities to calculate NEP (since
the event
> > had
> > > > > > > already been defined at the grid point in determining
the
> > ensemble
> > > > > > probability [i.e. likelihood of exceeding
> > > > > > > 0.1" of precipitation]). Maybe, I should understand
how Craig
> > was
> > > > > using
> > > > > > > MET for his paper.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > Chris
> > > > > > >
> > > > > > >
> > > > > > > //SIGNED//
> > > > > > >
> > > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> > Verification
> > > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > Weather
> > > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Wednesday, February 7, 2018 7:14 PM
> > > > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > christopher.melick at us.af.mil>
> > > > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > matthew.dawson.8 at us.af.mil>;
> > > > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > robert.craig.2 at us.af.mil
> > > >
> > > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > with
> > > > > > > grid_stat on Neighborhood statistics
> > > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > I was thinking about what changes would be needed to
compute
> FSS
> > on
> > > > > input
> > > > > > > probability data. We could add a “field” option to the
“nbrhd”
> > > > > section,
> > > > > > > just like we have in the “interp” section. It would be
set to
> > > FCST,
> > > > > OBS,
> > > > > > > or BOTH (default) and would control the logic as to
which
> fields
> > > > should
> > > > > > be
> > > > > > > passed through the logic for deriving fractional
coverage
> fields.
> > > In
> > > > > > your
> > > > > > > case, you’d set it to OBS and we’d just use the raw
input
> > forecast
> > > > > > > probability values.
> > > > > > >
> > > > > > > Would that logic make sense to you?
> > > > > > >
> > > > > > > Thanks
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > In fact, SPC's 40-km radius probabilities were the
motivation
> > for
> > > > > > > > adding the shape = circle option to MET.
> > > > > > > >
> > > > > > > > For your first question, no, there is not currently a
> Gaussian
> > > > > > > > smoother option for the "interp" section in Grid-Stat.
> > > Currently,
> > > > > the
> > > > > > > > only valid options there are MIN, MAX, MEDIAN, and
UW_MEAN
> (for
> > > > > > > unweighted mean).
> > > > > > > > There is a distance-weighted mean option for point
data where
> > the
> > > > > > > > weight is 1/distance squared. But that doesn't work
for
> > > Grid-Stat
> > > > > > > > because we'd end up dividing by 0.
> > > > > > > >
> > > > > > > > I can see how adding a gaussian smoother would be
useful.
> > > > > > > >
> > > > > > > > Here's what we've done in the past with this SPC 40-km
> > > > probabilities:
> > > > > > > >
> > > > > > > > (1) Evaluate that field as a probability field (i.e.
set
> "prob
> > =
> > > > > > > > TRUE;" in the config file).
> > > > > > > > (2) Use the "interp" section to define a smoothing
operation
> on
> > > the
> > > > > > > > observations:
> > > > > > > > interp = {
> > > > > > > > field = OBS;
> > > > > > > > vld_thresh = 1.0;
> > > > > > > > shape = CIRCLE;
> > > > > > > > type = [
> > > > > > > > { method = MAX; width = 11; }
> > > > > > > > ];
> > > > > > > > }
> > > > > > > > Or set width to some circle diameter in grid units
that
> > works
> > > > out
> > > > > > > > to 40km in the real world.
> > > > > > > >
> > > > > > > > In this example, we post-process the observations to
make
> them
> > > > > > > > comparable the event for which the probability was
defined.
> > > > > > > >
> > > > > > > > But I think I understand what you'd like to do...
interpret
> as
> > > the
> > > > > > > > probability value at each grid point as if it was a
> fractional
> > > > > > > > coverage value. So don't apply a threshold and
neighborhood
> > size
> > > > to
> > > > > > > > the forecast data at all. Just pass it directly to
the
> routine
> > > > that
> > > > > > > computes FSS.
> > > > > > > >
> > > > > > > > Unfortunately no, I don't think there's a way of
configuring
> > > > > Grid-Stat
> > > > > > > > to do that in met-6.1. If you think this logic would
be
> > > generally
> > > > > > > > useful, we could consider adding a flag to the config
file to
> > > > enable
> > > > > > > that logic.
> > > > > > > >
> > > > > > > > Sorry I don't have better answers for you.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 7, 2018 at 4:14 PM,
christopher.melick at us.af.mil
> > via
> > > > RT
> > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >>
> > > > > > > >> <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83986
> > >
> > > > > > > >>
> > > > > > > >> Hi John,
> > > > > > > >>
> > > > > > > >> Your suggestion for accomplishing #2 sounds
promising. I
> need
> > > to
> > > > > > think
> > > > > > > >> the details through some. On a somewhat related
note, is
> > > there a
> > > > > > > Gaussian
> > > > > > > >> smoother flag for smoothing the probabilistic fields?
I
> know
> > > > that
> > > > > > was
> > > > > > > >> mentioned in Craig's paper and I have pursued that
option in
> > the
> > > > > past
> > > > > > > >> (but just not using MET).
> > > > > > > >>
> > > > > > > >> I guess my basic question is can you apply a
neighborhood
> > width
> > > > that
> > > > > > > >> just includes the grid point maybe because you
already have
> > > > > > > >> inherently a probabilistic field with spatial
uncertainty?
> > > Would
> > > > > that
> > > > > > > be a width of 1
> > > > > > > >> or does that also include the nearest grid point?
A good
> > > > example
> > > > > of
> > > > > > > this
> > > > > > > >> would be a probabilistic forecast issued from the
Storm
> > > Prediction
> > > > > > > >> Center where the probabilities implicitly assume a
40-km
> > radius
> > > of
> > > > > > > >> influence (so there is no need to compute a
fractional
> > coverage
> > > of
> > > > > the
> > > > > > > event).
> > > > > > > >>
> > > > > > > >> Thanks again,
> > > > > > > >> Chris
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> //SIGNED//
> > > > > > > >>
> > > > > > > >> Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > Verification
> > > > > > > >> Exploitation Team 16th Weather Squadron (16 WS/WXEV)
557th
> > > Weather
> > > > > > > >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> -----Original Message-----
> > > > > > > >> From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > >> Sent: Wednesday, February 7, 2018 4:49 PM
> > > > > > > >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > > > >> christopher.melick at us.af.mil>
> > > > > > > >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> > > > > > > >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12
USAF
> ACC
> > 16
> > > > > > > >> WS/WXN <robert.craig.2 at us.af.mil>
> > > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> > Help
> > > > > with
> > > > > > > >> grid_stat on Neighborhood statistics
> > > > > > > >>
> > > > > > > >> Chris,
> > > > > > > >>
> > > > > > > >> (1) The default shape is a square since that's what
MET
> > > previously
> > > > > > did.
> > > > > > > >> The circle is the *new* option.
> > > > > > > >>
> > > > > > > >> (2) For this one, the short answer is yes. FSS is
computed
> in
> > > MET
> > > > > by
> > > > > > > >> doing the following...
> > > > > > > >> - For each field separately (i.e. fcst and obs),
apply the
> > > > > > > >> threshold
> > > > > > > >> (cat_thresh) to convert the input data to a 0/1
bitmap.
> > > > > > > >> - Apply the neighborhood width and shape to compute
a
> > > fractional
> > > > > > > >> coverage value for each grid point.
> > > > > > > >> - Compare the fcst and obs fractional coverage
fields to
> > > compute
> > > > > > FSS.
> > > > > > > >>
> > > > > > > >> But there are some other options that you may find
useful.
> > The
> > > > > > "interp"
> > > > > > > >> section in the config file is used in Grid-Stat as a
> smoothing
> > > > > > > operation.
> > > > > > > >> After reading the raw input data, you can specify
if/how to
> > > smooth
> > > > > > > >> that data prior to computing stats. For example,
since you
> > > > > mentioned
> > > > > > > >> maximums, you could replace the value at each grid
point
> with
> > > the
> > > > > > > >> maximum of the values in a 5x5 box surrounding that
point:
> > > > > > > >>
> > > > > > > >> interp = {
> > > > > > > >> field = FCST;
> > > > > > > >> vld_thresh = 1.0;
> > > > > > > >> shape = SQUARE;
> > > > > > > >>
> > > > > > > >> type = [
> > > > > > > >> { method = MEAREST; width = 1; }, // i.e. no
> smoothing
> > > > > > > >> { method = MAX; width = 5; }, // i.e. max of
25
> closest
> > > > > points
> > > > > > > >> { method = UW_MEAN; width = 5; }, // i.e. mean
of the
> > 25
> > > > > > > >> closest points
> > > > > > > >> ];
> > > > > > > >> }
> > > > > > > >>
> > > > > > > >> Using the example above, you'll get 3 times the
amount of
> > output
> > > > > from
> > > > > > > MET.
> > > > > > > >> The first smoother really is no smoothing at all. It
would
> > run
> > > on
> > > > > > > >> the raw input data. The second smoother replaces the
value
> at
> > > > each
> > > > > > > >> grid point with the maximum of the 25 surrounding
points.
> The
> > > > third
> > > > > > > >> smoother replaces the value at each grid point with
the mean
> > of
> > > > > those
> > > > > > > 25.
> > > > > > > >>
> > > > > > > >> I worked with Craig Schwartz when he was running MET
for
> that
> > > > paper.
> > > > > > > >> My recollection is that he was able to use the
configuration
> > > > options
> > > > > > > >> of Grid-Stat to accomplish the types of processing he
was
> > after.
> > > > > But
> > > > > > > >> I don't recall all the details.
> > > > > > > >>
> > > > > > > >> If you explain exactly what you're trying to do, it
may be
> > > > possible
> > > > > > > >> to configure Grid-Stat to accomplish that.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Wed, Feb 7, 2018 at 3:26 PM,
> christopher.melick at us.af.mil
> > > via
> > > > > RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=83986
> > > >
> > > > > > > >> >
> > > > > > > >> > Hi John,
> > > > > > > >> >
> > > > > > > >> > Your explanation and suggestions are very helpful
and do
> > > explain
> > > > > my
> > > > > > > >> > problems quite substantially. I will have to give
it a
> try
> > to
> > > > see
> > > > > > > >> > what happens.
> > > > > > > >> >
> > > > > > > >> > I do have a couple of other minor questions dealing
with
> how
> > > the
> > > > > > > >> > neighborhood verification is performed:
> > > > > > > >> > 1.) I noticed in the documentation that there is a
flag
> for
> > > > > > > >> > defining the shape of the neighborhood area (i.e.
circular
> > or
> > > > > > > >> > square). What is the default if you don't specify?
> > > > > > > >> > 2.) In the literature, there are a couple of
different
> ways
> > > to
> > > > > > > >> > handle generating neighborhood probabilities (e.g.,
> Schwartz
> > > and
> > > > > > > >> > Sobash 2017;
> > > > > > > >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-
D-16-
> > 0400.1
> > > ).
> > > > > I
> > > > > > > >> > was curious is the only way to get the Fractions
Skill
> Score
> > > > (FSS)
> > > > > > > >> > output is to define a neighborhood in which MET
> calculates a
> > > > > > > >> > fractional coverage of the threshold event? I have
> > previously
> > > > > > > >> > explored computing the neighborhood maximum value
within a
> > > > radius
> > > > > of
> > > > > > > each grid point before calculating a probability.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Chris
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > //SIGNED//
> > > > > > > >> >
> > > > > > > >> > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > Verification
> > > > > > > >> > Exploitation Team 16th Weather Squadron (16
WS/WXEV) 557th
> > > > Weather
> > > > > > > >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-
3693
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > -----Original Message-----
> > > > > > > >> > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> ]
> > > > > > > >> > Sent: Wednesday, February 7, 2018 3:49 PM
> > > > > > > >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<
> > > > > > > >> > christopher.melick at us.af.mil>
> > > > > > > >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > > > > >> matthew.dawson.8 at us.af.mil>;
> > > > > > > >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <
> > > > > robert.craig.2 at us.af.mil>
> > > > > > > >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#83986]
> Help
> > > > with
> > > > > > > >> > grid_stat on Neighborhood statistics
> > > > > > > >> >
> > > > > > > >> > Hi Chris,
> > > > > > > >> >
> > > > > > > >> > I see that you're having difficulty computing
neighborhood
> > > > > > > >> > statistics in Grid-Stat.
> > > > > > > >> >
> > > > > > > >> > First, Grid-Stat definitely does have the ability
to
> > threshold
> > > > the
> > > > > > > >> StageIV
> > > > > > > >> > data on the fly. So pre-processing to 0's and 1's
first
> is
> > > not
> > > > > > > needed.
> > > > > > > >> > Instead, in the "obs" dictionary of the config
file, you'd
> > > just
> > > > > set:
> > > > > > > >> > cat_thresh = [ >=2.54 ];
> > > > > > > >> >
> > > > > > > >> > Since StageIV precip is millimeters, >=2.54 mm is
the same
> > as
> > > > > >=0.1
> > > > > > > >> inches.
> > > > > > > >> >
> > > > > > > >> > If for some reason you have data that's already 0's
and
> 1's,
> > > > you'd
> > > > > > > >> > just use a different threshold in the config file:
> > > > > > > >> > cat_thresh = [ ==1 ];
> > > > > > > >> >
> > > > > > > >> > This tells MET that anywhere you see a 1 in the
data, the
> > > event
> > > > is
> > > > > > > >> > occurring. Otherwise, it's not.
> > > > > > > >> >
> > > > > > > >> > Now on the to real issue. Looking at your config
file, I
> > see
> > > > that
> > > > > > > >> you're
> > > > > > > >> > processing probabilistic data. When "prob = TRUE",
MET
> only
> > > > > > > >> > computes
> > > > > > > >> the
> > > > > > > >> > probabilistic line types (PCT, PSTD, PRC, and PJC).
> That's
> > > why
> > > > > > > >> > you're getting no output in the NBRCNT line type,
which is
> > the
> > > > one
> > > > > > > >> > that
> > > > > > > >> contains
> > > > > > > >> > Fractions Skill Score (FSS).
> > > > > > > >> >
> > > > > > > >> > In the attached config file, I've made a few
changes:
> > > > > > > >> > - Set the obtype to indicate that you're verifying
> against
> > > > > StageIV
> > > > > > > >> > - Defined 2 fcst.field entries.
> > > > > > > >> > - In the first we process QP010 as a
probability
> > field.
> > > > > Here
> > > > > > > >> > cat_thresh is set to ==0.10 which is shorthand
notation
> for
> > > > > > > >> > defining
> > > > > > > >> bins
> > > > > > > >> > for probabilistic verification from 0 to 1 of width
0.1.
> > > > > > > >> > - In the second we process it as a field of
scalars.
> > > Here
> > > > > > > >> > cat_thresh is set to >=0.5. So that's the
probability
> > > threshold
> > > > > > > >> > I'm
> > > > > > > >> using
> > > > > > > >> > for defining events for fractions skill score.
> > > > > > > >> > - Defined 2 corresponding obs.field entries to
verify the
> > > > > forecast
> > > > > > > >> data.
> > > > > > > >> > In both the verifying threshold is >=2.54. This
assumes
> that
> > > you
> > > > > > > >> > pass
> > > > > > > >> the
> > > > > > > >> > raw StageIV data as input.
> > > > > > > >> > - In the output_flag section, I turned off the
line types
> > > that
> > > > > > > >> > don't apply.
> > > > > > > >> >
> > > > > > > >> > Lastly, while setting "level = [ "R19" ];" works as
a way
> of
> > > > > > > >> > grabbing record number 19 from the input file, it's
not
> > > > > > > >> > recommended. Instead,
> > > > > > > >> it'd
> > > > > > > >> > be better to figure out how to set the level string
so
> that
> > > MET
> > > > > > > >> > always grabs the same record, regardless if it's in
spot
> > > number
> > > > 19
> > > > > > > >> > or 20 or any other location in the GRIB file.
> > > > > > > >> >
> > > > > > > >> > Hope this helps clarify. What other questions do
you
> have?
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> > John Halley Gotway
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, Feb 7, 2018 at 2:12 PM,
> > christopher.melick at us.af.mil
> > > > via
> > > > > RT
> > > > > > > >> > < met_help at ucar.edu> wrote:
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted
upon.
> > > > > > > >> > > Transaction: Ticket created by
> > christopher.melick at us.af.mil
> > > > > > > >> > > Queue: met_help
> > > > > > > >> > > Subject: Help with grid_stat on Neighborhood
> > statistics
> > > > > > > >> > > Owner: Nobody
> > > > > > > >> > > Requestors: christopher.melick at us.af.mil
> > > > > > > >> > > Status: new
> > > > > > > >> > > Ticket <URL:
> > > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Hi John,
> > > > > > > >> > >
> > > > > > > >> > > I am trying to run grid_stat in MET to calculate
> > > Neighborhood
> > > > > > > >> > > statistics for probability of precipitation from
some
> > > ensemble
> > > > > > > >> > > data with the observations
> > > > > > > >> > > coming from Stage IV analyses. Unfortunately, I
am
> > getting
> > > > no
> > > > > > > output
> > > > > > > >> > > (just
> > > > > > > >> > > headers) in many of the ASCII files from the
program
> when
> > I
> > > > run
> > > > > > it.
> > > > > > > >> > Could
> > > > > > > >> > > you help me figure out what I am doing wrong? I
have
> > > provided
> > > > > > > >> > > the command line specifics I use as well as the
output
> log
> > > > file
> > > > > > > >> > > that is
> > > > > > > >> > created when
> > > > > > > >> > > running that command. I also have attached the
> > > configuration
> > > > > > file.
> > > > > > > >> I
> > > > > > > >> > am
> > > > > > > >> > > having trouble using ftp to send the raw data
from any
> of
> > > the
> > > > > > > clients
> > > > > > > >> > > available to us. I believe this restriction
occurred
> > > > recently
> > > > > > > since
> > > > > > > >> Bob
> > > > > > > >> > > Craig had mentioned that he had been able to
connect
> > > > externally
> > > > > > > >> > > in the past.
> > > > > > > >> > >
> > > > > > > >> > > I should mention that I have pre-processed the
Stage IV
> > > > analyses
> > > > > > > >> > > already to be either 0's or 1's for the threshold
> (>=0.1")
> > > > since
> > > > > > > >> > > I was unclear as to whether MET was able to do
that on
> the
> > > fly
> > > > > > > >> > > when running
> > > > > > > >> > grid_stat.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Chris
> > > > > > > >> > >
> > > > > > > >> > > Python command:
> > > > > > > >> > > Metcall = [metpath + 'grid_stat',
> 'grib.2017091422.0006',
> > > > > > > >> > > 'precip/post-processed-obs.nc', \
> > > > > > > >> > > 'GridStatConfig_POP', '-outdir',
'tmp',
> > '-v',
> > > > '5]
> > > > > > > >> > > Call(metcall)
> > > > > > > >> > >
> > > > > > > >> > > //SIGNED//
> > > > > > > >> > >
> > > > > > > >> > > Dr. Christopher J. Melick, DAF Civilian
Meteorologist,
> > > > > > > >> > > Verification Exploitation Team 16th Weather
Squadron (16
> > > > > WS/WXEV)
> > > > > > > >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693;
Comm
> > (402)
> > > > > > > >> > > 294-3693
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list