[Met_help] [rt.rap.ucar.edu #83986] History for Help with grid_stat on Neighborhood statistics
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 14 09:35:24 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John,
I am trying to run grid_stat in MET to calculate Neighborhood statistics for
probability of precipitation from some ensemble data with the observations
coming from Stage IV analyses. Unfortunately, I am getting no output (just
headers) in many of the ASCII files from the program when I run it. Could
you help me figure out what I am doing wrong? I have provided the command
line specifics I use as well as the output log file that is created when
running that command. I also have attached the configuration file. I am
having trouble using ftp to send the raw data from any of the clients
available to us. I believe this restriction occurred recently since Bob
Craig had mentioned that he had been able to connect externally in the
past.
I should mention that I have pre-processed the Stage IV analyses already to
be either 0's or 1's for the threshold (>=0.1") since I was unclear as to
whether MET was able to do that on the fly when running grid_stat.
Thanks,
Chris
Python command:
Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
'precip/post-processed-obs.nc', \
'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
Call(metcall)
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 14:49:07 2018
Hi Chris,
I see that you're having difficulty computing neighborhood statistics
in
Grid-Stat.
First, Grid-Stat definitely does have the ability to threshold the
StageIV
data on the fly. So pre-processing to 0's and 1's first is not
needed.
Instead, in the "obs" dictionary of the config file, you'd just set:
cat_thresh = [ >=2.54 ];
Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
If for some reason you have data that's already 0's and 1's, you'd
just use
a different threshold in the config file:
cat_thresh = [ ==1 ];
This tells MET that anywhere you see a 1 in the data, the event is
occurring. Otherwise, it's not.
Now on the to real issue. Looking at your config file, I see that
you're
processing probabilistic data. When "prob = TRUE", MET only computes
the
probabilistic line types (PCT, PSTD, PRC, and PJC). That's why you're
getting no output in the NBRCNT line type, which is the one that
contains
Fractions Skill Score (FSS).
In the attached config file, I've made a few changes:
- Set the obtype to indicate that you're verifying against StageIV
- Defined 2 fcst.field entries.
- In the first we process QP010 as a probability field. Here
cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
for probabilistic verification from 0 to 1 of width 0.1.
- In the second we process it as a field of scalars. Here
cat_thresh
is set to >=0.5. So that's the probability threshold I'm using for
defining events for fractions skill score.
- Defined 2 corresponding obs.field entries to verify the forecast
data.
In both the verifying threshold is >=2.54. This assumes that you pass
the
raw StageIV data as input.
- In the output_flag section, I turned off the line types that don't
apply.
Lastly, while setting "level = [ "R19" ];" works as a way of grabbing
record number 19 from the input file, it's not recommended. Instead,
it'd
be better to figure out how to set the level string so that MET always
grabs the same record, regardless if it's in spot number 19 or 20 or
any
other location in the GRIB file.
Hope this helps clarify. What other questions do you have?
Thanks,
John Halley Gotway
On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> Transaction: Ticket created by christopher.melick at us.af.mil
> Queue: met_help
> Subject: Help with grid_stat on Neighborhood statistics
> Owner: Nobody
> Requestors: christopher.melick at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
>
> Hi John,
>
> I am trying to run grid_stat in MET to calculate Neighborhood
statistics
> for
> probability of precipitation from some ensemble data with the
observations
> coming from Stage IV analyses. Unfortunately, I am getting no
output
> (just
> headers) in many of the ASCII files from the program when I run it.
Could
> you help me figure out what I am doing wrong? I have provided the
command
> line specifics I use as well as the output log file that is created
when
> running that command. I also have attached the configuration file.
I am
> having trouble using ftp to send the raw data from any of the
clients
> available to us. I believe this restriction occurred recently
since Bob
> Craig had mentioned that he had been able to connect externally in
the
> past.
>
> I should mention that I have pre-processed the Stage IV analyses
already to
> be either 0's or 1's for the threshold (>=0.1") since I was unclear
as to
> whether MET was able to do that on the fly when running grid_stat.
>
> Thanks,
> Chris
>
> Python command:
> Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> 'precip/post-processed-obs.nc', \
> 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> Call(metcall)
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 07 15:26:40 2018
Hi John,
Your explanation and suggestions are very helpful and do explain my
problems quite substantially. I will have to give it a try to see
what happens.
I do have a couple of other minor questions dealing with how the
neighborhood verification is performed:
1.) I noticed in the documentation that there is a flag for defining
the shape of the neighborhood area (i.e. circular or square). What is
the default if you don't specify?
2.) In the literature, there are a couple of different ways to handle
generating neighborhood probabilities (e.g., Schwartz and Sobash 2017;
https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I was
curious is the only way to get the Fractions Skill Score (FSS) output
is to define a neighborhood in which MET calculates a fractional
coverage of the threshold event? I have previously explored computing
the neighborhood maximum value within a radius of each grid point
before calculating a probability.
Thanks,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 3:49 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I see that you're having difficulty computing neighborhood statistics
in Grid-Stat.
First, Grid-Stat definitely does have the ability to threshold the
StageIV data on the fly. So pre-processing to 0's and 1's first is
not needed.
Instead, in the "obs" dictionary of the config file, you'd just set:
cat_thresh = [ >=2.54 ];
Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
If for some reason you have data that's already 0's and 1's, you'd
just use a different threshold in the config file:
cat_thresh = [ ==1 ];
This tells MET that anywhere you see a 1 in the data, the event is
occurring. Otherwise, it's not.
Now on the to real issue. Looking at your config file, I see that
you're processing probabilistic data. When "prob = TRUE", MET only
computes the probabilistic line types (PCT, PSTD, PRC, and PJC).
That's why you're getting no output in the NBRCNT line type, which is
the one that contains Fractions Skill Score (FSS).
In the attached config file, I've made a few changes:
- Set the obtype to indicate that you're verifying against StageIV
- Defined 2 fcst.field entries.
- In the first we process QP010 as a probability field. Here
cat_thresh is set to ==0.10 which is shorthand notation for defining
bins for probabilistic verification from 0 to 1 of width 0.1.
- In the second we process it as a field of scalars. Here
cat_thresh is set to >=0.5. So that's the probability threshold I'm
using for defining events for fractions skill score.
- Defined 2 corresponding obs.field entries to verify the forecast
data.
In both the verifying threshold is >=2.54. This assumes that you pass
the raw StageIV data as input.
- In the output_flag section, I turned off the line types that don't
apply.
Lastly, while setting "level = [ "R19" ];" works as a way of grabbing
record number 19 from the input file, it's not recommended. Instead,
it'd be better to figure out how to set the level string so that MET
always grabs the same record, regardless if it's in spot number 19 or
20 or any other location in the GRIB file.
Hope this helps clarify. What other questions do you have?
Thanks,
John Halley Gotway
On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> Transaction: Ticket created by christopher.melick at us.af.mil
> Queue: met_help
> Subject: Help with grid_stat on Neighborhood statistics
> Owner: Nobody
> Requestors: christopher.melick at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >
>
>
> Hi John,
>
> I am trying to run grid_stat in MET to calculate Neighborhood
> statistics for probability of precipitation from some ensemble data
> with the observations
> coming from Stage IV analyses. Unfortunately, I am getting no
output
> (just
> headers) in many of the ASCII files from the program when I run it.
Could
> you help me figure out what I am doing wrong? I have provided the
> command line specifics I use as well as the output log file that is
created when
> running that command. I also have attached the configuration file.
I am
> having trouble using ftp to send the raw data from any of the
clients
> available to us. I believe this restriction occurred recently
since Bob
> Craig had mentioned that he had been able to connect externally in
> the past.
>
> I should mention that I have pre-processed the Stage IV analyses
> already to be either 0's or 1's for the threshold (>=0.1") since I
was
> unclear as to whether MET was able to do that on the fly when
running grid_stat.
>
> Thanks,
> Chris
>
> Python command:
> Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> 'precip/post-processed-obs.nc', \
> 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> Call(metcall)
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 15:48:30 2018
Chris,
(1) The default shape is a square since that's what MET previously
did.
The circle is the *new* option.
(2) For this one, the short answer is yes. FSS is computed in MET by
doing
the following...
- For each field separately (i.e. fcst and obs), apply the threshold
(cat_thresh) to convert the input data to a 0/1 bitmap.
- Apply the neighborhood width and shape to compute a fractional
coverage
value for each grid point.
- Compare the fcst and obs fractional coverage fields to compute
FSS.
But there are some other options that you may find useful. The
"interp"
section in the config file is used in Grid-Stat as a smoothing
operation.
After reading the raw input data, you can specify if/how to smooth
that
data prior to computing stats. For example, since you mentioned
maximums,
you could replace the value at each grid point with the maximum of the
values in a 5x5 box surrounding that point:
interp = {
field = FCST;
vld_thresh = 1.0;
shape = SQUARE;
type = [
{ method = MEAREST; width = 1; }, // i.e. no smoothing
{ method = MAX; width = 5; }, // i.e. max of 25 closest points
{ method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
points
];
}
Using the example above, you'll get 3 times the amount of output from
MET.
The first smoother really is no smoothing at all. It would run on the
raw
input data. The second smoother replaces the value at each grid point
with
the maximum of the 25 surrounding points. The third smoother replaces
the
value at each grid point with the mean of those 25.
I worked with Craig Schwartz when he was running MET for that paper.
My
recollection is that he was able to use the configuration options of
Grid-Stat to accomplish the types of processing he was after. But I
don't
recall all the details.
If you explain exactly what you're trying to do, it may be possible to
configure Grid-Stat to accomplish that.
Thanks,
John
On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your explanation and suggestions are very helpful and do explain my
> problems quite substantially. I will have to give it a try to see
what
> happens.
>
> I do have a couple of other minor questions dealing with how the
> neighborhood verification is performed:
> 1.) I noticed in the documentation that there is a flag for
defining the
> shape of the neighborhood area (i.e. circular or square). What is
the
> default if you don't specify?
> 2.) In the literature, there are a couple of different ways to
handle
> generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> curious is the only way to get the Fractions Skill Score (FSS)
output is to
> define a neighborhood in which MET calculates a fractional coverage
of the
> threshold event? I have previously explored computing the
neighborhood
> maximum value within a radius of each grid point before calculating
a
> probability.
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 3:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I see that you're having difficulty computing neighborhood
statistics in
> Grid-Stat.
>
> First, Grid-Stat definitely does have the ability to threshold the
StageIV
> data on the fly. So pre-processing to 0's and 1's first is not
needed.
> Instead, in the "obs" dictionary of the config file, you'd just set:
> cat_thresh = [ >=2.54 ];
>
> Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
>
> If for some reason you have data that's already 0's and 1's, you'd
just
> use a different threshold in the config file:
> cat_thresh = [ ==1 ];
>
> This tells MET that anywhere you see a 1 in the data, the event is
> occurring. Otherwise, it's not.
>
> Now on the to real issue. Looking at your config file, I see that
you're
> processing probabilistic data. When "prob = TRUE", MET only
computes the
> probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> getting no output in the NBRCNT line type, which is the one that
contains
> Fractions Skill Score (FSS).
>
> In the attached config file, I've made a few changes:
> - Set the obtype to indicate that you're verifying against StageIV
> - Defined 2 fcst.field entries.
> - In the first we process QP010 as a probability field. Here
> cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
> for probabilistic verification from 0 to 1 of width 0.1.
> - In the second we process it as a field of scalars. Here
> cat_thresh is set to >=0.5. So that's the probability threshold I'm
using
> for defining events for fractions skill score.
> - Defined 2 corresponding obs.field entries to verify the forecast
data.
> In both the verifying threshold is >=2.54. This assumes that you
pass the
> raw StageIV data as input.
> - In the output_flag section, I turned off the line types that
don't
> apply.
>
> Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> record number 19 from the input file, it's not recommended.
Instead, it'd
> be better to figure out how to set the level string so that MET
always
> grabs the same record, regardless if it's in spot number 19 or 20 or
any
> other location in the GRIB file.
>
> Hope this helps clarify. What other questions do you have?
>
> Thanks,
>
> John Halley Gotway
>
>
>
> On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > Transaction: Ticket created by christopher.melick at us.af.mil
> > Queue: met_help
> > Subject: Help with grid_stat on Neighborhood statistics
> > Owner: Nobody
> > Requestors: christopher.melick at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >
> >
> >
> > Hi John,
> >
> > I am trying to run grid_stat in MET to calculate Neighborhood
> > statistics for probability of precipitation from some ensemble
data
> > with the observations
> > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > (just
> > headers) in many of the ASCII files from the program when I run
it.
> Could
> > you help me figure out what I am doing wrong? I have provided the
> > command line specifics I use as well as the output log file that
is
> created when
> > running that command. I also have attached the configuration
file. I
> am
> > having trouble using ftp to send the raw data from any of the
clients
> > available to us. I believe this restriction occurred recently
since Bob
> > Craig had mentioned that he had been able to connect externally
in
> > the past.
> >
> > I should mention that I have pre-processed the Stage IV analyses
> > already to be either 0's or 1's for the threshold (>=0.1") since I
was
> > unclear as to whether MET was able to do that on the fly when
running
> grid_stat.
> >
> > Thanks,
> > Chris
> >
> > Python command:
> > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > 'precip/post-processed-obs.nc', \
> > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > Call(metcall)
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Wed Feb 07 16:14:07 2018
Hi John,
Your suggestion for accomplishing #2 sounds promising. I need to
think the details through some. On a somewhat related note, is there
a Gaussian smoother flag for smoothing the probabilistic fields? I
know that was mentioned in Craig's paper and I have pursued that
option in the past (but just not using MET).
I guess my basic question is can you apply a neighborhood width that
just includes the grid point maybe because you already have inherently
a probabilistic field with spatial uncertainty? Would that be a width
of 1 or does that also include the nearest grid point? A good
example of this would be a probabilistic forecast issued from the
Storm Prediction Center where the probabilities implicitly assume a
40-km radius of influence (so there is no need to compute a fractional
coverage of the event).
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 4:49 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
(1) The default shape is a square since that's what MET previously
did.
The circle is the *new* option.
(2) For this one, the short answer is yes. FSS is computed in MET by
doing the following...
- For each field separately (i.e. fcst and obs), apply the threshold
(cat_thresh) to convert the input data to a 0/1 bitmap.
- Apply the neighborhood width and shape to compute a fractional
coverage value for each grid point.
- Compare the fcst and obs fractional coverage fields to compute
FSS.
But there are some other options that you may find useful. The
"interp"
section in the config file is used in Grid-Stat as a smoothing
operation.
After reading the raw input data, you can specify if/how to smooth
that data prior to computing stats. For example, since you mentioned
maximums, you could replace the value at each grid point with the
maximum of the values in a 5x5 box surrounding that point:
interp = {
field = FCST;
vld_thresh = 1.0;
shape = SQUARE;
type = [
{ method = MEAREST; width = 1; }, // i.e. no smoothing
{ method = MAX; width = 5; }, // i.e. max of 25 closest points
{ method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest points
];
}
Using the example above, you'll get 3 times the amount of output from
MET.
The first smoother really is no smoothing at all. It would run on the
raw input data. The second smoother replaces the value at each grid
point with the maximum of the 25 surrounding points. The third
smoother replaces the value at each grid point with the mean of those
25.
I worked with Craig Schwartz when he was running MET for that paper.
My recollection is that he was able to use the configuration options
of Grid-Stat to accomplish the types of processing he was after. But
I don't recall all the details.
If you explain exactly what you're trying to do, it may be possible to
configure Grid-Stat to accomplish that.
Thanks,
John
On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your explanation and suggestions are very helpful and do explain my
> problems quite substantially. I will have to give it a try to see
> what happens.
>
> I do have a couple of other minor questions dealing with how the
> neighborhood verification is performed:
> 1.) I noticed in the documentation that there is a flag for
defining
> the shape of the neighborhood area (i.e. circular or square). What
is
> the default if you don't specify?
> 2.) In the literature, there are a couple of different ways to
handle
> generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> curious is the only way to get the Fractions Skill Score (FSS)
output
> is to define a neighborhood in which MET calculates a fractional
> coverage of the threshold event? I have previously explored
computing
> the neighborhood maximum value within a radius of each grid point
> before calculating a probability.
>
> Thanks,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 3:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I see that you're having difficulty computing neighborhood
statistics in
> Grid-Stat.
>
> First, Grid-Stat definitely does have the ability to threshold the
StageIV
> data on the fly. So pre-processing to 0's and 1's first is not
needed.
> Instead, in the "obs" dictionary of the config file, you'd just set:
> cat_thresh = [ >=2.54 ];
>
> Since StageIV precip is millimeters, >=2.54 mm is the same as >=0.1
inches.
>
> If for some reason you have data that's already 0's and 1's, you'd
just
> use a different threshold in the config file:
> cat_thresh = [ ==1 ];
>
> This tells MET that anywhere you see a 1 in the data, the event is
> occurring. Otherwise, it's not.
>
> Now on the to real issue. Looking at your config file, I see that
you're
> processing probabilistic data. When "prob = TRUE", MET only
computes the
> probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> getting no output in the NBRCNT line type, which is the one that
contains
> Fractions Skill Score (FSS).
>
> In the attached config file, I've made a few changes:
> - Set the obtype to indicate that you're verifying against StageIV
> - Defined 2 fcst.field entries.
> - In the first we process QP010 as a probability field. Here
> cat_thresh is set to ==0.10 which is shorthand notation for defining
bins
> for probabilistic verification from 0 to 1 of width 0.1.
> - In the second we process it as a field of scalars. Here
> cat_thresh is set to >=0.5. So that's the probability threshold I'm
using
> for defining events for fractions skill score.
> - Defined 2 corresponding obs.field entries to verify the forecast
data.
> In both the verifying threshold is >=2.54. This assumes that you
pass the
> raw StageIV data as input.
> - In the output_flag section, I turned off the line types that
don't
> apply.
>
> Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> record number 19 from the input file, it's not recommended.
Instead, it'd
> be better to figure out how to set the level string so that MET
always
> grabs the same record, regardless if it's in spot number 19 or 20 or
any
> other location in the GRIB file.
>
> Hope this helps clarify. What other questions do you have?
>
> Thanks,
>
> John Halley Gotway
>
>
>
> On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > Transaction: Ticket created by christopher.melick at us.af.mil
> > Queue: met_help
> > Subject: Help with grid_stat on Neighborhood statistics
> > Owner: Nobody
> > Requestors: christopher.melick at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > >
> >
> >
> > Hi John,
> >
> > I am trying to run grid_stat in MET to calculate Neighborhood
> > statistics for probability of precipitation from some ensemble
data
> > with the observations
> > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > (just
> > headers) in many of the ASCII files from the program when I run
it.
> Could
> > you help me figure out what I am doing wrong? I have provided the
> > command line specifics I use as well as the output log file that
is
> created when
> > running that command. I also have attached the configuration
file. I
> am
> > having trouble using ftp to send the raw data from any of the
clients
> > available to us. I believe this restriction occurred recently
since Bob
> > Craig had mentioned that he had been able to connect externally
in
> > the past.
> >
> > I should mention that I have pre-processed the Stage IV analyses
> > already to be either 0's or 1's for the threshold (>=0.1") since I
was
> > unclear as to whether MET was able to do that on the fly when
running
> grid_stat.
> >
> > Thanks,
> > Chris
> >
> > Python command:
> > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > 'precip/post-processed-obs.nc', \
> > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > Call(metcall)
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 16:35:49 2018
Chris,
In fact, SPC's 40-km radius probabilities were the motivation for
adding
the shape = circle option to MET.
For your first question, no, there is not currently a Gaussian
smoother
option for the "interp" section in Grid-Stat. Currently, the only
valid
options there are MIN, MAX, MEDIAN, and UW_MEAN (for unweighted mean).
There is a distance-weighted mean option for point data where the
weight is
1/distance squared. But that doesn't work for Grid-Stat because we'd
end
up dividing by 0.
I can see how adding a gaussian smoother would be useful.
Here's what we've done in the past with this SPC 40-km probabilities:
(1) Evaluate that field as a probability field (i.e. set "prob =
TRUE;" in
the config file).
(2) Use the "interp" section to define a smoothing operation on the
observations:
interp = {
field = OBS;
vld_thresh = 1.0;
shape = CIRCLE;
type = [
{ method = MAX; width = 11; }
];
}
Or set width to some circle diameter in grid units that works out
to
40km in the real world.
In this example, we post-process the observations to make them
comparable
the event for which the probability was defined.
But I think I understand what you'd like to do... interpret as the
probability value at each grid point as if it was a fractional
coverage
value. So don't apply a threshold and neighborhood size to the
forecast
data at all. Just pass it directly to the routine that computes FSS.
Unfortunately no, I don't think there's a way of configuring Grid-Stat
to
do that in met-6.1. If you think this logic would be generally
useful, we
could consider adding a flag to the config file to enable that logic.
Sorry I don't have better answers for you.
John
On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Your suggestion for accomplishing #2 sounds promising. I need to
think
> the details through some. On a somewhat related note, is there a
Gaussian
> smoother flag for smoothing the probabilistic fields? I know that
was
> mentioned in Craig's paper and I have pursued that option in the
past (but
> just not using MET).
>
> I guess my basic question is can you apply a neighborhood width that
just
> includes the grid point maybe because you already have inherently a
> probabilistic field with spatial uncertainty? Would that be a width
of 1
> or does that also include the nearest grid point? A good example
of this
> would be a probabilistic forecast issued from the Storm Prediction
Center
> where the probabilities implicitly assume a 40-km radius of
influence (so
> there is no need to compute a fractional coverage of the event).
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 4:49 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Chris,
>
> (1) The default shape is a square since that's what MET previously
did.
> The circle is the *new* option.
>
> (2) For this one, the short answer is yes. FSS is computed in MET
by
> doing the following...
> - For each field separately (i.e. fcst and obs), apply the
threshold
> (cat_thresh) to convert the input data to a 0/1 bitmap.
> - Apply the neighborhood width and shape to compute a fractional
> coverage value for each grid point.
> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>
> But there are some other options that you may find useful. The
"interp"
> section in the config file is used in Grid-Stat as a smoothing
operation.
> After reading the raw input data, you can specify if/how to smooth
that
> data prior to computing stats. For example, since you mentioned
maximums,
> you could replace the value at each grid point with the maximum of
the
> values in a 5x5 box surrounding that point:
>
> interp = {
> field = FCST;
> vld_thresh = 1.0;
> shape = SQUARE;
>
> type = [
> { method = MEAREST; width = 1; }, // i.e. no smoothing
> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
> points
> ];
> }
>
> Using the example above, you'll get 3 times the amount of output
from MET.
> The first smoother really is no smoothing at all. It would run on
the raw
> input data. The second smoother replaces the value at each grid
point with
> the maximum of the 25 surrounding points. The third smoother
replaces the
> value at each grid point with the mean of those 25.
>
> I worked with Craig Schwartz when he was running MET for that paper.
My
> recollection is that he was able to use the configuration options of
> Grid-Stat to accomplish the types of processing he was after. But I
don't
> recall all the details.
>
> If you explain exactly what you're trying to do, it may be possible
to
> configure Grid-Stat to accomplish that.
>
> Thanks,
> John
>
> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >
> > Hi John,
> >
> > Your explanation and suggestions are very helpful and do explain
my
> > problems quite substantially. I will have to give it a try to see
> > what happens.
> >
> > I do have a couple of other minor questions dealing with how the
> > neighborhood verification is performed:
> > 1.) I noticed in the documentation that there is a flag for
defining
> > the shape of the neighborhood area (i.e. circular or square).
What is
> > the default if you don't specify?
> > 2.) In the literature, there are a couple of different ways to
handle
> > generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1). I
was
> > curious is the only way to get the Fractions Skill Score (FSS)
output
> > is to define a neighborhood in which MET calculates a fractional
> > coverage of the threshold event? I have previously explored
computing
> > the neighborhood maximum value within a radius of each grid point
> > before calculating a probability.
> >
> > Thanks,
> > Chris
> >
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, February 7, 2018 3:49 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> > grid_stat on Neighborhood statistics
> >
> > Hi Chris,
> >
> > I see that you're having difficulty computing neighborhood
statistics in
> > Grid-Stat.
> >
> > First, Grid-Stat definitely does have the ability to threshold the
> StageIV
> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> > cat_thresh = [ >=2.54 ];
> >
> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> inches.
> >
> > If for some reason you have data that's already 0's and 1's, you'd
just
> > use a different threshold in the config file:
> > cat_thresh = [ ==1 ];
> >
> > This tells MET that anywhere you see a 1 in the data, the event is
> > occurring. Otherwise, it's not.
> >
> > Now on the to real issue. Looking at your config file, I see that
you're
> > processing probabilistic data. When "prob = TRUE", MET only
computes the
> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
> > getting no output in the NBRCNT line type, which is the one that
contains
> > Fractions Skill Score (FSS).
> >
> > In the attached config file, I've made a few changes:
> > - Set the obtype to indicate that you're verifying against
StageIV
> > - Defined 2 fcst.field entries.
> > - In the first we process QP010 as a probability field.
Here
> > cat_thresh is set to ==0.10 which is shorthand notation for
defining bins
> > for probabilistic verification from 0 to 1 of width 0.1.
> > - In the second we process it as a field of scalars. Here
> > cat_thresh is set to >=0.5. So that's the probability threshold
I'm
> using
> > for defining events for fractions skill score.
> > - Defined 2 corresponding obs.field entries to verify the
forecast data.
> > In both the verifying threshold is >=2.54. This assumes that you
pass the
> > raw StageIV data as input.
> > - In the output_flag section, I turned off the line types that
don't
> > apply.
> >
> > Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
> > record number 19 from the input file, it's not recommended.
Instead,
> it'd
> > be better to figure out how to set the level string so that MET
always
> > grabs the same record, regardless if it's in spot number 19 or 20
or any
> > other location in the GRIB file.
> >
> > Hope this helps clarify. What other questions do you have?
> >
> > Thanks,
> >
> > John Halley Gotway
> >
> >
> >
> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> > > Transaction: Ticket created by christopher.melick at us.af.mil
> > > Queue: met_help
> > > Subject: Help with grid_stat on Neighborhood statistics
> > > Owner: Nobody
> > > Requestors: christopher.melick at us.af.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> > > >
> > >
> > >
> > > Hi John,
> > >
> > > I am trying to run grid_stat in MET to calculate Neighborhood
> > > statistics for probability of precipitation from some ensemble
data
> > > with the observations
> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
> > > (just
> > > headers) in many of the ASCII files from the program when I run
it.
> > Could
> > > you help me figure out what I am doing wrong? I have provided
the
> > > command line specifics I use as well as the output log file that
is
> > created when
> > > running that command. I also have attached the configuration
file. I
> > am
> > > having trouble using ftp to send the raw data from any of the
clients
> > > available to us. I believe this restriction occurred recently
since
> Bob
> > > Craig had mentioned that he had been able to connect externally
in
> > > the past.
> > >
> > > I should mention that I have pre-processed the Stage IV analyses
> > > already to be either 0's or 1's for the threshold (>=0.1") since
I was
> > > unclear as to whether MET was able to do that on the fly when
running
> > grid_stat.
> > >
> > > Thanks,
> > > Chris
> > >
> > > Python command:
> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> > > 'precip/post-processed-obs.nc', \
> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
> > > Call(metcall)
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Wed Feb 07 18:13:44 2018
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input
probability data. We could add a “field” option to the “nbrhd”
section,
just like we have in the “interp” section. It would be set to FCST,
OBS,
or BOTH (default) and would control the logic as to which fields
should be
passed through the logic for deriving fractional coverage fields. In
your
case, you’d set it to OBS and we’d just use the raw input forecast
probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
adding
> the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
smoother
> option for the "interp" section in Grid-Stat. Currently, the only
valid
> options there are MIN, MAX, MEDIAN, and UW_MEAN (for unweighted
mean).
> There is a distance-weighted mean option for point data where the
weight is
> 1/distance squared. But that doesn't work for Grid-Stat because
we'd end
> up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
TRUE;" in
> the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
to
> 40km in the real world.
>
> In this example, we post-process the observations to make them
comparable
> the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
coverage
> value. So don't apply a threshold and neighborhood size to the
forecast
> data at all. Just pass it directly to the routine that computes
FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat to
> do that in met-6.1. If you think this logic would be generally
useful, we
> could consider adding a flag to the config file to enable that
logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past (but
>> just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that just
>> includes the grid point maybe because you already have inherently a
>> probabilistic field with spatial uncertainty? Would that be a
width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
Center
>> where the probabilities implicitly assume a 40-km radius of
influence (so
>> there is no need to compute a fractional coverage of the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian
>> Meteorologist, Verification Exploitation Team
>> 16th Weather Squadron (16 WS/WXEV)
>> 557th Weather Wing, Offutt AFB, NE
>> DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
>> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
that
>> data prior to computing stats. For example, since you mentioned
maximums,
>> you could replace the value at each grid point with the maximum of
the
>> values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
closest
>> points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
the
>> raw input data. The second smoother replaces the value at each
grid point
>> with the maximum of the 25 surrounding points. The third smoother
replaces
>> the value at each grid point with the mean of those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper. My
>> recollection is that he was able to use the configuration options
of
>> Grid-Stat to accomplish the types of processing he was after. But
I don't
>> recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
to
>> configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
defining
>> > the shape of the neighborhood area (i.e. circular or square).
What is
>> > the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
handle
>> > generating neighborhood probabilities (e.g., Schwartz and Sobash
2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I was
>> > curious is the only way to get the Fractions Skill Score (FSS)
output
>> > is to define a neighborhood in which MET calculates a fractional
>> > coverage of the threshold event? I have previously explored
computing
>> > the neighborhood maximum value within a radius of each grid point
>> > before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
statistics in
>> > Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd just
>> > use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
you're
>> > getting no output in the NBRCNT line type, which is the one that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
don't
>> > apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
grabbing
>> > record number 19 from the input file, it's not recommended.
Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
always
>> > grabs the same record, regardless if it's in spot number 19 or 20
or any
>> > other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
data
>> > > with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
the
>> > > command line specifics I use as well as the output log file
that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally in
>> > > the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since I was
>> > > unclear as to whether MET was able to do that on the fly when
running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Thu Feb 08 08:00:11 2018
Hi John,
That sounds like a great solution to the situation I proposed for
computing FSS. My only other suggestion would be to add an additional
option in the case where both the OBS and FCST have been processed
before- hand to be probabilistic. In that case, you could call it
NONE. The default could still be BOTH, though.
How does that sound?
As for now, I will use what is available and it is completely
appropriate. I just wanted to make sure how everything was done
behind the scenes so I can explain to others how FSS is determined (if
it ever comes up). It would be nice to have down the road, though,
more flexibility built into how that metric is computed (not just for
myself but for others as well). I greatly appreciate your assistance.
On a slightly different note, is there a particular reference you
would recommend for citing MET when using the statistical output in
presentations?
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 7:14 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input probability data. We could add a “field” option to the “nbrhd”
section, just like we have in the “interp” section. It would be set
to FCST, OBS, or BOTH (default) and would control the logic as to
which fields should be passed through the logic for deriving
fractional coverage fields. In your case, you’d set it to OBS and
we’d just use the raw input forecast probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
> adding the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
> smoother option for the "interp" section in Grid-Stat. Currently,
the
> only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
unweighted mean).
> There is a distance-weighted mean option for point data where the
> weight is 1/distance squared. But that doesn't work for Grid-Stat
> because we'd end up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
> TRUE;" in the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
> to 40km in the real world.
>
> In this example, we post-process the observations to make them
> comparable the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
> coverage value. So don't apply a threshold and neighborhood size to
> the forecast data at all. Just pass it directly to the routine that
computes FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> to do that in met-6.1. If you think this logic would be generally
> useful, we could consider adding a flag to the config file to enable
that logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past
>> (but just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that
>> just includes the grid point maybe because you already have
>> inherently a probabilistic field with spatial uncertainty? Would
that be a width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
>> Center where the probabilities implicitly assume a 40-km radius of
>> influence (so there is no need to compute a fractional coverage of
the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
>> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
>> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
>> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
>> WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
>> threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
>> that data prior to computing stats. For example, since you
mentioned
>> maximums, you could replace the value at each grid point with the
>> maximum of the values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
>> closest points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
>> the raw input data. The second smoother replaces the value at each
>> grid point with the maximum of the 25 surrounding points. The
third
>> smoother replaces the value at each grid point with the mean of
those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper.
>> My recollection is that he was able to use the configuration
options
>> of Grid-Stat to accomplish the types of processing he was after.
But
>> I don't recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
>> to configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
>> > defining the shape of the neighborhood area (i.e. circular or
>> > square). What is the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
>> > handle generating neighborhood probabilities (e.g., Schwartz and
>> > Sobash 2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
>> > was curious is the only way to get the Fractions Skill Score
(FSS)
>> > output is to define a neighborhood in which MET calculates a
>> > fractional coverage of the threshold event? I have previously
>> > explored computing the neighborhood maximum value within a radius
of each grid point before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
>> > statistics in Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd
>> > just use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
>> > computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
>> > you're getting no output in the NBRCNT line type, which is the
one
>> > that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
>> > defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
>> > I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
>> > pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
>> > don't apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
>> > grabbing record number 19 from the input file, it's not
>> > recommended. Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
>> > always grabs the same record, regardless if it's in spot number
19
>> > or 20 or any other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT
>> > < met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
>> > > data with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
>> > > the command line specifics I use as well as the output log file
>> > > that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally
>> > > in the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since
>> > > I was unclear as to whether MET was able to do that on the fly
>> > > when running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
>> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
>> > > 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Thu Feb 08 14:09:23 2018
Hi John,
Getting back to your initial response, I was able to run grid_stat
with your changes in the attached configuration file to get FSS output
(as well as the other statistics for the neighborhood type results).
However, I do have a question about the forecast categorical threshold
(cat_thresh is set to >=0.5) used in determining the neighborhood
probabilities from the ensemble probabilities: What is the actual
formula that MET uses to create the final field to calculate FSS? Is
it a strict neighborhood fractional coverage of ensemble probabilities
above 0.5 in some set radius of influence? Or is it some other
operation or does the threshold represent something else? I know in
Craig's paper, they were calling the last step a neighborhood average
of the ensemble probabilities to calculate NEP (since the event had
already been defined at the grid point in determining the ensemble
probability [i.e. likelihood of exceeding 0.1" of precipitation]).
Maybe, I should understand how Craig was using MET for his paper.
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, February 7, 2018 7:14 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Hi Chris,
I was thinking about what changes would be needed to compute FSS on
input probability data. We could add a “field” option to the “nbrhd”
section, just like we have in the “interp” section. It would be set
to FCST, OBS, or BOTH (default) and would control the logic as to
which fields should be passed through the logic for deriving
fractional coverage fields. In your case, you’d set it to OBS and
we’d just use the raw input forecast probability values.
Would that logic make sense to you?
Thanks
John
On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Chris,
>
> In fact, SPC's 40-km radius probabilities were the motivation for
> adding the shape = circle option to MET.
>
> For your first question, no, there is not currently a Gaussian
> smoother option for the "interp" section in Grid-Stat. Currently,
the
> only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
unweighted mean).
> There is a distance-weighted mean option for point data where the
> weight is 1/distance squared. But that doesn't work for Grid-Stat
> because we'd end up dividing by 0.
>
> I can see how adding a gaussian smoother would be useful.
>
> Here's what we've done in the past with this SPC 40-km
probabilities:
>
> (1) Evaluate that field as a probability field (i.e. set "prob =
> TRUE;" in the config file).
> (2) Use the "interp" section to define a smoothing operation on the
> observations:
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
> type = [
> { method = MAX; width = 11; }
> ];
> }
> Or set width to some circle diameter in grid units that works out
> to 40km in the real world.
>
> In this example, we post-process the observations to make them
> comparable the event for which the probability was defined.
>
> But I think I understand what you'd like to do... interpret as the
> probability value at each grid point as if it was a fractional
> coverage value. So don't apply a threshold and neighborhood size to
> the forecast data at all. Just pass it directly to the routine that
computes FSS.
>
> Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> to do that in met-6.1. If you think this logic would be generally
> useful, we could consider adding a flag to the config file to enable
that logic.
>
> Sorry I don't have better answers for you.
>
> John
>
>
>
> On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>>
>> Hi John,
>>
>> Your suggestion for accomplishing #2 sounds promising. I need to
think
>> the details through some. On a somewhat related note, is there a
Gaussian
>> smoother flag for smoothing the probabilistic fields? I know that
was
>> mentioned in Craig's paper and I have pursued that option in the
past
>> (but just not using MET).
>>
>> I guess my basic question is can you apply a neighborhood width
that
>> just includes the grid point maybe because you already have
>> inherently a probabilistic field with spatial uncertainty? Would
that be a width of 1
>> or does that also include the nearest grid point? A good example
of this
>> would be a probabilistic forecast issued from the Storm Prediction
>> Center where the probabilities implicitly assume a 40-km radius of
>> influence (so there is no need to compute a fractional coverage of
the event).
>>
>> Thanks again,
>> Chris
>>
>>
>> //SIGNED//
>>
>> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
>> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
>> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, February 7, 2018 4:49 PM
>> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> christopher.melick at us.af.mil>
>> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
>> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
>> WS/WXN <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
>> grid_stat on Neighborhood statistics
>>
>> Chris,
>>
>> (1) The default shape is a square since that's what MET previously
did.
>> The circle is the *new* option.
>>
>> (2) For this one, the short answer is yes. FSS is computed in MET
by
>> doing the following...
>> - For each field separately (i.e. fcst and obs), apply the
>> threshold
>> (cat_thresh) to convert the input data to a 0/1 bitmap.
>> - Apply the neighborhood width and shape to compute a fractional
>> coverage value for each grid point.
>> - Compare the fcst and obs fractional coverage fields to compute
FSS.
>>
>> But there are some other options that you may find useful. The
"interp"
>> section in the config file is used in Grid-Stat as a smoothing
operation.
>> After reading the raw input data, you can specify if/how to smooth
>> that data prior to computing stats. For example, since you
mentioned
>> maximums, you could replace the value at each grid point with the
>> maximum of the values in a 5x5 box surrounding that point:
>>
>> interp = {
>> field = FCST;
>> vld_thresh = 1.0;
>> shape = SQUARE;
>>
>> type = [
>> { method = MEAREST; width = 1; }, // i.e. no smoothing
>> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
>> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
>> closest points
>> ];
>> }
>>
>> Using the example above, you'll get 3 times the amount of output
from MET.
>> The first smoother really is no smoothing at all. It would run on
>> the raw input data. The second smoother replaces the value at each
>> grid point with the maximum of the 25 surrounding points. The
third
>> smoother replaces the value at each grid point with the mean of
those 25.
>>
>> I worked with Craig Schwartz when he was running MET for that
paper.
>> My recollection is that he was able to use the configuration
options
>> of Grid-Stat to accomplish the types of processing he was after.
But
>> I don't recall all the details.
>>
>> If you explain exactly what you're trying to do, it may be possible
>> to configure Grid-Stat to accomplish that.
>>
>> Thanks,
>> John
>>
>> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>> >
>> > Hi John,
>> >
>> > Your explanation and suggestions are very helpful and do explain
my
>> > problems quite substantially. I will have to give it a try to
see
>> > what happens.
>> >
>> > I do have a couple of other minor questions dealing with how the
>> > neighborhood verification is performed:
>> > 1.) I noticed in the documentation that there is a flag for
>> > defining the shape of the neighborhood area (i.e. circular or
>> > square). What is the default if you don't specify?
>> > 2.) In the literature, there are a couple of different ways to
>> > handle generating neighborhood probabilities (e.g., Schwartz and
>> > Sobash 2017;
>> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
>> > was curious is the only way to get the Fractions Skill Score
(FSS)
>> > output is to define a neighborhood in which MET calculates a
>> > fractional coverage of the threshold event? I have previously
>> > explored computing the neighborhood maximum value within a radius
of each grid point before calculating a probability.
>> >
>> > Thanks,
>> > Chris
>> >
>> >
>> > //SIGNED//
>> >
>> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
>> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
>> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, February 7, 2018 3:49 PM
>> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
>> > christopher.melick at us.af.mil>
>> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
>> matthew.dawson.8 at us.af.mil>;
>> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
>> > grid_stat on Neighborhood statistics
>> >
>> > Hi Chris,
>> >
>> > I see that you're having difficulty computing neighborhood
>> > statistics in Grid-Stat.
>> >
>> > First, Grid-Stat definitely does have the ability to threshold
the
>> StageIV
>> > data on the fly. So pre-processing to 0's and 1's first is not
needed.
>> > Instead, in the "obs" dictionary of the config file, you'd just
set:
>> > cat_thresh = [ >=2.54 ];
>> >
>> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
>> inches.
>> >
>> > If for some reason you have data that's already 0's and 1's,
you'd
>> > just use a different threshold in the config file:
>> > cat_thresh = [ ==1 ];
>> >
>> > This tells MET that anywhere you see a 1 in the data, the event
is
>> > occurring. Otherwise, it's not.
>> >
>> > Now on the to real issue. Looking at your config file, I see
that
>> you're
>> > processing probabilistic data. When "prob = TRUE", MET only
>> > computes
>> the
>> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
>> > you're getting no output in the NBRCNT line type, which is the
one
>> > that
>> contains
>> > Fractions Skill Score (FSS).
>> >
>> > In the attached config file, I've made a few changes:
>> > - Set the obtype to indicate that you're verifying against
StageIV
>> > - Defined 2 fcst.field entries.
>> > - In the first we process QP010 as a probability field.
Here
>> > cat_thresh is set to ==0.10 which is shorthand notation for
>> > defining
>> bins
>> > for probabilistic verification from 0 to 1 of width 0.1.
>> > - In the second we process it as a field of scalars. Here
>> > cat_thresh is set to >=0.5. So that's the probability threshold
>> > I'm
>> using
>> > for defining events for fractions skill score.
>> > - Defined 2 corresponding obs.field entries to verify the
forecast
>> data.
>> > In both the verifying threshold is >=2.54. This assumes that you
>> > pass
>> the
>> > raw StageIV data as input.
>> > - In the output_flag section, I turned off the line types that
>> > don't apply.
>> >
>> > Lastly, while setting "level = [ "R19" ];" works as a way of
>> > grabbing record number 19 from the input file, it's not
>> > recommended. Instead,
>> it'd
>> > be better to figure out how to set the level string so that MET
>> > always grabs the same record, regardless if it's in spot number
19
>> > or 20 or any other location in the GRIB file.
>> >
>> > Hope this helps clarify. What other questions do you have?
>> >
>> > Thanks,
>> >
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil via
RT
>> > < met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
>> > > Transaction: Ticket created by christopher.melick at us.af.mil
>> > > Queue: met_help
>> > > Subject: Help with grid_stat on Neighborhood statistics
>> > > Owner: Nobody
>> > > Requestors: christopher.melick at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
>> > > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I am trying to run grid_stat in MET to calculate Neighborhood
>> > > statistics for probability of precipitation from some ensemble
>> > > data with the observations
>> > > coming from Stage IV analyses. Unfortunately, I am getting no
output
>> > > (just
>> > > headers) in many of the ASCII files from the program when I run
it.
>> > Could
>> > > you help me figure out what I am doing wrong? I have provided
>> > > the command line specifics I use as well as the output log file
>> > > that is
>> > created when
>> > > running that command. I also have attached the configuration
file.
>> I
>> > am
>> > > having trouble using ftp to send the raw data from any of the
clients
>> > > available to us. I believe this restriction occurred recently
since
>> Bob
>> > > Craig had mentioned that he had been able to connect
externally
>> > > in the past.
>> > >
>> > > I should mention that I have pre-processed the Stage IV
analyses
>> > > already to be either 0's or 1's for the threshold (>=0.1")
since
>> > > I was unclear as to whether MET was able to do that on the fly
>> > > when running
>> > grid_stat.
>> > >
>> > > Thanks,
>> > > Chris
>> > >
>> > > Python command:
>> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
>> > > 'precip/post-processed-obs.nc', \
>> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v', '5]
>> > > Call(metcall)
>> > >
>> > > //SIGNED//
>> > >
>> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
>> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
>> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
>> > > 294-3693
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
------------------------------------------------
Subject: Help with grid_stat on Neighborhood statistics
From: John Halley Gotway
Time: Thu Feb 08 14:24:26 2018
Chris,
It sounds like you understand how the fractional coverage field is
computed.
- Read the raw field... probability values between 0 and 1 in your
case.
- Apply the categorical threshold to that data to generate a field of
0's
and 1's... >0.5 in your case.
- For each grid point, replace the value at that grid point with the
"fraction" of 1's in the neighborhood around that point... e.g. the
event
occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current
point, so fractional coverage = 4/9.
- Compare the forecast fractional coverage values to the observed
fractional coverage values to compute FSS.
Yes, I do think it would be a good idea to talk to Craig about how he
configured MET for his work.
Thanks,
John
On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Getting back to your initial response, I was able to run grid_stat
with
> your changes in the attached configuration file to get FSS output
(as well
> as the other statistics for the neighborhood type results).
>
> However, I do have a question about the forecast categorical
threshold
> (cat_thresh is set to >=0.5) used in determining the neighborhood
> probabilities from the ensemble probabilities: What is the actual
formula
> that MET uses to create the final field to calculate FSS? Is it a
strict
> neighborhood fractional coverage of ensemble probabilities above 0.5
in
> some set radius of influence? Or is it some other operation or does
the
> threshold represent something else? I know in Craig's paper, they
were
> calling the last step a neighborhood average of the ensemble
probabilities
> to calculate NEP (since the event had already been defined at the
grid
> point in determining the ensemble probability [i.e. likelihood of
exceeding
> 0.1" of precipitation]). Maybe, I should understand how Craig was
using
> MET for his paper.
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 7:14 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I was thinking about what changes would be needed to compute FSS on
input
> probability data. We could add a “field” option to the “nbrhd”
section,
> just like we have in the “interp” section. It would be set to FCST,
OBS,
> or BOTH (default) and would control the logic as to which fields
should be
> passed through the logic for deriving fractional coverage fields.
In your
> case, you’d set it to OBS and we’d just use the raw input forecast
> probability values.
>
> Would that logic make sense to you?
>
> Thanks
> John
>
> On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Chris,
> >
> > In fact, SPC's 40-km radius probabilities were the motivation for
> > adding the shape = circle option to MET.
> >
> > For your first question, no, there is not currently a Gaussian
> > smoother option for the "interp" section in Grid-Stat. Currently,
the
> > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> unweighted mean).
> > There is a distance-weighted mean option for point data where the
> > weight is 1/distance squared. But that doesn't work for Grid-Stat
> > because we'd end up dividing by 0.
> >
> > I can see how adding a gaussian smoother would be useful.
> >
> > Here's what we've done in the past with this SPC 40-km
probabilities:
> >
> > (1) Evaluate that field as a probability field (i.e. set "prob =
> > TRUE;" in the config file).
> > (2) Use the "interp" section to define a smoothing operation on
the
> > observations:
> > interp = {
> > field = OBS;
> > vld_thresh = 1.0;
> > shape = CIRCLE;
> > type = [
> > { method = MAX; width = 11; }
> > ];
> > }
> > Or set width to some circle diameter in grid units that works
out
> > to 40km in the real world.
> >
> > In this example, we post-process the observations to make them
> > comparable the event for which the probability was defined.
> >
> > But I think I understand what you'd like to do... interpret as the
> > probability value at each grid point as if it was a fractional
> > coverage value. So don't apply a threshold and neighborhood size
to
> > the forecast data at all. Just pass it directly to the routine
that
> computes FSS.
> >
> > Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> > to do that in met-6.1. If you think this logic would be generally
> > useful, we could consider adding a flag to the config file to
enable
> that logic.
> >
> > Sorry I don't have better answers for you.
> >
> > John
> >
> >
> >
> > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >>
> >> Hi John,
> >>
> >> Your suggestion for accomplishing #2 sounds promising. I need to
think
> >> the details through some. On a somewhat related note, is there
a
> Gaussian
> >> smoother flag for smoothing the probabilistic fields? I know
that was
> >> mentioned in Craig's paper and I have pursued that option in the
past
> >> (but just not using MET).
> >>
> >> I guess my basic question is can you apply a neighborhood width
that
> >> just includes the grid point maybe because you already have
> >> inherently a probabilistic field with spatial uncertainty? Would
that
> be a width of 1
> >> or does that also include the nearest grid point? A good
example of
> this
> >> would be a probabilistic forecast issued from the Storm
Prediction
> >> Center where the probabilities implicitly assume a 40-km radius
of
> >> influence (so there is no need to compute a fractional coverage
of the
> event).
> >>
> >> Thanks again,
> >> Chris
> >>
> >>
> >> //SIGNED//
> >>
> >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, February 7, 2018 4:49 PM
> >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> christopher.melick at us.af.mil>
> >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> >> WS/WXN <robert.craig.2 at us.af.mil>
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> grid_stat on Neighborhood statistics
> >>
> >> Chris,
> >>
> >> (1) The default shape is a square since that's what MET
previously did.
> >> The circle is the *new* option.
> >>
> >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> >> doing the following...
> >> - For each field separately (i.e. fcst and obs), apply the
> >> threshold
> >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> >> - Apply the neighborhood width and shape to compute a
fractional
> >> coverage value for each grid point.
> >> - Compare the fcst and obs fractional coverage fields to
compute FSS.
> >>
> >> But there are some other options that you may find useful. The
"interp"
> >> section in the config file is used in Grid-Stat as a smoothing
> operation.
> >> After reading the raw input data, you can specify if/how to
smooth
> >> that data prior to computing stats. For example, since you
mentioned
> >> maximums, you could replace the value at each grid point with the
> >> maximum of the values in a 5x5 box surrounding that point:
> >>
> >> interp = {
> >> field = FCST;
> >> vld_thresh = 1.0;
> >> shape = SQUARE;
> >>
> >> type = [
> >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> >> closest points
> >> ];
> >> }
> >>
> >> Using the example above, you'll get 3 times the amount of output
from
> MET.
> >> The first smoother really is no smoothing at all. It would run
on
> >> the raw input data. The second smoother replaces the value at
each
> >> grid point with the maximum of the 25 surrounding points. The
third
> >> smoother replaces the value at each grid point with the mean of
those
> 25.
> >>
> >> I worked with Craig Schwartz when he was running MET for that
paper.
> >> My recollection is that he was able to use the configuration
options
> >> of Grid-Stat to accomplish the types of processing he was after.
But
> >> I don't recall all the details.
> >>
> >> If you explain exactly what you're trying to do, it may be
possible
> >> to configure Grid-Stat to accomplish that.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >> >
> >> > Hi John,
> >> >
> >> > Your explanation and suggestions are very helpful and do
explain my
> >> > problems quite substantially. I will have to give it a try to
see
> >> > what happens.
> >> >
> >> > I do have a couple of other minor questions dealing with how
the
> >> > neighborhood verification is performed:
> >> > 1.) I noticed in the documentation that there is a flag for
> >> > defining the shape of the neighborhood area (i.e. circular or
> >> > square). What is the default if you don't specify?
> >> > 2.) In the literature, there are a couple of different ways to
> >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> >> > Sobash 2017;
> >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
> >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> >> > output is to define a neighborhood in which MET calculates a
> >> > fractional coverage of the threshold event? I have previously
> >> > explored computing the neighborhood maximum value within a
radius of
> each grid point before calculating a probability.
> >> >
> >> > Thanks,
> >> > Chris
> >> >
> >> >
> >> > //SIGNED//
> >> >
> >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, February 7, 2018 3:49 PM
> >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> > christopher.melick at us.af.mil>
> >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> >> matthew.dawson.8 at us.af.mil>;
> >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> > grid_stat on Neighborhood statistics
> >> >
> >> > Hi Chris,
> >> >
> >> > I see that you're having difficulty computing neighborhood
> >> > statistics in Grid-Stat.
> >> >
> >> > First, Grid-Stat definitely does have the ability to threshold
the
> >> StageIV
> >> > data on the fly. So pre-processing to 0's and 1's first is not
> needed.
> >> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> >> > cat_thresh = [ >=2.54 ];
> >> >
> >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> >> inches.
> >> >
> >> > If for some reason you have data that's already 0's and 1's,
you'd
> >> > just use a different threshold in the config file:
> >> > cat_thresh = [ ==1 ];
> >> >
> >> > This tells MET that anywhere you see a 1 in the data, the event
is
> >> > occurring. Otherwise, it's not.
> >> >
> >> > Now on the to real issue. Looking at your config file, I see
that
> >> you're
> >> > processing probabilistic data. When "prob = TRUE", MET only
> >> > computes
> >> the
> >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
> >> > you're getting no output in the NBRCNT line type, which is the
one
> >> > that
> >> contains
> >> > Fractions Skill Score (FSS).
> >> >
> >> > In the attached config file, I've made a few changes:
> >> > - Set the obtype to indicate that you're verifying against
StageIV
> >> > - Defined 2 fcst.field entries.
> >> > - In the first we process QP010 as a probability field.
Here
> >> > cat_thresh is set to ==0.10 which is shorthand notation for
> >> > defining
> >> bins
> >> > for probabilistic verification from 0 to 1 of width 0.1.
> >> > - In the second we process it as a field of scalars.
Here
> >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> >> > I'm
> >> using
> >> > for defining events for fractions skill score.
> >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> >> data.
> >> > In both the verifying threshold is >=2.54. This assumes that
you
> >> > pass
> >> the
> >> > raw StageIV data as input.
> >> > - In the output_flag section, I turned off the line types that
> >> > don't apply.
> >> >
> >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> >> > grabbing record number 19 from the input file, it's not
> >> > recommended. Instead,
> >> it'd
> >> > be better to figure out how to set the level string so that MET
> >> > always grabs the same record, regardless if it's in spot number
19
> >> > or 20 or any other location in the GRIB file.
> >> >
> >> > Hope this helps clarify. What other questions do you have?
> >> >
> >> > Thanks,
> >> >
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> >> > < met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: Help with grid_stat on Neighborhood statistics
> >> > > Owner: Nobody
> >> > > Requestors: christopher.melick at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >> > > >
> >> > >
> >> > >
> >> > > Hi John,
> >> > >
> >> > > I am trying to run grid_stat in MET to calculate Neighborhood
> >> > > statistics for probability of precipitation from some
ensemble
> >> > > data with the observations
> >> > > coming from Stage IV analyses. Unfortunately, I am getting
no
> output
> >> > > (just
> >> > > headers) in many of the ASCII files from the program when I
run it.
> >> > Could
> >> > > you help me figure out what I am doing wrong? I have
provided
> >> > > the command line specifics I use as well as the output log
file
> >> > > that is
> >> > created when
> >> > > running that command. I also have attached the
configuration file.
> >> I
> >> > am
> >> > > having trouble using ftp to send the raw data from any of the
> clients
> >> > > available to us. I believe this restriction occurred
recently
> since
> >> Bob
> >> > > Craig had mentioned that he had been able to connect
externally
> >> > > in the past.
> >> > >
> >> > > I should mention that I have pre-processed the Stage IV
analyses
> >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> >> > > I was unclear as to whether MET was able to do that on the
fly
> >> > > when running
> >> > grid_stat.
> >> > >
> >> > > Thanks,
> >> > > Chris
> >> > >
> >> > > Python command:
> >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> >> > > 'precip/post-processed-obs.nc', \
> >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> >> > > Call(metcall)
> >> > >
> >> > > //SIGNED//
> >> > >
> >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> >> > > 294-3693
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >>
> >>
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with grid_stat on Neighborhood statistics
From: christopher.melick at us.af.mil
Time: Fri Feb 09 08:10:22 2018
Hi John,
Yes, thanks for the explanations. That clears up any possible
ambiguity and confusion. I kind of thought that was the procedure.
When do you think that the next MET would be released with the
enhancement that we discussed for the neighborhood statistics? Is it
an easy or difficult update to the software?
Thanks again,
Chris
//SIGNED//
Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 8, 2018 3:24 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
grid_stat on Neighborhood statistics
Chris,
It sounds like you understand how the fractional coverage field is
computed.
- Read the raw field... probability values between 0 and 1 in your
case.
- Apply the categorical threshold to that data to generate a field of
0's and 1's... >0.5 in your case.
- For each grid point, replace the value at that grid point with the
"fraction" of 1's in the neighborhood around that point... e.g. the
event occurs at 4 of the 9 grid points in a 3x3 square surrounding the
current point, so fractional coverage = 4/9.
- Compare the forecast fractional coverage values to the observed
fractional coverage values to compute FSS.
Yes, I do think it would be a good idea to talk to Craig about how he
configured MET for his work.
Thanks,
John
On Thu, Feb 8, 2018 at 2:09 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
>
> Hi John,
>
> Getting back to your initial response, I was able to run grid_stat
> with your changes in the attached configuration file to get FSS
output
> (as well as the other statistics for the neighborhood type results).
>
> However, I do have a question about the forecast categorical
threshold
> (cat_thresh is set to >=0.5) used in determining the neighborhood
> probabilities from the ensemble probabilities: What is the actual
> formula that MET uses to create the final field to calculate FSS?
Is
> it a strict neighborhood fractional coverage of ensemble
probabilities
> above 0.5 in some set radius of influence? Or is it some other
> operation or does the threshold represent something else? I know in
> Craig's paper, they were calling the last step a neighborhood
average
> of the ensemble probabilities to calculate NEP (since the event had
> already been defined at the grid point in determining the ensemble
probability [i.e. likelihood of exceeding
> 0.1" of precipitation]). Maybe, I should understand how Craig was
using
> MET for his paper.
>
> Thanks again,
> Chris
>
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, February 7, 2018 7:14 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help with
> grid_stat on Neighborhood statistics
>
> Hi Chris,
>
> I was thinking about what changes would be needed to compute FSS on
input
> probability data. We could add a “field” option to the “nbrhd”
section,
> just like we have in the “interp” section. It would be set to FCST,
OBS,
> or BOTH (default) and would control the logic as to which fields
should be
> passed through the logic for deriving fractional coverage fields.
In your
> case, you’d set it to OBS and we’d just use the raw input forecast
> probability values.
>
> Would that logic make sense to you?
>
> Thanks
> John
>
> On Wed, Feb 7, 2018 at 4:35 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Chris,
> >
> > In fact, SPC's 40-km radius probabilities were the motivation for
> > adding the shape = circle option to MET.
> >
> > For your first question, no, there is not currently a Gaussian
> > smoother option for the "interp" section in Grid-Stat. Currently,
the
> > only valid options there are MIN, MAX, MEDIAN, and UW_MEAN (for
> unweighted mean).
> > There is a distance-weighted mean option for point data where the
> > weight is 1/distance squared. But that doesn't work for Grid-Stat
> > because we'd end up dividing by 0.
> >
> > I can see how adding a gaussian smoother would be useful.
> >
> > Here's what we've done in the past with this SPC 40-km
probabilities:
> >
> > (1) Evaluate that field as a probability field (i.e. set "prob =
> > TRUE;" in the config file).
> > (2) Use the "interp" section to define a smoothing operation on
the
> > observations:
> > interp = {
> > field = OBS;
> > vld_thresh = 1.0;
> > shape = CIRCLE;
> > type = [
> > { method = MAX; width = 11; }
> > ];
> > }
> > Or set width to some circle diameter in grid units that works
out
> > to 40km in the real world.
> >
> > In this example, we post-process the observations to make them
> > comparable the event for which the probability was defined.
> >
> > But I think I understand what you'd like to do... interpret as the
> > probability value at each grid point as if it was a fractional
> > coverage value. So don't apply a threshold and neighborhood size
to
> > the forecast data at all. Just pass it directly to the routine
that
> computes FSS.
> >
> > Unfortunately no, I don't think there's a way of configuring Grid-
Stat
> > to do that in met-6.1. If you think this logic would be generally
> > useful, we could consider adding a flag to the config file to
enable
> that logic.
> >
> > Sorry I don't have better answers for you.
> >
> > John
> >
> >
> >
> > On Wed, Feb 7, 2018 at 4:14 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >>
> >> Hi John,
> >>
> >> Your suggestion for accomplishing #2 sounds promising. I need to
think
> >> the details through some. On a somewhat related note, is there
a
> Gaussian
> >> smoother flag for smoothing the probabilistic fields? I know
that was
> >> mentioned in Craig's paper and I have pursued that option in the
past
> >> (but just not using MET).
> >>
> >> I guess my basic question is can you apply a neighborhood width
that
> >> just includes the grid point maybe because you already have
> >> inherently a probabilistic field with spatial uncertainty? Would
that
> be a width of 1
> >> or does that also include the nearest grid point? A good
example of
> this
> >> would be a probabilistic forecast issued from the Storm
Prediction
> >> Center where the probabilities implicitly assume a 40-km radius
of
> >> influence (so there is no need to compute a fractional coverage
of the
> event).
> >>
> >> Thanks again,
> >> Chris
> >>
> >>
> >> //SIGNED//
> >>
> >> Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, February 7, 2018 4:49 PM
> >> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> christopher.melick at us.af.mil>
> >> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
> >> <matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16
> >> WS/WXN <robert.craig.2 at us.af.mil>
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> grid_stat on Neighborhood statistics
> >>
> >> Chris,
> >>
> >> (1) The default shape is a square since that's what MET
previously did.
> >> The circle is the *new* option.
> >>
> >> (2) For this one, the short answer is yes. FSS is computed in
MET by
> >> doing the following...
> >> - For each field separately (i.e. fcst and obs), apply the
> >> threshold
> >> (cat_thresh) to convert the input data to a 0/1 bitmap.
> >> - Apply the neighborhood width and shape to compute a
fractional
> >> coverage value for each grid point.
> >> - Compare the fcst and obs fractional coverage fields to
compute FSS.
> >>
> >> But there are some other options that you may find useful. The
"interp"
> >> section in the config file is used in Grid-Stat as a smoothing
> operation.
> >> After reading the raw input data, you can specify if/how to
smooth
> >> that data prior to computing stats. For example, since you
mentioned
> >> maximums, you could replace the value at each grid point with the
> >> maximum of the values in a 5x5 box surrounding that point:
> >>
> >> interp = {
> >> field = FCST;
> >> vld_thresh = 1.0;
> >> shape = SQUARE;
> >>
> >> type = [
> >> { method = MEAREST; width = 1; }, // i.e. no smoothing
> >> { method = MAX; width = 5; }, // i.e. max of 25 closest
points
> >> { method = UW_MEAN; width = 5; }, // i.e. mean of the 25
> >> closest points
> >> ];
> >> }
> >>
> >> Using the example above, you'll get 3 times the amount of output
from
> MET.
> >> The first smoother really is no smoothing at all. It would run
on
> >> the raw input data. The second smoother replaces the value at
each
> >> grid point with the maximum of the 25 surrounding points. The
third
> >> smoother replaces the value at each grid point with the mean of
those
> 25.
> >>
> >> I worked with Craig Schwartz when he was running MET for that
paper.
> >> My recollection is that he was able to use the configuration
options
> >> of Grid-Stat to accomplish the types of processing he was after.
But
> >> I don't recall all the details.
> >>
> >> If you explain exactly what you're trying to do, it may be
possible
> >> to configure Grid-Stat to accomplish that.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Feb 7, 2018 at 3:26 PM, christopher.melick at us.af.mil via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986 >
> >> >
> >> > Hi John,
> >> >
> >> > Your explanation and suggestions are very helpful and do
explain my
> >> > problems quite substantially. I will have to give it a try to
see
> >> > what happens.
> >> >
> >> > I do have a couple of other minor questions dealing with how
the
> >> > neighborhood verification is performed:
> >> > 1.) I noticed in the documentation that there is a flag for
> >> > defining the shape of the neighborhood area (i.e. circular or
> >> > square). What is the default if you don't specify?
> >> > 2.) In the literature, there are a couple of different ways to
> >> > handle generating neighborhood probabilities (e.g., Schwartz
and
> >> > Sobash 2017;
> >> > https://journals.ametsoc.org/doi/full/10.1175/MWR-D-16-0400.1).
I
> >> > was curious is the only way to get the Fractions Skill Score
(FSS)
> >> > output is to define a neighborhood in which MET calculates a
> >> > fractional coverage of the threshold event? I have previously
> >> > explored computing the neighborhood maximum value within a
radius of
> each grid point before calculating a probability.
> >> >
> >> > Thanks,
> >> > Chris
> >> >
> >> >
> >> > //SIGNED//
> >> >
> >> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> >> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> >> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, February 7, 2018 3:49 PM
> >> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> >> > christopher.melick at us.af.mil>
> >> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> >> matthew.dawson.8 at us.af.mil>;
> >> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> >> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #83986] Help
with
> >> > grid_stat on Neighborhood statistics
> >> >
> >> > Hi Chris,
> >> >
> >> > I see that you're having difficulty computing neighborhood
> >> > statistics in Grid-Stat.
> >> >
> >> > First, Grid-Stat definitely does have the ability to threshold
the
> >> StageIV
> >> > data on the fly. So pre-processing to 0's and 1's first is not
> needed.
> >> > Instead, in the "obs" dictionary of the config file, you'd just
set:
> >> > cat_thresh = [ >=2.54 ];
> >> >
> >> > Since StageIV precip is millimeters, >=2.54 mm is the same as
>=0.1
> >> inches.
> >> >
> >> > If for some reason you have data that's already 0's and 1's,
you'd
> >> > just use a different threshold in the config file:
> >> > cat_thresh = [ ==1 ];
> >> >
> >> > This tells MET that anywhere you see a 1 in the data, the event
is
> >> > occurring. Otherwise, it's not.
> >> >
> >> > Now on the to real issue. Looking at your config file, I see
that
> >> you're
> >> > processing probabilistic data. When "prob = TRUE", MET only
> >> > computes
> >> the
> >> > probabilistic line types (PCT, PSTD, PRC, and PJC). That's why
> >> > you're getting no output in the NBRCNT line type, which is the
one
> >> > that
> >> contains
> >> > Fractions Skill Score (FSS).
> >> >
> >> > In the attached config file, I've made a few changes:
> >> > - Set the obtype to indicate that you're verifying against
StageIV
> >> > - Defined 2 fcst.field entries.
> >> > - In the first we process QP010 as a probability field.
Here
> >> > cat_thresh is set to ==0.10 which is shorthand notation for
> >> > defining
> >> bins
> >> > for probabilistic verification from 0 to 1 of width 0.1.
> >> > - In the second we process it as a field of scalars.
Here
> >> > cat_thresh is set to >=0.5. So that's the probability
threshold
> >> > I'm
> >> using
> >> > for defining events for fractions skill score.
> >> > - Defined 2 corresponding obs.field entries to verify the
forecast
> >> data.
> >> > In both the verifying threshold is >=2.54. This assumes that
you
> >> > pass
> >> the
> >> > raw StageIV data as input.
> >> > - In the output_flag section, I turned off the line types that
> >> > don't apply.
> >> >
> >> > Lastly, while setting "level = [ "R19" ];" works as a way of
> >> > grabbing record number 19 from the input file, it's not
> >> > recommended. Instead,
> >> it'd
> >> > be better to figure out how to set the level string so that MET
> >> > always grabs the same record, regardless if it's in spot number
19
> >> > or 20 or any other location in the GRIB file.
> >> >
> >> > Hope this helps clarify. What other questions do you have?
> >> >
> >> > Thanks,
> >> >
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Wed, Feb 7, 2018 at 2:12 PM, christopher.melick at us.af.mil
via RT
> >> > < met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Feb 07 14:12:43 2018: Request 83986 was acted upon.
> >> > > Transaction: Ticket created by christopher.melick at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: Help with grid_stat on Neighborhood statistics
> >> > > Owner: Nobody
> >> > > Requestors: christopher.melick at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83986
> >> > > >
> >> > >
> >> > >
> >> > > Hi John,
> >> > >
> >> > > I am trying to run grid_stat in MET to calculate Neighborhood
> >> > > statistics for probability of precipitation from some
ensemble
> >> > > data with the observations
> >> > > coming from Stage IV analyses. Unfortunately, I am getting
no
> output
> >> > > (just
> >> > > headers) in many of the ASCII files from the program when I
run it.
> >> > Could
> >> > > you help me figure out what I am doing wrong? I have
provided
> >> > > the command line specifics I use as well as the output log
file
> >> > > that is
> >> > created when
> >> > > running that command. I also have attached the
configuration file.
> >> I
> >> > am
> >> > > having trouble using ftp to send the raw data from any of the
> clients
> >> > > available to us. I believe this restriction occurred
recently
> since
> >> Bob
> >> > > Craig had mentioned that he had been able to connect
externally
> >> > > in the past.
> >> > >
> >> > > I should mention that I have pre-processed the Stage IV
analyses
> >> > > already to be either 0's or 1's for the threshold (>=0.1")
since
> >> > > I was unclear as to whether MET was able to do that on the
fly
> >> > > when running
> >> > grid_stat.
> >> > >
> >> > > Thanks,
> >> > > Chris
> >> > >
> >> > > Python command:
> >> > > Metcall = [metpath + 'grid_stat', 'grib.2017091422.0006',
> >> > > 'precip/post-processed-obs.nc', \
> >> > > 'GridStatConfig_POP', '-outdir', 'tmp', '-v',
'5]
> >> > > Call(metcall)
> >> > >
> >> > > //SIGNED//
> >> > >
> >> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> >> > > Verification Exploitation Team 16th Weather Squadron (16
WS/WXEV)
> >> > > 557th Weather Wing, Offutt AFB, NE DSN 271-3693; Comm (402)
> >> > > 294-3693
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >>
> >>
> >
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list