[Met_help] [rt.rap.ucar.edu #77759] History for question about grid-stat (MET V4.1) (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Thu Sep 1 16:39:43 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
What is the averaging method used to upscale the forecast and observed
fields to successively coarser scales when neighborhood methods are used as
specified in the config file? Where is this averaging method specified in
the config file?
I specify nbrhd widths of 1,3,5,7,9 grid points in my config file. I'm using
the CSI and FBIAS scores for analysis.
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Aug 31 14:38:06 2016
For clarification and reference, I have attached my grid-stat
configuration file.
R/
John
________________________________________
From: met_help at ucar.edu via RT [met_help at ucar.edu]
Sent: Wednesday, August 31, 2016 2:00 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply: question
about grid-stat (MET V4.1) (UNCLASSIFIED)
Greetings,
This message has been automatically generated in response to the
creation of a trouble ticket regarding:
"question about grid-stat (MET V4.1) (UNCLASSIFIED)",
a summary of which appears below.
There is no need to reply to this message right now. Your ticket has
been
assigned an ID of [rt.rap.ucar.edu #77759].
Please include the string:
[rt.rap.ucar.edu #77759]
in the subject line of all future correspondence about this issue. To
do so,
you may reply to this message.
Thank you,
met_help at ucar.edu
-------------------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
What is the averaging method used to upscale the forecast and observed
fields to successively coarser scales when neighborhood methods are
used as
specified in the config file? Where is this averaging method specified
in
the config file?
I specify nbrhd widths of 1,3,5,7,9 grid points in my config file. I'm
using
the CSI and FBIAS scores for analysis.
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Sep 01 10:14:31 2016
Hello John,
I see you have some questions about the "width" settings specified in
the
Grid-Stat configuration file. Based on your question, I think it'd
best
for me to give you an overview of the logic involved. There are two
distinct methods in Grid-Stat that I suspect are causing confusion.
Since
your question (and your config file) are about the "nbrhd" section,
let me
start there...
(1) Grid-Stat can be used to apply neighborhood verification methods,
most
notably the Fractions Skill Score (FSS). These computations are
specified
by the "nbrhd" section of the config file and its output is found in
the
NBRCNT, NBRCTC, and NBRCTS line types. Here's how the logic works...
- For each forecast variable being verified and each categorical
threshold specified (i.e. the "cat_thresh" setting), apply that
threshold
to define a 0/1 bitmap field.
- For each "width", specified in the "nbrhd" section, use that
width to
convert the 0/1 bitmap field to a fractional coverage field. For
example,
for width = 9, look at the 9 points surrounding each grid point and
compute
the ratio of them that are 1. The resulting fractional coverage field
has
a range of [0, 1] and represents the ratio of nearby points where the
event
occurred.
- Apply the same logic to the observation field (apply threshold and
use
"width" to derive fractional coverage field).
- The statistics in the NBRCNT line type are computed by comparing
the
forecast and observed fractional coverage fields directly.
- The "cov_thresh" entries can then be applied to both the
fractional
coverage fields and contingency table counts (NBRCTC) and statistics
(NBRCTS) can be computed.
- NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
wouldn't
suggest using them.
- Apply this logic for all combinations of variables, categorical
thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
(2) As described above, the neighborhood output is computed AFTER the
categorical threshold has been applied. The other method in Grid-Stat
is
applied to the RAW data directly (so no thresholding). Upscaling
(i.e.
smoothing) in Grid-Stat is controlled by the "interp" section of the
config
file. The process of drawing a box around each grid point (i.e. width
of 3
means a 3x3 box) is the same as the neighborhood methods, but it's
simply
taking the arithmetic mean of the values in that box. So the value at
each
grid point is replaced by the mean of the values in the neighborhood.
When
applying upscaling, the raw data is first smoothed and then the
traditional
verification statistics are computed (i.e. SL1L2, CNT, CTC, CTS, and
so
on). The smoothed data is NOT used as input to the neighborhood
method
described above... neighborhood methods always start with the raw
data.
Now which field(s) actually get smoothed? That's based on the
"interp.field" setting... FCST for the forecast only, OBS for the
observation only, or BOTH for both of them.
Why would you want to try upscaling? It's often the case that
smoother,
less realistic looking forecasts score better than higher resolution,
more
realistic looking forecasts when assessed with traditional statistics
(like
RMSE). Upscaling is a way to quantify how your verification scores
would
change for a smoother forecast.
Lastly, Grid-Stat should enforce that all widths (in both the nbrhd
and
interp) sections be odd. That keeps the nxn box centered on each grid
point.
Hope that helps clarify. Just let me know if you have more questions.
Thanks,
John
On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> For clarification and reference, I have attached my grid-stat
> configuration file.
>
> R/
> John
> ________________________________________
> From: met_help at ucar.edu via RT [met_help at ucar.edu]
> Sent: Wednesday, August 31, 2016 2:00 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question
> about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "question about grid-stat (MET V4.1) (UNCLASSIFIED)",
> a summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #77759].
>
> Please include the string:
>
> [rt.rap.ucar.edu #77759]
>
> in the subject line of all future correspondence about this issue.
To do
> so,
> you may reply to this message.
>
> Thank you,
> met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> CLASSIFICATION: UNCLASSIFIED
>
> What is the averaging method used to upscale the forecast and
observed
> fields to successively coarser scales when neighborhood methods are
used as
> specified in the config file? Where is this averaging method
specified in
> the config file?
>
> I specify nbrhd widths of 1,3,5,7,9 grid points in my config file.
I'm
> using
> the CSI and FBIAS scores for analysis.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 01 14:34:17 2016
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for your response. Comes just in time as I'm finalizing a tech
report
on results from running Grid-Stat to generate neighborhood
verification
statistics to see how they could be applied to continuous variable
fields. I
used the nbrcts output which contains the CSI and FBIAS scores which
are of
interest to us. Based on seeing the logic below, I see that those
scores were
computed using the fractional coverage method instead of upscaling.
Since CSI
and FBIAS are contingency table statistics, I need to understand a
little more
about the logic for determining hits, misses, false alarms and correct
rejections to fill the 2x2 table such as appears in the User's Guide.
See the
attached diagram to confirm my understanding of the fractional
coverage method
you describe below. If this is correct, then Grid-Stat is taking these
results
(fractions) and developing the contingency table statistics which are
used to
calculate CSI and FBIAS. Can you describe this logic?
The reason I was thinking upscaling was because I read the journal
article by
Ebert (2008) which describes several fuzzy verification methods and
upscaling
was one of them. I saw that the CSI and the FBIAS scores were computed
for the
upscaling method, so I thought I'd better confirm how Grid-Stat is
computing
them rather than just assuming that it was computed the same way as
Ebert did.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 01, 2016 10:15 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question
about grid-stat (MET V4.1) (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Hello John,
I see you have some questions about the "width" settings specified in
the
Grid-Stat configuration file. Based on your question, I think it'd
best for
me to give you an overview of the logic involved. There are two
distinct
methods in Grid-Stat that I suspect are causing confusion. Since your
question (and your config file) are about the "nbrhd" section, let me
start
there...
(1) Grid-Stat can be used to apply neighborhood verification methods,
most
notably the Fractions Skill Score (FSS). These computations are
specified by
the "nbrhd" section of the config file and its output is found in the
NBRCNT,
NBRCTC, and NBRCTS line types. Here's how the logic works...
- For each forecast variable being verified and each categorical
threshold
specified (i.e. the "cat_thresh" setting), apply that threshold to
define a
0/1 bitmap field.
- For each "width", specified in the "nbrhd" section, use that
width to
convert the 0/1 bitmap field to a fractional coverage field. For
example, for
width = 9, look at the 9 points surrounding each grid point and
compute the
ratio of them that are 1. The resulting fractional coverage field has
a range
of [0, 1] and represents the ratio of nearby points where the event
occurred.
- Apply the same logic to the observation field (apply threshold and
use
"width" to derive fractional coverage field).
- The statistics in the NBRCNT line type are computed by comparing
the
forecast and observed fractional coverage fields directly.
- The "cov_thresh" entries can then be applied to both the
fractional
coverage fields and contingency table counts (NBRCTC) and statistics
(NBRCTS) can be computed.
- NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
wouldn't
suggest using them.
- Apply this logic for all combinations of variables, categorical
thresholds
(cat_thresh), and neighborhood widths (nbrhd.width).
(2) As described above, the neighborhood output is computed AFTER the
categorical threshold has been applied. The other method in Grid-Stat
is
applied to the RAW data directly (so no thresholding). Upscaling
(i.e.
smoothing) in Grid-Stat is controlled by the "interp" section of the
config
file. The process of drawing a box around each grid point (i.e. width
of 3
means a 3x3 box) is the same as the neighborhood methods, but it's
simply
taking the arithmetic mean of the values in that box. So the value at
each
grid point is replaced by the mean of the values in the neighborhood.
When
applying upscaling, the raw data is first smoothed and then the
traditional
verification statistics are computed (i.e. SL1L2, CNT, CTC, CTS, and
so on).
The smoothed data is NOT used as input to the neighborhood method
described
above... neighborhood methods always start with the raw data.
Now which field(s) actually get smoothed? That's based on the
"interp.field"
setting... FCST for the forecast only, OBS for the observation only,
or BOTH
for both of them.
Why would you want to try upscaling? It's often the case that
smoother, less
realistic looking forecasts score better than higher resolution, more
realistic looking forecasts when assessed with traditional statistics
(like
RMSE). Upscaling is a way to quantify how your verification scores
would
change for a smoother forecast.
Lastly, Grid-Stat should enforce that all widths (in both the nbrhd
and
interp) sections be odd. That keeps the nxn box centered on each grid
point.
Hope that helps clarify. Just let me know if you have more questions.
Thanks,
John
On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> For clarification and reference, I have attached my grid-stat
> configuration file.
>
> R/
> John
> ________________________________________
> From: met_help at ucar.edu via RT [met_help at ucar.edu]
> Sent: Wednesday, August 31, 2016 2:00 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question
> about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has
> been assigned an ID of [rt.rap.ucar.edu #77759].
>
> Please include the string:
>
> [rt.rap.ucar.edu #77759]
>
> in the subject line of all future correspondence about this issue.
To
> do so, you may reply to this message.
>
> Thank you,
> met_help at ucar.edu
>
>
----------------------------------------------------------------------
> ---
> CLASSIFICATION: UNCLASSIFIED
>
> What is the averaging method used to upscale the forecast and
observed
> fields to successively coarser scales when neighborhood methods are
> used as specified in the config file? Where is this averaging method
> specified in the config file?
>
> I specify nbrhd widths of 1,3,5,7,9 grid points in my config file.
I'm
> using the CSI and FBIAS scores for analysis.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 01 14:47:44 2016
CLASSIFICATION: UNCLASSIFIED
John -
I sent my previous email a little too soon and after some thought and
consultation with the User's Guide regards "cov_thresh", I think I
understand how the contingency table statistics are developed. Correct
me, if I'm wrong. The fractional coverage "score" for the neighborhood
(i.e. a number like 0.24 from the graphic I sent you) is compared with
the value I set for "cov_thresh" and if it is GE the value of
"cov_thresh" then its considered a hit and if it is LT, the it’s a
miss.
What say you?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 01, 2016 10:15 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question about grid-stat (MET V4.1) (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hello John,
I see you have some questions about the "width" settings specified in
the Grid-Stat configuration file. Based on your question, I think
it'd best for me to give you an overview of the logic involved. There
are two distinct methods in Grid-Stat that I suspect are causing
confusion. Since your question (and your config file) are about the
"nbrhd" section, let me start there...
(1) Grid-Stat can be used to apply neighborhood verification methods,
most notably the Fractions Skill Score (FSS). These computations are
specified by the "nbrhd" section of the config file and its output is
found in the NBRCNT, NBRCTC, and NBRCTS line types. Here's how the
logic works...
- For each forecast variable being verified and each categorical
threshold specified (i.e. the "cat_thresh" setting), apply that
threshold to define a 0/1 bitmap field.
- For each "width", specified in the "nbrhd" section, use that
width to convert the 0/1 bitmap field to a fractional coverage field.
For example, for width = 9, look at the 9 points surrounding each grid
point and compute the ratio of them that are 1. The resulting
fractional coverage field has a range of [0, 1] and represents the
ratio of nearby points where the event occurred.
- Apply the same logic to the observation field (apply threshold and
use "width" to derive fractional coverage field).
- The statistics in the NBRCNT line type are computed by comparing
the forecast and observed fractional coverage fields directly.
- The "cov_thresh" entries can then be applied to both the
fractional coverage fields and contingency table counts (NBRCTC) and
statistics
(NBRCTS) can be computed.
- NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
wouldn't suggest using them.
- Apply this logic for all combinations of variables, categorical
thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
(2) As described above, the neighborhood output is computed AFTER the
categorical threshold has been applied. The other method in Grid-Stat
is applied to the RAW data directly (so no thresholding). Upscaling
(i.e.
smoothing) in Grid-Stat is controlled by the "interp" section of the
config file. The process of drawing a box around each grid point
(i.e. width of 3 means a 3x3 box) is the same as the neighborhood
methods, but it's simply taking the arithmetic mean of the values in
that box. So the value at each grid point is replaced by the mean of
the values in the neighborhood. When applying upscaling, the raw data
is first smoothed and then the traditional verification statistics are
computed (i.e. SL1L2, CNT, CTC, CTS, and so on). The smoothed data is
NOT used as input to the neighborhood method described above...
neighborhood methods always start with the raw data.
Now which field(s) actually get smoothed? That's based on the
"interp.field" setting... FCST for the forecast only, OBS for the
observation only, or BOTH for both of them.
Why would you want to try upscaling? It's often the case that
smoother, less realistic looking forecasts score better than higher
resolution, more realistic looking forecasts when assessed with
traditional statistics (like RMSE). Upscaling is a way to quantify
how your verification scores would change for a smoother forecast.
Lastly, Grid-Stat should enforce that all widths (in both the nbrhd
and
interp) sections be odd. That keeps the nxn box centered on each grid
point.
Hope that helps clarify. Just let me know if you have more questions.
Thanks,
John
On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> For clarification and reference, I have attached my grid-stat
> configuration file.
>
> R/
> John
> ________________________________________
> From: met_help at ucar.edu via RT [met_help at ucar.edu]
> Sent: Wednesday, August 31, 2016 2:00 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question
> about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has
> been assigned an ID of [rt.rap.ucar.edu #77759].
>
> Please include the string:
>
> [rt.rap.ucar.edu #77759]
>
> in the subject line of all future correspondence about this issue.
To
> do so, you may reply to this message.
>
> Thank you,
> met_help at ucar.edu
>
>
----------------------------------------------------------------------
> ---
> CLASSIFICATION: UNCLASSIFIED
>
> What is the averaging method used to upscale the forecast and
observed
> fields to successively coarser scales when neighborhood methods are
> used as specified in the config file? Where is this averaging method
> specified in the config file?
>
> I specify nbrhd widths of 1,3,5,7,9 grid points in my config file.
I'm
> using the CSI and FBIAS scores for analysis.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Sep 01 14:57:08 2016
John,
Yes, you are correct. The "cov_thresh" setting is used to threshold
the
fractional coverage fields generated during the neighborhood
verification
process. Any points meeting that threshold criteria are events while
those
not meeting it are non-events. If the event occurs at a grid point in
both
the forecast and observed data, it's a hit. No event in either is a
correct negative... and so on for false alarms and misses.
I'm always careful about saying "meeting the threshold criteria"
instead of
"greater than the threshold" because you could just as easily define
>0.5
or <=0.5. The resulting statistics should be exactly "opposite".
John
On Thu, Sep 1, 2016 at 2:47 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I sent my previous email a little too soon and after some thought
and
> consultation with the User's Guide regards "cov_thresh", I think I
> understand how the contingency table statistics are developed.
Correct me,
> if I'm wrong. The fractional coverage "score" for the neighborhood
(i.e. a
> number like 0.24 from the graphic I sent you) is compared with the
value I
> set for "cov_thresh" and if it is GE the value of "cov_thresh" then
its
> considered a hit and if it is LT, the it’s a miss.
>
> What say you?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 01, 2016 10:15 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> question about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hello John,
>
> I see you have some questions about the "width" settings specified
in the
> Grid-Stat configuration file. Based on your question, I think it'd
best
> for me to give you an overview of the logic involved. There are two
> distinct methods in Grid-Stat that I suspect are causing confusion.
Since
> your question (and your config file) are about the "nbrhd" section,
let me
> start there...
>
> (1) Grid-Stat can be used to apply neighborhood verification
methods, most
> notably the Fractions Skill Score (FSS). These computations are
specified
> by the "nbrhd" section of the config file and its output is found in
the
> NBRCNT, NBRCTC, and NBRCTS line types. Here's how the logic
works...
> - For each forecast variable being verified and each categorical
> threshold specified (i.e. the "cat_thresh" setting), apply that
threshold
> to define a 0/1 bitmap field.
> - For each "width", specified in the "nbrhd" section, use that
width to
> convert the 0/1 bitmap field to a fractional coverage field. For
example,
> for width = 9, look at the 9 points surrounding each grid point and
compute
> the ratio of them that are 1. The resulting fractional coverage
field has
> a range of [0, 1] and represents the ratio of nearby points where
the event
> occurred.
> - Apply the same logic to the observation field (apply threshold
and use
> "width" to derive fractional coverage field).
> - The statistics in the NBRCNT line type are computed by comparing
the
> forecast and observed fractional coverage fields directly.
> - The "cov_thresh" entries can then be applied to both the
fractional
> coverage fields and contingency table counts (NBRCTC) and statistics
> (NBRCTS) can be computed.
> - NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
wouldn't
> suggest using them.
> - Apply this logic for all combinations of variables, categorical
> thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
>
> (2) As described above, the neighborhood output is computed AFTER
the
> categorical threshold has been applied. The other method in Grid-
Stat is
> applied to the RAW data directly (so no thresholding). Upscaling
(i.e.
> smoothing) in Grid-Stat is controlled by the "interp" section of the
> config file. The process of drawing a box around each grid point
(i.e.
> width of 3 means a 3x3 box) is the same as the neighborhood methods,
but
> it's simply taking the arithmetic mean of the values in that box.
So the
> value at each grid point is replaced by the mean of the values in
the
> neighborhood. When applying upscaling, the raw data is first
smoothed and
> then the traditional verification statistics are computed (i.e.
SL1L2, CNT,
> CTC, CTS, and so on). The smoothed data is NOT used as input to the
> neighborhood method described above... neighborhood methods always
start
> with the raw data.
> Now which field(s) actually get smoothed? That's based on the
> "interp.field" setting... FCST for the forecast only, OBS for the
> observation only, or BOTH for both of them.
>
> Why would you want to try upscaling? It's often the case that
smoother,
> less realistic looking forecasts score better than higher
resolution, more
> realistic looking forecasts when assessed with traditional
statistics (like
> RMSE). Upscaling is a way to quantify how your verification scores
would
> change for a smoother forecast.
>
> Lastly, Grid-Stat should enforce that all widths (in both the nbrhd
and
> interp) sections be odd. That keeps the nxn box centered on each
grid
> point.
>
> Hope that helps clarify. Just let me know if you have more
questions.
>
> Thanks,
> John
>
> On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
> >
> > For clarification and reference, I have attached my grid-stat
> > configuration file.
> >
> > R/
> > John
> > ________________________________________
> > From: met_help at ucar.edu via RT [met_help at ucar.edu]
> > Sent: Wednesday, August 31, 2016 2:00 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question
> > about grid-stat (MET V4.1) (UNCLASSIFIED)
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
> > creation of a trouble ticket regarding:
> > "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> > summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
has
> > been assigned an ID of [rt.rap.ucar.edu #77759].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #77759]
> >
> > in the subject line of all future correspondence about this issue.
To
> > do so, you may reply to this message.
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
----------------------------------------------------------------------
> > ---
> > CLASSIFICATION: UNCLASSIFIED
> >
> > What is the averaging method used to upscale the forecast and
observed
> > fields to successively coarser scales when neighborhood methods
are
> > used as specified in the config file? Where is this averaging
method
> > specified in the config file?
> >
> > I specify nbrhd widths of 1,3,5,7,9 grid points in my config file.
I'm
> > using the CSI and FBIAS scores for analysis.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 01 16:03:51 2016
CLASSIFICATION: UNCLASSIFIED
John - Been good talking to you. Thanks for the timely confirmation of
the logic. From my reading in Ebert's paper, I think the method you
are describing below is called "Minimum coverage"
which seems to be more aligned with the concept of having a minimum
value for neighborhood event coverage to decide if the threshold
criterion was met. This matches up closely with the User's Guide
description of the functionality of "cov_thresh" too. In your opinion,
does this method description from Ebert ("minimum coverage") matchup
with the logic used in Grid-Stat?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 01, 2016 2:57 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question about grid-stat (MET V4.1) (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Yes, you are correct. The "cov_thresh" setting is used to threshold
the fractional coverage fields generated during the neighborhood
verification process. Any points meeting that threshold criteria are
events while those not meeting it are non-events. If the event occurs
at a grid point in both the forecast and observed data, it's a hit.
No event in either is a correct negative... and so on for false alarms
and misses.
I'm always careful about saying "meeting the threshold criteria"
instead of "greater than the threshold" because you could just as
easily define >0.5 or <=0.5. The resulting statistics should be
exactly "opposite".
John
On Thu, Sep 1, 2016 at 2:47 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I sent my previous email a little too soon and after some thought
and
> consultation with the User's Guide regards "cov_thresh", I think I
> understand how the contingency table statistics are developed.
Correct
> me, if I'm wrong. The fractional coverage "score" for the
neighborhood
> (i.e. a number like 0.24 from the graphic I sent you) is compared
with
> the value I set for "cov_thresh" and if it is GE the value of
> "cov_thresh" then its considered a hit and if it is LT, the it’s a
miss.
>
> What say you?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, September 01, 2016 10:15 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> question about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hello John,
>
> I see you have some questions about the "width" settings specified
in
> the Grid-Stat configuration file. Based on your question, I think
> it'd best for me to give you an overview of the logic involved.
There
> are two distinct methods in Grid-Stat that I suspect are causing
> confusion. Since your question (and your config file) are about the
> "nbrhd" section, let me start there...
>
> (1) Grid-Stat can be used to apply neighborhood verification
methods,
> most notably the Fractions Skill Score (FSS). These computations
are
> specified by the "nbrhd" section of the config file and its output
is
> found in the NBRCNT, NBRCTC, and NBRCTS line types. Here's how the
logic works...
> - For each forecast variable being verified and each categorical
> threshold specified (i.e. the "cat_thresh" setting), apply that
> threshold to define a 0/1 bitmap field.
> - For each "width", specified in the "nbrhd" section, use that
> width to convert the 0/1 bitmap field to a fractional coverage
field.
> For example, for width = 9, look at the 9 points surrounding each
grid
> point and compute the ratio of them that are 1. The resulting
> fractional coverage field has a range of [0, 1] and represents the
> ratio of nearby points where the event occurred.
> - Apply the same logic to the observation field (apply threshold
and
> use "width" to derive fractional coverage field).
> - The statistics in the NBRCNT line type are computed by comparing
> the forecast and observed fractional coverage fields directly.
> - The "cov_thresh" entries can then be applied to both the
> fractional coverage fields and contingency table counts (NBRCTC) and
> statistics
> (NBRCTS) can be computed.
> - NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
> wouldn't suggest using them.
> - Apply this logic for all combinations of variables, categorical
> thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
>
> (2) As described above, the neighborhood output is computed AFTER
the
> categorical threshold has been applied. The other method in Grid-
Stat
> is applied to the RAW data directly (so no thresholding). Upscaling
(i.e.
> smoothing) in Grid-Stat is controlled by the "interp" section of the
> config file. The process of drawing a box around each grid point
(i.e.
> width of 3 means a 3x3 box) is the same as the neighborhood methods,
> but it's simply taking the arithmetic mean of the values in that
box.
> So the value at each grid point is replaced by the mean of the
values
> in the neighborhood. When applying upscaling, the raw data is first
> smoothed and then the traditional verification statistics are
computed
> (i.e. SL1L2, CNT, CTC, CTS, and so on). The smoothed data is NOT
used
> as input to the neighborhood method described above... neighborhood
> methods always start with the raw data.
> Now which field(s) actually get smoothed? That's based on the
> "interp.field" setting... FCST for the forecast only, OBS for the
> observation only, or BOTH for both of them.
>
> Why would you want to try upscaling? It's often the case that
> smoother, less realistic looking forecasts score better than higher
> resolution, more realistic looking forecasts when assessed with
> traditional statistics (like RMSE). Upscaling is a way to quantify
> how your verification scores would change for a smoother forecast.
>
> Lastly, Grid-Stat should enforce that all widths (in both the nbrhd
> and
> interp) sections be odd. That keeps the nxn box centered on each
grid
> point.
>
> Hope that helps clarify. Just let me know if you have more
questions.
>
> Thanks,
> John
>
> On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77
> > 759 >
> >
> > For clarification and reference, I have attached my grid-stat
> > configuration file.
> >
> > R/
> > John
> > ________________________________________
> > From: met_help at ucar.edu via RT [met_help at ucar.edu]
> > Sent: Wednesday, August 31, 2016 2:00 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> > question about grid-stat (MET V4.1) (UNCLASSIFIED)
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
> > creation of a trouble ticket regarding:
> > "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> > summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
> > has been assigned an ID of [rt.rap.ucar.edu #77759].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #77759]
> >
> > in the subject line of all future correspondence about this issue.
> > To do so, you may reply to this message.
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
--------------------------------------------------------------------
> > --
> > ---
> > CLASSIFICATION: UNCLASSIFIED
> >
> > What is the averaging method used to upscale the forecast and
> > observed fields to successively coarser scales when neighborhood
> > methods are used as specified in the config file? Where is this
> > averaging method specified in the config file?
> >
> > I specify nbrhd widths of 1,3,5,7,9 grid points in my config file.
> > I'm using the CSI and FBIAS scores for analysis.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Sep 01 16:29:03 2016
Sorry John, I haven't read the paper well enough to comment on that.
If
you'd like, I could reassign this ticket to a statistician to get an
answer
for you on that. Just let me know.
Thanks,
John
On Thu, Sep 1, 2016 at 4:03 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John - Been good talking to you. Thanks for the timely confirmation
of the
> logic. From my reading in Ebert's paper, I think the method you are
> describing below is called "Minimum coverage"
> which seems to be more aligned with the concept of having a minimum
value
> for neighborhood event coverage to decide if the threshold criterion
was
> met. This matches up closely with the User's Guide description of
the
> functionality of "cov_thresh" too. In your opinion, does this method
> description from Ebert ("minimum coverage") matchup with the logic
used in
> Grid-Stat?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 01, 2016 2:57 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> question about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, you are correct. The "cov_thresh" setting is used to threshold
the
> fractional coverage fields generated during the neighborhood
verification
> process. Any points meeting that threshold criteria are events
while those
> not meeting it are non-events. If the event occurs at a grid point
in both
> the forecast and observed data, it's a hit. No event in either is a
> correct negative... and so on for false alarms and misses.
>
> I'm always careful about saying "meeting the threshold criteria"
instead
> of "greater than the threshold" because you could just as easily
define
> >0.5 or <=0.5. The resulting statistics should be exactly
"opposite".
>
> John
>
> On Thu, Sep 1, 2016 at 2:47 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I sent my previous email a little too soon and after some thought
and
> > consultation with the User's Guide regards "cov_thresh", I think I
> > understand how the contingency table statistics are developed.
Correct
> > me, if I'm wrong. The fractional coverage "score" for the
neighborhood
> > (i.e. a number like 0.24 from the graphic I sent you) is compared
with
> > the value I set for "cov_thresh" and if it is GE the value of
> > "cov_thresh" then its considered a hit and if it is LT, the it’s a
miss.
> >
> > What say you?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, September 01, 2016 10:15 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> > question about grid-stat (MET V4.1) (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see you have some questions about the "width" settings specified
in
> > the Grid-Stat configuration file. Based on your question, I think
> > it'd best for me to give you an overview of the logic involved.
There
> > are two distinct methods in Grid-Stat that I suspect are causing
> > confusion. Since your question (and your config file) are about
the
> > "nbrhd" section, let me start there...
> >
> > (1) Grid-Stat can be used to apply neighborhood verification
methods,
> > most notably the Fractions Skill Score (FSS). These computations
are
> > specified by the "nbrhd" section of the config file and its output
is
> > found in the NBRCNT, NBRCTC, and NBRCTS line types. Here's how
the
> logic works...
> > - For each forecast variable being verified and each
categorical
> > threshold specified (i.e. the "cat_thresh" setting), apply that
> > threshold to define a 0/1 bitmap field.
> > - For each "width", specified in the "nbrhd" section, use that
> > width to convert the 0/1 bitmap field to a fractional coverage
field.
> > For example, for width = 9, look at the 9 points surrounding each
grid
> > point and compute the ratio of them that are 1. The resulting
> > fractional coverage field has a range of [0, 1] and represents the
> > ratio of nearby points where the event occurred.
> > - Apply the same logic to the observation field (apply threshold
and
> > use "width" to derive fractional coverage field).
> > - The statistics in the NBRCNT line type are computed by
comparing
> > the forecast and observed fractional coverage fields directly.
> > - The "cov_thresh" entries can then be applied to both the
> > fractional coverage fields and contingency table counts (NBRCTC)
and
> > statistics
> > (NBRCTS) can be computed.
> > - NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
> > wouldn't suggest using them.
> > - Apply this logic for all combinations of variables,
categorical
> > thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
> >
> > (2) As described above, the neighborhood output is computed AFTER
the
> > categorical threshold has been applied. The other method in Grid-
Stat
> > is applied to the RAW data directly (so no thresholding).
Upscaling
> (i.e.
> > smoothing) in Grid-Stat is controlled by the "interp" section of
the
> > config file. The process of drawing a box around each grid point
(i.e.
> > width of 3 means a 3x3 box) is the same as the neighborhood
methods,
> > but it's simply taking the arithmetic mean of the values in that
box.
> > So the value at each grid point is replaced by the mean of the
values
> > in the neighborhood. When applying upscaling, the raw data is
first
> > smoothed and then the traditional verification statistics are
computed
> > (i.e. SL1L2, CNT, CTC, CTS, and so on). The smoothed data is NOT
used
> > as input to the neighborhood method described above...
neighborhood
> > methods always start with the raw data.
> > Now which field(s) actually get smoothed? That's based on the
> > "interp.field" setting... FCST for the forecast only, OBS for the
> > observation only, or BOTH for both of them.
> >
> > Why would you want to try upscaling? It's often the case that
> > smoother, less realistic looking forecasts score better than
higher
> > resolution, more realistic looking forecasts when assessed with
> > traditional statistics (like RMSE). Upscaling is a way to
quantify
> > how your verification scores would change for a smoother forecast.
> >
> > Lastly, Grid-Stat should enforce that all widths (in both the
nbrhd
> > and
> > interp) sections be odd. That keeps the nxn box centered on each
grid
> > point.
> >
> > Hope that helps clarify. Just let me know if you have more
questions.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77
> > > 759 >
> > >
> > > For clarification and reference, I have attached my grid-stat
> > > configuration file.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: met_help at ucar.edu via RT [met_help at ucar.edu]
> > > Sent: Wednesday, August 31, 2016 2:00 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> > > question about grid-stat (MET V4.1) (UNCLASSIFIED)
> > >
> > > Greetings,
> > >
> > > This message has been automatically generated in response to the
> > > creation of a trouble ticket regarding:
> > > "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> > > summary of which appears below.
> > >
> > > There is no need to reply to this message right now. Your
ticket
> > > has been assigned an ID of [rt.rap.ucar.edu #77759].
> > >
> > > Please include the string:
> > >
> > > [rt.rap.ucar.edu #77759]
> > >
> > > in the subject line of all future correspondence about this
issue.
> > > To do so, you may reply to this message.
> > >
> > > Thank you,
> > > met_help at ucar.edu
> > >
> > >
--------------------------------------------------------------------
> > > --
> > > ---
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > What is the averaging method used to upscale the forecast and
> > > observed fields to successively coarser scales when neighborhood
> > > methods are used as specified in the config file? Where is this
> > > averaging method specified in the config file?
> > >
> > > I specify nbrhd widths of 1,3,5,7,9 grid points in my config
file.
> > > I'm using the CSI and FBIAS scores for analysis.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr John W. Raby, Meteorologist
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about grid-stat (MET V4.1) (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 01 16:32:14 2016
CLASSIFICATION: UNCLASSIFIED
John -
I have enough information on the logic for now. Thanks for your help.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 01, 2016 4:29 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
question about grid-stat (MET V4.1) (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Sorry John, I haven't read the paper well enough to comment on that.
If you'd like, I could reassign this ticket to a statistician to get
an answer for you on that. Just let me know.
Thanks,
John
On Thu, Sep 1, 2016 at 4:03 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77759 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John - Been good talking to you. Thanks for the timely confirmation
of
> the logic. From my reading in Ebert's paper, I think the method you
> are describing below is called "Minimum coverage"
> which seems to be more aligned with the concept of having a minimum
> value for neighborhood event coverage to decide if the threshold
> criterion was met. This matches up closely with the User's Guide
> description of the functionality of "cov_thresh" too. In your
opinion,
> does this method description from Ebert ("minimum coverage") matchup
> with the logic used in Grid-Stat?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, September 01, 2016 2:57 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> question about grid-stat (MET V4.1) (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Yes, you are correct. The "cov_thresh" setting is used to threshold
> the fractional coverage fields generated during the neighborhood
> verification process. Any points meeting that threshold criteria
are
> events while those not meeting it are non-events. If the event
occurs
> at a grid point in both the forecast and observed data, it's a hit.
> No event in either is a correct negative... and so on for false
alarms and misses.
>
> I'm always careful about saying "meeting the threshold criteria"
> instead of "greater than the threshold" because you could just as
> easily define
> >0.5 or <=0.5. The resulting statistics should be exactly
"opposite".
>
> John
>
> On Thu, Sep 1, 2016 at 2:47 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77
> > 759 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I sent my previous email a little too soon and after some thought
> > and consultation with the User's Guide regards "cov_thresh", I
think
> > I understand how the contingency table statistics are developed.
> > Correct me, if I'm wrong. The fractional coverage "score" for the
> > neighborhood (i.e. a number like 0.24 from the graphic I sent you)
> > is compared with the value I set for "cov_thresh" and if it is GE
> > the value of "cov_thresh" then its considered a hit and if it is
LT, the it’s a miss.
> >
> > What say you?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, September 01, 2016 10:15 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> > question about grid-stat (MET V4.1) (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see you have some questions about the "width" settings specified
> > in the Grid-Stat configuration file. Based on your question, I
> > think it'd best for me to give you an overview of the logic
> > involved. There are two distinct methods in Grid-Stat that I
> > suspect are causing confusion. Since your question (and your
config
> > file) are about the "nbrhd" section, let me start there...
> >
> > (1) Grid-Stat can be used to apply neighborhood verification
> > methods, most notably the Fractions Skill Score (FSS). These
> > computations are specified by the "nbrhd" section of the config
file
> > and its output is found in the NBRCNT, NBRCTC, and NBRCTS line
> > types. Here's how the
> logic works...
> > - For each forecast variable being verified and each
categorical
> > threshold specified (i.e. the "cat_thresh" setting), apply that
> > threshold to define a 0/1 bitmap field.
> > - For each "width", specified in the "nbrhd" section, use that
> > width to convert the 0/1 bitmap field to a fractional coverage
field.
> > For example, for width = 9, look at the 9 points surrounding each
> > grid point and compute the ratio of them that are 1. The
resulting
> > fractional coverage field has a range of [0, 1] and represents the
> > ratio of nearby points where the event occurred.
> > - Apply the same logic to the observation field (apply threshold
> > and use "width" to derive fractional coverage field).
> > - The statistics in the NBRCNT line type are computed by
comparing
> > the forecast and observed fractional coverage fields directly.
> > - The "cov_thresh" entries can then be applied to both the
> > fractional coverage fields and contingency table counts (NBRCTC)
and
> > statistics
> > (NBRCTS) can be computed.
> > - NBRCNT is much more widely used than NBRCTC and NBRCTS, so I
> > wouldn't suggest using them.
> > - Apply this logic for all combinations of variables,
categorical
> > thresholds (cat_thresh), and neighborhood widths (nbrhd.width).
> >
> > (2) As described above, the neighborhood output is computed AFTER
> > the categorical threshold has been applied. The other method in
> > Grid-Stat is applied to the RAW data directly (so no
thresholding).
> > Upscaling
> (i.e.
> > smoothing) in Grid-Stat is controlled by the "interp" section of
the
> > config file. The process of drawing a box around each grid point
(i.e.
> > width of 3 means a 3x3 box) is the same as the neighborhood
methods,
> > but it's simply taking the arithmetic mean of the values in that
box.
> > So the value at each grid point is replaced by the mean of the
> > values in the neighborhood. When applying upscaling, the raw data
> > is first smoothed and then the traditional verification statistics
> > are computed (i.e. SL1L2, CNT, CTC, CTS, and so on). The smoothed
> > data is NOT used as input to the neighborhood method described
> > above... neighborhood methods always start with the raw data.
> > Now which field(s) actually get smoothed? That's based on the
> > "interp.field" setting... FCST for the forecast only, OBS for the
> > observation only, or BOTH for both of them.
> >
> > Why would you want to try upscaling? It's often the case that
> > smoother, less realistic looking forecasts score better than
higher
> > resolution, more realistic looking forecasts when assessed with
> > traditional statistics (like RMSE). Upscaling is a way to
quantify
> > how your verification scores would change for a smoother forecast.
> >
> > Lastly, Grid-Stat should enforce that all widths (in both the
nbrhd
> > and
> > interp) sections be odd. That keeps the nxn box centered on each
> > grid point.
> >
> > Hope that helps clarify. Just let me know if you have more
questions.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 31, 2016 at 2:38 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=77
> > > 759 >
> > >
> > > For clarification and reference, I have attached my grid-stat
> > > configuration file.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: met_help at ucar.edu via RT [met_help at ucar.edu]
> > > Sent: Wednesday, August 31, 2016 2:00 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #77759] AutoReply:
> > > question about grid-stat (MET V4.1) (UNCLASSIFIED)
> > >
> > > Greetings,
> > >
> > > This message has been automatically generated in response to the
> > > creation of a trouble ticket regarding:
> > > "question about grid-stat (MET V4.1) (UNCLASSIFIED)", a
> > > summary of which appears below.
> > >
> > > There is no need to reply to this message right now. Your
ticket
> > > has been assigned an ID of [rt.rap.ucar.edu #77759].
> > >
> > > Please include the string:
> > >
> > > [rt.rap.ucar.edu #77759]
> > >
> > > in the subject line of all future correspondence about this
issue.
> > > To do so, you may reply to this message.
> > >
> > > Thank you,
> > > met_help at ucar.edu
> > >
> > >
------------------------------------------------------------------
> > > --
> > > --
> > > ---
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > What is the averaging method used to upscale the forecast and
> > > observed fields to successively coarser scales when neighborhood
> > > methods are used as specified in the config file? Where is this
> > > averaging method specified in the config file?
> > >
> > > I specify nbrhd widths of 1,3,5,7,9 grid points in my config
file.
> > > I'm using the CSI and FBIAS scores for analysis.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr John W. Raby, Meteorologist
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list