[Met_help] [rt.rap.ucar.edu #95678] History for MET V9.0 Grid-Stat question (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Fri Jun 19 11:49:43 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I used plot_data_plane to plot the following fields from the ncpairs output
file:
OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
The plots show a range of colors along the y-axis with associated values
ranging from 0 to 1.0. Are these values the same as the BASER and FMEAN
computed without applying the COV_THRESH (set to 0.5 for all fields)? For
example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge21.0)
shows values which ranged from 0 to .280, but the value of BASER is 0 in the
NBRCTS output. For the plot of FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.0),
the y-axis values range from 0 to .784 and the corresponding value of FMEAN
is a small non-zero number.
Thanks.
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
Office:(575) 678-2004 DSN 258-2004
Teleworking: Contact Admin Specialist at (410) 278-6974
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
From: Minna Win
Time: Fri Jun 19 09:58:19 2020
Hi John,
It looks like you have some questions about the output from
plot_data_plane. I'm assigning this ticket to John Halley Gotway.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Fri, Jun 19, 2020 at 9:32 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Fri Jun 19 09:32:38 2020: Request 95678 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I used plot_data_plane to plot the following fields from the ncpairs
output
> file:
> OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
>
> The plots show a range of colors along the y-axis with associated
values
> ranging from 0 to 1.0. Are these values the same as the BASER and
FMEAN
> computed without applying the COV_THRESH (set to 0.5 for all
fields)? For
> example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
>
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge21.0)
> shows values which ranged from 0 to .280, but the value of BASER is
0 in
> the
> NBRCTS output. For the plot of FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
>
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.0),
> the y-axis values range from 0 to .784 and the corresponding value
of FMEAN
> is a small non-zero number.
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Jun 19 10:22:05 2020
John,
I see you have a question about interpreting the fields in the NetCDF
matched pairs output file from Grid-Stat:
OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
Breaking these variable names down...
- FCST or OBS ... obviously indicates gridded forecast or observation
data.
- GUST_Z2 ... means that this is 2-m Wind Gust data.
- FULL ... means that this data covers the full verification domain,
rather
than some masking region subset.
- ge21.0 ... means that you've applied a threshold of >=21 to convert
the
raw data values into a 0/1 binary field.
- NBRHD_361 ... means that this data has been smoothed over a
neighborhood
of size 361 (i.e. 19x19 square)
You're wondering how these fields relate to the BASER and FMEAN
statistics.
Grid-Stat computes those statistics as a spatial average over multiple
grid
points, whereas these fields show values for individual grid points.
Hopefully stepping through the processing logic will clarify things:
(1) Grid-Stat reads forecast and observation fields for 2-m wind
gusts.
(2) It applies the threshold >= 21 to convert that continuous data
into
binary fields of 0's and 1's.
(3) It steps through all the grid points in the FULL verification
domain
and constructs a 2x2 contingency table.
(4) It writes the CTC line type with those contingency table counts,
as
well as the corresponding CTS statistics to the output.
In the CTS line type, BASER is the fraction of grid points in the
domain at
which the event occurred (i.e. a value of 1).
FMEAN is the fraction of grid points in the domains at which the
forecast
predicted the event to occur (i.e. a value of 1).
(5) Using the binary field from (2), convert the binary fields into
fractional coverage fields. Replace the value at each grid point with
the
frequency of the event within the current neighborhood. In our case,
that's
the fraction of the 361 points in the 9x9 square at which the event
occurred.
(6) Process the forecast and observation fractional coverage fields to
produce the NBRCNT output statistics.
(7) Additionally, if the user has requested NBRCTS output, apply the
cov_thresh to perform a categorical evaluation of those fractional
coverage
fields and write that output to the NBRCTC and NBRCTS output line
types.
Honestly, when it comes to neighborhood verification, the fractions
skill
score (FSS) and fractions brier score (FBS) in the NBRCNT line type
are
widely used. I am not aware of anyone actually using the NBRCTC and
NBRCTS
outputs. In retrospect, we probably should NOT have included them in
the
Grid-Stat output since they aren't widely used and accepted. But once
it's
in there, it's difficult to remove without a great reason for doing
so.
For your specific question, the numbers you describe make sense to me.
OBS_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.280.
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.784.
When you threshold those values using cov_thresh > 0.5, there are 0
events
in the OBS and a small number of events in the FCST. In the NBRCTS
output,
that yields BASER = 0 and FMEAN = small number.
Hope this helps clarify.
Thanks,
John Halley Gotway
On Fri, Jun 19, 2020 at 9:58 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
>
> Hi John,
>
> It looks like you have some questions about the output from
> plot_data_plane. I'm assigning this ticket to John Halley Gotway.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Fri, Jun 19, 2020 at 9:32 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Jun 19 09:32:38 2020: Request 95678 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I used plot_data_plane to plot the following fields from the
ncpairs
> output
> > file:
> > OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
> > The plots show a range of colors along the y-axis with associated
values
> > ranging from 0 to 1.0. Are these values the same as the BASER and
FMEAN
> > computed without applying the COV_THRESH (set to 0.5 for all
fields)? For
> > example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge21.0)
> > shows values which ranged from 0 to .280, but the value of BASER
is 0 in
> > the
> > NBRCTS output. For the plot of FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.0),
> > the y-axis values range from 0 to .784 and the corresponding value
of
> FMEAN
> > is a small non-zero number.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Jun 19 10:57:41 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for breaking down the process which produces the data I'm
working with.
This really helps to confirm how things are working. So the plotted
fields
could be considered spatial distributions of the BASER and FMEAN.
Would this
be true?
Actually, we almost exclusively work with the FSS and other statistics
form
the NBRCNT file. In this case I am expanding the use of Grid-Stat in a
way
which we haven't done before, by looking at many sfc met variables at
several
thresholds and neighborhood sizes and I wanted to see the actual
fields which
it is using to compute the scores and statistics to compare with the
scores to
see if the thresholds I applied are going to provide useful info. At
first I
didn't understand the values on the y-axis and I tried looking at the
other
output files (since I had already generated them) to see if it would
help
clarify what I was seeing. You can't quite get there since the
statistics are
all domain averaged and the plots show the values at each grid point.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, June 19, 2020 10:22 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95678] MET V9.0 Grid-
Stat
question (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
I see you have a question about interpreting the fields in the NetCDF
matched
pairs output file from Grid-Stat:
OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
Breaking these variable names down...
- FCST or OBS ... obviously indicates gridded forecast or observation
data.
- GUST_Z2 ... means that this is 2-m Wind Gust data.
- FULL ... means that this data covers the full verification domain,
rather
than some masking region subset.
- ge21.0 ... means that you've applied a threshold of >=21 to convert
the raw
data values into a 0/1 binary field.
- NBRHD_361 ... means that this data has been smoothed over a
neighborhood of
size 361 (i.e. 19x19 square)
You're wondering how these fields relate to the BASER and FMEAN
statistics.
Grid-Stat computes those statistics as a spatial average over multiple
grid
points, whereas these fields show values for individual grid points.
Hopefully stepping through the processing logic will clarify things:
(1) Grid-Stat reads forecast and observation fields for 2-m wind
gusts.
(2) It applies the threshold >= 21 to convert that continuous data
into binary
fields of 0's and 1's.
(3) It steps through all the grid points in the FULL verification
domain and
constructs a 2x2 contingency table.
(4) It writes the CTC line type with those contingency table counts,
as well
as the corresponding CTS statistics to the output.
In the CTS line type, BASER is the fraction of grid points in the
domain at
which the event occurred (i.e. a value of 1).
FMEAN is the fraction of grid points in the domains at which the
forecast
predicted the event to occur (i.e. a value of 1).
(5) Using the binary field from (2), convert the binary fields into
fractional
coverage fields. Replace the value at each grid point with the
frequency of
the event within the current neighborhood. In our case, that's the
fraction of
the 361 points in the 9x9 square at which the event occurred.
(6) Process the forecast and observation fractional coverage fields to
produce
the NBRCNT output statistics.
(7) Additionally, if the user has requested NBRCTS output, apply the
cov_thresh to perform a categorical evaluation of those fractional
coverage
fields and write that output to the NBRCTC and NBRCTS output line
types.
Honestly, when it comes to neighborhood verification, the fractions
skill
score (FSS) and fractions brier score (FBS) in the NBRCNT line type
are widely
used. I am not aware of anyone actually using the NBRCTC and NBRCTS
outputs.
In retrospect, we probably should NOT have included them in the Grid-
Stat
output since they aren't widely used and accepted. But once it's in
there,
it's difficult to remove without a great reason for doing so.
For your specific question, the numbers you describe make sense to me.
OBS_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.280.
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.784.
When you threshold those values using cov_thresh > 0.5, there are 0
events in
the OBS and a small number of events in the FCST. In the NBRCTS
output, that
yields BASER = 0 and FMEAN = small number.
Hope this helps clarify.
Thanks,
John Halley Gotway
On Fri, Jun 19, 2020 at 9:58 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
>
> Hi John,
>
> It looks like you have some questions about the output from
> plot_data_plane. I'm assigning this ticket to John Halley Gotway.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research Developmental Testbed
Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Fri, Jun 19, 2020 at 9:32 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Jun 19 09:32:38 2020: Request 95678 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I used plot_data_plane to plot the following fields from the
ncpairs
> output
> > file:
> > OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
> > The plots show a range of colors along the y-axis with associated
values
> > ranging from 0 to 1.0. Are these values the same as the BASER and
FMEAN
> > computed without applying the COV_THRESH (set to 0.5 for all
fields)? For
> > example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge21.0)
> > shows values which ranged from 0 to .280, but the value of BASER
is 0 in
> > the
> > NBRCTS output. For the plot of FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.0),
> > the y-axis values range from 0 to .784 and the corresponding value
of
> FMEAN
> > is a small non-zero number.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Jun 19 11:04:23 2020
John,
Great, glad that helped. As for whether "the plotted fields could be
considered spatial distributions of the BASER and FMEAN", I'm not
totally
sure. It is true that the value at each grid point is the frequency of
the
event within the surrounding neighborhood. And it's also true the
BASER and
FMEAN are the frequency of the event across all points within a
verification region. So they are very, very similar, but the devil is
in
the details of the processing logic. I usually just call those plotted
fields the forecast and observation "fractional coverage fields".
Thanks,
John
On Fri, Jun 19, 2020 at 10:58 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for breaking down the process which produces the data I'm
working
> with.
> This really helps to confirm how things are working. So the plotted
fields
> could be considered spatial distributions of the BASER and FMEAN.
Would
> this
> be true?
>
> Actually, we almost exclusively work with the FSS and other
statistics
> form
> the NBRCNT file. In this case I am expanding the use of Grid-Stat in
a way
> which we haven't done before, by looking at many sfc met variables
at
> several
> thresholds and neighborhood sizes and I wanted to see the actual
fields
> which
> it is using to compute the scores and statistics to compare with the
> scores to
> see if the thresholds I applied are going to provide useful info. At
first
> I
> didn't understand the values on the y-axis and I tried looking at
the
> other
> output files (since I had already generated them) to see if it would
help
> clarify what I was seeing. You can't quite get there since the
statistics
> are
> all domain averaged and the plots show the values at each grid
point.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, June 19, 2020 10:22 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95678] MET V9.0
Grid-Stat
> question (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I see you have a question about interpreting the fields in the
NetCDF
> matched
> pairs output file from Grid-Stat:
>
> OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
>
> Breaking these variable names down...
> - FCST or OBS ... obviously indicates gridded forecast or
observation data.
> - GUST_Z2 ... means that this is 2-m Wind Gust data.
> - FULL ... means that this data covers the full verification domain,
> rather
> than some masking region subset.
> - ge21.0 ... means that you've applied a threshold of >=21 to
convert the
> raw
> data values into a 0/1 binary field.
> - NBRHD_361 ... means that this data has been smoothed over a
neighborhood
> of
> size 361 (i.e. 19x19 square)
>
> You're wondering how these fields relate to the BASER and FMEAN
statistics.
> Grid-Stat computes those statistics as a spatial average over
multiple
> grid
> points, whereas these fields show values for individual grid points.
> Hopefully stepping through the processing logic will clarify things:
>
> (1) Grid-Stat reads forecast and observation fields for 2-m wind
gusts.
> (2) It applies the threshold >= 21 to convert that continuous data
into
> binary
> fields of 0's and 1's.
> (3) It steps through all the grid points in the FULL verification
domain
> and
> constructs a 2x2 contingency table.
> (4) It writes the CTC line type with those contingency table counts,
as
> well
> as the corresponding CTS statistics to the output.
> In the CTS line type, BASER is the fraction of grid points in the
domain
> at
> which the event occurred (i.e. a value of 1).
> FMEAN is the fraction of grid points in the domains at which the
forecast
> predicted the event to occur (i.e. a value of 1).
> (5) Using the binary field from (2), convert the binary fields into
> fractional
> coverage fields. Replace the value at each grid point with the
frequency
> of
> the event within the current neighborhood. In our case, that's the
> fraction of
> the 361 points in the 9x9 square at which the event occurred.
> (6) Process the forecast and observation fractional coverage fields
to
> produce
> the NBRCNT output statistics.
> (7) Additionally, if the user has requested NBRCTS output, apply the
> cov_thresh to perform a categorical evaluation of those fractional
> coverage
> fields and write that output to the NBRCTC and NBRCTS output line
types.
>
> Honestly, when it comes to neighborhood verification, the fractions
skill
> score (FSS) and fractions brier score (FBS) in the NBRCNT line type
are
> widely
> used. I am not aware of anyone actually using the NBRCTC and NBRCTS
> outputs.
> In retrospect, we probably should NOT have included them in the
Grid-Stat
> output since they aren't widely used and accepted. But once it's in
there,
> it's difficult to remove without a great reason for doing so.
>
> For your specific question, the numbers you describe make sense to
me.
>
> OBS_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.280.
> FCST_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.784.
> When you threshold those values using cov_thresh > 0.5, there are 0
events
> in
> the OBS and a small number of events in the FCST. In the NBRCTS
output,
> that
> yields BASER = 0 and FMEAN = small number.
>
> Hope this helps clarify.
>
> Thanks,
> John Halley Gotway
>
>
> On Fri, Jun 19, 2020 at 9:58 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
> >
> > Hi John,
> >
> > It looks like you have some questions about the output from
> > plot_data_plane. I'm assigning this ticket to John Halley Gotway.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research Developmental Testbed
Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Fri, Jun 19, 2020 at 9:32 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Jun 19 09:32:38 2020: Request 95678 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-url:
> > > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678
>
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I used plot_data_plane to plot the following fields from the
ncpairs
> > output
> > > file:
> > > OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > > FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
> > > The plots show a range of colors along the y-axis with
associated
> values
> > > ranging from 0 to 1.0. Are these values the same as the BASER
and FMEAN
> > > computed without applying the COV_THRESH (set to 0.5 for all
fields)?
> For
> > > example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge21.0)
> > > shows values which ranged from 0 to .280, but the value of BASER
is 0
> in
> > > the
> > > NBRCTS output. For the plot of
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
>
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.0),
> > > the y-axis values range from 0 to .784 and the corresponding
value of
> > FMEAN
> > > is a small non-zero number.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Jun 19 11:49:15 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for sharing that. I didn't want to stretch the definition too
much and
your term "fractional coverage fields" certainly works for me.
Take care.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, June 19, 2020 11:04 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95678] MET V9.0
Grid-Stat
question (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Great, glad that helped. As for whether "the plotted fields could be
considered spatial distributions of the BASER and FMEAN", I'm not
totally
sure. It is true that the value at each grid point is the frequency of
the
event within the surrounding neighborhood. And it's also true the
BASER and
FMEAN are the frequency of the event across all points within a
verification
region. So they are very, very similar, but the devil is in the
details of the
processing logic. I usually just call those plotted fields the
forecast and
observation "fractional coverage fields".
Thanks,
John
On Fri, Jun 19, 2020 at 10:58 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95678 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for breaking down the process which produces the data I'm
> working with.
> This really helps to confirm how things are working. So the plotted
> fields could be considered spatial distributions of the BASER and
> FMEAN. Would this be true?
>
> Actually, we almost exclusively work with the FSS and other
statistics
> form the NBRCNT file. In this case I am expanding the use of Grid-
Stat
> in a way which we haven't done before, by looking at many sfc met
> variables at several thresholds and neighborhood sizes and I wanted
to
> see the actual fields which it is using to compute the scores and
> statistics to compare with the scores to see if the thresholds I
> applied are going to provide useful info. At first I didn't
understand
> the values on the y-axis and I tried looking at the other output
files
> (since I had already generated them) to see if it would help clarify
> what I was seeing. You can't quite get there since the statistics
are
> all domain averaged and the plots show the values at each grid
point.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, June 19, 2020 10:22 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95678] MET V9.0
> Grid-Stat question (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I see you have a question about interpreting the fields in the
NetCDF
> matched pairs output file from Grid-Stat:
>
> OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
>
> Breaking these variable names down...
> - FCST or OBS ... obviously indicates gridded forecast or
observation data.
> - GUST_Z2 ... means that this is 2-m Wind Gust data.
> - FULL ... means that this data covers the full verification domain,
> rather than some masking region subset.
> - ge21.0 ... means that you've applied a threshold of >=21 to
convert
> the raw data values into a 0/1 binary field.
> - NBRHD_361 ... means that this data has been smoothed over a
> neighborhood of size 361 (i.e. 19x19 square)
>
> You're wondering how these fields relate to the BASER and FMEAN
statistics.
> Grid-Stat computes those statistics as a spatial average over
multiple
> grid points, whereas these fields show values for individual grid
> points.
> Hopefully stepping through the processing logic will clarify things:
>
> (1) Grid-Stat reads forecast and observation fields for 2-m wind
gusts.
> (2) It applies the threshold >= 21 to convert that continuous data
> into binary fields of 0's and 1's.
> (3) It steps through all the grid points in the FULL verification
> domain and constructs a 2x2 contingency table.
> (4) It writes the CTC line type with those contingency table counts,
> as well as the corresponding CTS statistics to the output.
> In the CTS line type, BASER is the fraction of grid points in the
> domain at which the event occurred (i.e. a value of 1).
> FMEAN is the fraction of grid points in the domains at which the
> forecast predicted the event to occur (i.e. a value of 1).
> (5) Using the binary field from (2), convert the binary fields into
> fractional coverage fields. Replace the value at each grid point
with
> the frequency of the event within the current neighborhood. In our
> case, that's the fraction of the 361 points in the 9x9 square at
which
> the event occurred.
> (6) Process the forecast and observation fractional coverage fields
to
> produce the NBRCNT output statistics.
> (7) Additionally, if the user has requested NBRCTS output, apply the
> cov_thresh to perform a categorical evaluation of those fractional
> coverage fields and write that output to the NBRCTC and NBRCTS
output
> line types.
>
> Honestly, when it comes to neighborhood verification, the fractions
> skill score (FSS) and fractions brier score (FBS) in the NBRCNT line
> type are widely used. I am not aware of anyone actually using the
> NBRCTC and NBRCTS outputs.
> In retrospect, we probably should NOT have included them in the
> Grid-Stat output since they aren't widely used and accepted. But
once
> it's in there, it's difficult to remove without a great reason for
doing so.
>
> For your specific question, the numbers you describe make sense to
me.
>
> OBS_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.280.
> FCST_GUST_Z2_FULL_ge21.0_NBRHD_361 ranges from 0 to 0.784.
> When you threshold those values using cov_thresh > 0.5, there are 0
> events in the OBS and a small number of events in the FCST. In the
> NBRCTS output, that yields BASER = 0 and FMEAN = small number.
>
> Hope this helps clarify.
>
> Thanks,
> John Halley Gotway
>
>
> On Fri, Jun 19, 2020 at 9:58 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > 678 >
> >
> > Hi John,
> >
> > It looks like you have some questions about the output from
> > plot_data_plane. I'm assigning this ticket to John Halley Gotway.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research Developmental Testbed
> > Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Fri, Jun 19, 2020 at 9:32 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Jun 19 09:32:38 2020: Request 95678 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V9.0 Grid-Stat question (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=
> > > 95678 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I used plot_data_plane to plot the following fields from the
> > > ncpairs
> > output
> > > file:
> > > OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > > FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
> > > The plots show a range of colors along the y-axis with
associated
> values
> > > ranging from 0 to 1.0. Are these values the same as the BASER
and
> > > FMEAN computed without applying the COV_THRESH (set to 0.5 for
all
> > > fields)?
> For
> > > example, the attached plot of OBS_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_OBS_GUSTge
> > > 21.0) shows values which ranged from 0 to .280, but the value of
> > > BASER is 0
> in
> > > the
> > > NBRCTS output. For the plot of
FCST_GUST_Z2_FULL_ge21.0_NBRHD_361
> > >
>
(grid_stat_WREN_URMA_d02_test6_030000L_20161127_120000V_FCST_GUSTge21.
> 0),
> > > the y-axis values range from 0 to .784 and the corresponding
value
> > > of
> > FMEAN
> > > is a small non-zero number.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list