[Met_help] [rt.rap.ucar.edu #75637] History for question about Grid-Stat (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Fri Mar 25 15:04:31 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I want to know if it is possible to specify ranges of the variable (i.e. bins) rather than thresholds for calculating discrete statistics?
I am using MET V4.1 and the specifications for the cat_thresh setting appear to only define thresholds except for the use of the convention "eq" of "ne" unless combinations of the others can be used to apply upper and lower bounds.
An example of what I'm looking for would be the calculation of the statistics for a 2m AGL temperature forecast for ranges of temperature such as 265-270, 270-275, 275-280, 280-285,
285-290, and 290-295 K. I'm also thinking of overlapping bins too, such as 280-283, 281-284, 282-285.
I looked at the release notes for V5.0 and V5.1, but didn't see evidence of recent enhancements to Grid-Stat which might indicate the addition of this capability.
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Mar 24 10:03:02 2016
John,
MET version 4.1 has very limited support for this. The MCTC
(multi-category contingency table) enables you to analyze the
performance
across multiple categories. For example, suppose you set:
cat_thresh = [ ">=265", ">=270", ">=275", ">=280", ">=285" ];
Those 5 thresholds are used to partition the space into 6 discrete
bins:
less than 265, 265-270, 270-275, 275-280, 280-285, and more than 285.
Those 6 bins are used to construct a 6x6 contingency table and the
counts
for each bin are written out in the MCTC line type. Also, the MCTS
line
type contains the (handful of) statistics that can be derived from the
6x6
table.
MET version 5.1 includes much more support for this, as listed in the
release notes:
http://www.dtcenter.org/met/users/support/release_notes/METv5.1_release_notes.php
Both "Generalized thresholding" and "Conditional continuous
verification"
are relevant here.
Earlier versions of MET only let you define simple thresholds, like
">5" or
"<=10". In MET 5.1, you can use "&&" and "||" notation. See page 44:
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
For example, ">=265&&<=270" would define all values between 265 and
270.
When doing categorical verification, any value falling in that range
is
considered an "event" and any value falling outside is a "non-event".
On to conditional continuous verification... In earlier versions of
MET,
RMSE and ME (and all the other continuous statistics) were computed
over
the full range of forecast and observation matched pairs. However,
you now
have the option of filtering the forecast and/or observation values
prior
to computing that statistics. The "cnt_thresh" and "cnt_logic"
entries
were added to the Grid-Stat configuration file to control this
behavior.
For example, suppose you want to compute RMSE but only when the
observed
value is above freezing. Set:
In the "obs" dictionary, set: cnt_thresh = [ >273 ];
In the "fcst" dictionary, set: cnt_thresh = [ NA ];
cnt_logic = INTERSECTION;
The "fcst" NA threshold will match all values while the "obs" >273
threshold will only match observations greater than 273. Taking the
"INTERSECTION" of those sets will yield the matched pairs where the
observations are greater than >273.
Some more examples (description ... fcst cnt_thresh ... obs cnt_thresh
...
cnt_logic):
fcst and obs greater than freezing ... >273 ... >273 ...
INTESERCTION
fcst or obs greater than freezing ... >273 ... >273 ... UNION
fcst and obs between 265 and 270 ... >=265&&<=270 ... >=265&&<=270
...
INTERSECTION
Doing conditional continuous verification and more generalized
categorical
verification are both very reasonable ways to process your data.
However,
the methods you choose to apply should really be driven by the
forecast
verification questions you have.
Are you more interested in continuous performance in a specific range
of
values? Or is the forecast really used in a more discrete, binned
way,
where turning the ranges into categories makes more sense?
Hope that helps clarify.
Thanks,
John
On Thu, Mar 24, 2016 at 9:21 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Thu Mar 24 09:21:01 2016: Request 75637 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about Grid-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75637 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I want to know if it is possible to specify ranges of the variable
(i.e.
> bins) rather than thresholds for calculating discrete statistics?
>
> I am using MET V4.1 and the specifications for the cat_thresh
setting
> appear to only define thresholds except for the use of the
convention "eq"
> of "ne" unless combinations of the others can be used to apply upper
and
> lower bounds.
>
> An example of what I'm looking for would be the calculation of the
> statistics for a 2m AGL temperature forecast for ranges of
temperature such
> as 265-270, 270-275, 275-280, 280-285,
> 285-290, and 290-295 K. I'm also thinking of overlapping bins too,
such as
> 280-283, 281-284, 282-285.
>
> I looked at the release notes for V5.0 and V5.1, but didn't see
evidence
> of recent enhancements to Grid-Stat which might indicate the
addition of
> this capability.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75637] question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Mar 24 10:50:43 2016
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for this detailed information.
I have the MCTC and MCTS output and will see if this will provide the
information we were looking for.
Nice to see the enhancements you mentioned below. Looking forward to
installing V5.1.
Regards your question about whether we are interested in continuous
performance in a specific range of values or is the forecast really
used in a more discrete, binned way, where turning the ranges into
categories makes more sense? We are in the very early stages of
learning how the application of neighborhood methods for fuzzy
verification adds to the information gleaned from our previous work
using the traditional grid-to-point methods applied to forecasts of
continuous variables. My sense is that the information coming from the
two approaches serves to provide a more comprehensive assessment of
the model, but we understand that each different
score/statistic/method provides information about only certain aspects
of the model performance. We are also running MODE on the same input
files to see what that tells us about performance spatially. Our hope
is that by stitching together all results in a way which makes sense
will give an accurate and more complete picture of the model
performance and point to specific deficiencies which can be worked on
to make improvements. So, I guess the short answer is "all of the
above" for now.
Brian, please chime in if you have some more specific interests
regards to the information provided or the questions asked.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 24, 2016 10:03 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Cc: Reen, Brian P CIV USARMY RDECOM ARL (US)
<brian.p.reen.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75637] question about
Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
MET version 4.1 has very limited support for this. The MCTC (multi-
category contingency table) enables you to analyze the performance
across multiple categories. For example, suppose you set:
cat_thresh = [ ">=265", ">=270", ">=275", ">=280", ">=285" ];
Those 5 thresholds are used to partition the space into 6 discrete
bins:
less than 265, 265-270, 270-275, 275-280, 280-285, and more than 285.
Those 6 bins are used to construct a 6x6 contingency table and the
counts for each bin are written out in the MCTC line type. Also, the
MCTS line type contains the (handful of) statistics that can be
derived from the 6x6 table.
MET version 5.1 includes much more support for this, as listed in the
release notes:
Caution-
http://www.dtcenter.org/met/users/support/release_notes/METv5.1_release_notes.php
Both "Generalized thresholding" and "Conditional continuous
verification"
are relevant here.
Earlier versions of MET only let you define simple thresholds, like
">5" or "<=10". In MET 5.1, you can use "&&" and "||" notation. See
page 44:
Caution-
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
For example, ">=265&&<=270" would define all values between 265 and
270.
When doing categorical verification, any value falling in that range
is considered an "event" and any value falling outside is a "non-
event".
On to conditional continuous verification... In earlier versions of
MET, RMSE and ME (and all the other continuous statistics) were
computed over the full range of forecast and observation matched
pairs. However, you now have the option of filtering the forecast
and/or observation values prior to computing that statistics. The
"cnt_thresh" and "cnt_logic" entries were added to the Grid-Stat
configuration file to control this behavior.
For example, suppose you want to compute RMSE but only when the
observed value is above freezing. Set:
In the "obs" dictionary, set: cnt_thresh = [ >273 ];
In the "fcst" dictionary, set: cnt_thresh = [ NA ];
cnt_logic = INTERSECTION;
The "fcst" NA threshold will match all values while the "obs" >273
threshold will only match observations greater than 273. Taking the
"INTERSECTION" of those sets will yield the matched pairs where the
observations are greater than >273.
Some more examples (description ... fcst cnt_thresh ... obs cnt_thresh
...
cnt_logic):
fcst and obs greater than freezing ... >273 ... >273 ...
INTESERCTION
fcst or obs greater than freezing ... >273 ... >273 ... UNION
fcst and obs between 265 and 270 ... >=265&&<=270 ... >=265&&<=270
...
INTERSECTION
Doing conditional continuous verification and more generalized
categorical verification are both very reasonable ways to process your
data. However, the methods you choose to apply should really be
driven by the forecast verification questions you have.
Are you more interested in continuous performance in a specific range
of values? Or is the forecast really used in a more discrete, binned
way, where turning the ranges into categories makes more sense?
Hope that helps clarify.
Thanks,
John
On Thu, Mar 24, 2016 at 9:21 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Thu Mar 24 09:21:01 2016: Request 75637 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about Grid-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75637 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I want to know if it is possible to specify ranges of the variable
(i.e.
> bins) rather than thresholds for calculating discrete statistics?
>
> I am using MET V4.1 and the specifications for the cat_thresh
setting
> appear to only define thresholds except for the use of the
convention "eq"
> of "ne" unless combinations of the others can be used to apply upper
> and lower bounds.
>
> An example of what I'm looking for would be the calculation of the
> statistics for a 2m AGL temperature forecast for ranges of
temperature
> such as 265-270, 270-275, 275-280, 280-285, 285-290, and 290-295 K.
> I'm also thinking of overlapping bins too, such as 280-283, 281-284,
> 282-285.
>
> I looked at the release notes for V5.0 and V5.1, but didn't see
> evidence of recent enhancements to Grid-Stat which might indicate
the
> addition of this capability.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Mar 25 15:04:08 2016
John,
Sounds good. I'll resolve the ticket for now. Just write us if more
issues or questions arise.
Thanks,
John
On Thu, Mar 24, 2016 at 10:50 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75637 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for this detailed information.
>
> I have the MCTC and MCTS output and will see if this will provide
the
> information we were looking for.
>
> Nice to see the enhancements you mentioned below. Looking forward to
> installing V5.1.
>
> Regards your question about whether we are interested in continuous
> performance in a specific range of values or is the forecast really
used in
> a more discrete, binned way, where turning the ranges into
categories makes
> more sense? We are in the very early stages of learning how the
application
> of neighborhood methods for fuzzy verification adds to the
information
> gleaned from our previous work using the traditional grid-to-point
methods
> applied to forecasts of continuous variables. My sense is that the
> information coming from the two approaches serves to provide a more
> comprehensive assessment of the model, but we understand that each
> different score/statistic/method provides information about only
certain
> aspects of the model performance. We are also running MODE on the
same
> input files to see what that tells us about performance spatially.
Our hope
> is that by stitching together all results in a way which makes sense
will
> give an accurate and more complete picture of the m!
> odel performance and point to specific deficiencies which can be
worked
> on to make improvements. So, I guess the short answer is "all of the
above"
> for now.
>
> Brian, please chime in if you have some more specific interests
regards to
> the information provided or the questions asked.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 24, 2016 10:03 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Cc: Reen, Brian P CIV USARMY RDECOM ARL (US)
<brian.p.reen.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75637] question
about
> Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> MET version 4.1 has very limited support for this. The MCTC
> (multi-category contingency table) enables you to analyze the
performance
> across multiple categories. For example, suppose you set:
> cat_thresh = [ ">=265", ">=270", ">=275", ">=280", ">=285" ];
>
> Those 5 thresholds are used to partition the space into 6 discrete
bins:
> less than 265, 265-270, 270-275, 275-280, 280-285, and more than
285.
> Those 6 bins are used to construct a 6x6 contingency table and the
counts
> for each bin are written out in the MCTC line type. Also, the MCTS
line
> type contains the (handful of) statistics that can be derived from
the 6x6
> table.
>
> MET version 5.1 includes much more support for this, as listed in
the
> release notes:
>
> Caution-
>
http://www.dtcenter.org/met/users/support/release_notes/METv5.1_release_notes.php
>
> Both "Generalized thresholding" and "Conditional continuous
verification"
> are relevant here.
>
> Earlier versions of MET only let you define simple thresholds, like
">5"
> or "<=10". In MET 5.1, you can use "&&" and "||" notation. See
page 44:
>
> Caution-
>
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
>
> For example, ">=265&&<=270" would define all values between 265 and
270.
> When doing categorical verification, any value falling in that range
is
> considered an "event" and any value falling outside is a "non-
event".
>
> On to conditional continuous verification... In earlier versions of
MET,
> RMSE and ME (and all the other continuous statistics) were computed
over
> the full range of forecast and observation matched pairs. However,
you now
> have the option of filtering the forecast and/or observation values
prior
> to computing that statistics. The "cnt_thresh" and "cnt_logic"
entries
> were added to the Grid-Stat configuration file to control this
behavior.
> For example, suppose you want to compute RMSE but only when the
observed
> value is above freezing. Set:
> In the "obs" dictionary, set: cnt_thresh = [ >273 ];
> In the "fcst" dictionary, set: cnt_thresh = [ NA ];
> cnt_logic =
INTERSECTION;
>
> The "fcst" NA threshold will match all values while the "obs" >273
> threshold will only match observations greater than 273. Taking the
> "INTERSECTION" of those sets will yield the matched pairs where the
> observations are greater than >273.
>
> Some more examples (description ... fcst cnt_thresh ... obs
cnt_thresh ...
> cnt_logic):
> fcst and obs greater than freezing ... >273 ... >273 ...
INTESERCTION
> fcst or obs greater than freezing ... >273 ... >273 ... UNION
> fcst and obs between 265 and 270 ... >=265&&<=270 ...
>=265&&<=270 ...
> INTERSECTION
>
> Doing conditional continuous verification and more generalized
categorical
> verification are both very reasonable ways to process your data.
However,
> the methods you choose to apply should really be driven by the
forecast
> verification questions you have.
>
> Are you more interested in continuous performance in a specific
range of
> values? Or is the forecast really used in a more discrete, binned
way,
> where turning the ranges into categories makes more sense?
>
> Hope that helps clarify.
>
> Thanks,
> John
>
>
> On Thu, Mar 24, 2016 at 9:21 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Mar 24 09:21:01 2016: Request 75637 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about Grid-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75637 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I want to know if it is possible to specify ranges of the variable
(i.e.
> > bins) rather than thresholds for calculating discrete statistics?
> >
> > I am using MET V4.1 and the specifications for the cat_thresh
setting
> > appear to only define thresholds except for the use of the
convention
> "eq"
> > of "ne" unless combinations of the others can be used to apply
upper
> > and lower bounds.
> >
> > An example of what I'm looking for would be the calculation of the
> > statistics for a 2m AGL temperature forecast for ranges of
temperature
> > such as 265-270, 270-275, 275-280, 280-285, 285-290, and 290-295
K.
> > I'm also thinking of overlapping bins too, such as 280-283, 281-
284,
> > 282-285.
> >
> > I looked at the release notes for V5.0 and V5.1, but didn't see
> > evidence of recent enhancements to Grid-Stat which might indicate
the
> > addition of this capability.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
More information about the Met_help
mailing list