[Met_help] [rt.rap.ucar.edu #95790] History for MET V9.0 Point-Stat error (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Thu Jul 2 12:53:07 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I keep getting the same error when trying to run Point-Stat with the HiRA
verification logic enabled. The error involves the probability field
thresholds:
ERROR:check_prob_thresh() -> When verifying a probability field, you must
select at least 3 thresholds beginning with 0.0 and ending with 1.0.
During troubleshooting, I've changed the settings for prob_cat_thresh in the
attached config file several times thinking that the formatting of my
settings was the issue. Maybe I'm on the wrong track. Could you take a look
at the config file to check the format or other possible issue which might
be causing this issue?
Thanks.
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
Office:(575) 678-2004 DSN 258-2004
Teleworking: Contact Admin Specialist at (410) 278-6974
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Jun 30 17:11:05 2020
John,
Please try using this shortcut:
prob_cat_thresh = [ ==0.25 ];
Does that do it?
Also, your "desc" setting can't contain embedded whitespace. And
Point-Stat
should error out and tell you that.
John
On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I keep getting the same error when trying to run Point-Stat with the
HiRA
> verification logic enabled. The error involves the probability field
> thresholds:
>
> ERROR:check_prob_thresh() -> When verifying a probability field, you
must
> select at least 3 thresholds beginning with 0.0 and ending with 1.0.
>
> During troubleshooting, I've changed the settings for
prob_cat_thresh in
> the
> attached config file several times thinking that the formatting of
my
> settings was the issue. Maybe I'm on the wrong track. Could you take
a look
> at the config file to check the format or other possible issue which
might
> be causing this issue?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jun 30 18:11:01 2020
Thanks for taking a look, John. I'll give that a try. Appreciate the
heads-up on a future error in the "desc".
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, June 30, 2020 5:11 PM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-
Stat error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Please try using this shortcut:
prob_cat_thresh = [ ==0.25 ];
Does that do it?
Also, your "desc" setting can't contain embedded whitespace. And
Point-Stat
should error out and tell you that.
John
On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I keep getting the same error when trying to run Point-Stat with the
HiRA
> verification logic enabled. The error involves the probability field
> thresholds:
>
> ERROR:check_prob_thresh() -> When verifying a probability field, you
must
> select at least 3 thresholds beginning with 0.0 and ending with 1.0.
>
> During troubleshooting, I've changed the settings for
prob_cat_thresh in
> the
> attached config file several times thinking that the formatting of
my
> settings was the issue. Maybe I'm on the wrong track. Could you take
a look
> at the config file to check the format or other possible issue which
might
> be causing this issue?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jun 30 19:00:30 2020
I edited the config file to use the shortcut and corrected the
whitespace issue, but I get the same error.
R/
John
________________________________________
From: Raby, John W CIV USARMY CCDC ARL (USA)
Sent: Tuesday, June 30, 2020 6:10 PM
To: met_help at ucar.edu
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat error (UNCLASSIFIED)
Thanks for taking a look, John. I'll give that a try. Appreciate the
heads-up on a future error in the "desc".
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, June 30, 2020 5:11 PM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-
Stat error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Please try using this shortcut:
prob_cat_thresh = [ ==0.25 ];
Does that do it?
Also, your "desc" setting can't contain embedded whitespace. And
Point-Stat
should error out and tell you that.
John
On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I keep getting the same error when trying to run Point-Stat with the
HiRA
> verification logic enabled. The error involves the probability field
> thresholds:
>
> ERROR:check_prob_thresh() -> When verifying a probability field, you
must
> select at least 3 thresholds beginning with 0.0 and ending with 1.0.
>
> During troubleshooting, I've changed the settings for
prob_cat_thresh in
> the
> attached config file several times thinking that the formatting of
my
> settings was the issue. Maybe I'm on the wrong track. Could you take
a look
> at the config file to check the format or other possible issue which
might
> be causing this issue?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jun 30 19:12:00 2020
I forgot to attach the log file.
________________________________________
From: Raby, John W CIV USARMY CCDC ARL (USA)
Sent: Tuesday, June 30, 2020 7:00 PM
To: met_help at ucar.edu
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat error (UNCLASSIFIED)
I edited the config file to use the shortcut and corrected the
whitespace issue, but I get the same error.
R/
John
________________________________________
From: Raby, John W CIV USARMY CCDC ARL (USA)
Sent: Tuesday, June 30, 2020 6:10 PM
To: met_help at ucar.edu
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat error (UNCLASSIFIED)
Thanks for taking a look, John. I'll give that a try. Appreciate the
heads-up on a future error in the "desc".
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, June 30, 2020 5:11 PM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-
Stat error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Please try using this shortcut:
prob_cat_thresh = [ ==0.25 ];
Does that do it?
Also, your "desc" setting can't contain embedded whitespace. And
Point-Stat
should error out and tell you that.
John
On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I keep getting the same error when trying to run Point-Stat with the
HiRA
> verification logic enabled. The error involves the probability field
> thresholds:
>
> ERROR:check_prob_thresh() -> When verifying a probability field, you
must
> select at least 3 thresholds beginning with 0.0 and ending with 1.0.
>
> During troubleshooting, I've changed the settings for
prob_cat_thresh in
> the
> attached config file several times thinking that the formatting of
my
> settings was the issue. Maybe I'm on the wrong track. Could you take
a look
> at the config file to check the format or other possible issue which
might
> be causing this issue?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jul 01 11:17:15 2020
John,
Sorry for the confusion. I led you astray. The HiRA configuration
option
names are not entirely intuitive.
Please take a look at the README file starting on line 1418:
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f5d2b0db9a37/met/data/config/README#L1418
I modified your hira entry like this:
hira = {
flag = TRUE; *// set to TRUE/false to turn on/off
this
logic*
width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
size and must be > 1*
vld_thresh = 1.0;
cov_thresh = [ ==0.25 ]; *// this defines the probability bins
to be
used. You only get to pick one setting here (==0.25 is short for
>=0,>=0.25,>=0.5,>=0.75,>=1.0)*
shape = SQUARE;
prob_cat_thresh = [ >290, >300 ];* // this defines the threshold(s)
for
which HiRA logic should be applied*
}
So you just had things a bit mis-configured. The key thing to note
here is
that the hira.prob_cat_thresh setting defines the thresholds for which
HiRA
probabilities should be computed. This is different from the
cat_thresh
setting in the fcst and obs dictionaries. Those are used to compute
categorical counts and statistics and are currently separate.
It's nice that you don't have to use the same set of thresholds for
categorical stats and the HiRA logic... it's more configurable that
way.
However, I do see that it makes things a bit inconvenient. Since
you're
applying HiRA to many variables and each of those variables has
different
meaningful thresholds, you'd need to configure the HiRA dictionary
separately for each and every variable.
That's what is meant by this comment... that the hira dictionary can
be
specified separately for each variable:
*// HiRA verification method// May be set separately in each
"obs.field"
entry*
So you'd need to copy/paste the HiRA dictionary into every single
obs.field
dictionary entry and modify the prob_cat_thresh thresholds
accordingly.
First, please make sure that you can in fact produce HiRA output.
Second, I
can think of a tweak to the logic that might make using HiRA easier.
How
about, if the hira.prob_cat_thresh array is an empty list, we could
have
the code check to see if cat_thresh in the obs dictionary is non-
empty. And
if that's non-empty then we'd simply set prob_cat_thresh = cat_thresh.
I imagine that's what you were expecting in the first place... that
Point-Stat would just be using the thresholds from cat_thresh to
define the
HiRA probability thresholds. It still is nice to have prob_cat_thresh
be
configurable separately, but if left unset, defaulting to cat_thresh
seems
pretty reasonable to me. Do you think that would be a worthwhile
change?
Thanks,
John
On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> I forgot to attach the log file.
>
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, June 30, 2020 7:00 PM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> I edited the config file to use the shortcut and corrected the
whitespace
> issue, but I get the same error.
> R/
> John
>
>
>
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, June 30, 2020 6:10 PM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> Thanks for taking a look, John. I'll give that a try. Appreciate the
> heads-up on a future error in the "desc".
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, June 30, 2020 5:11 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Please try using this shortcut:
> prob_cat_thresh = [ ==0.25 ];
>
> Does that do it?
>
> Also, your "desc" setting can't contain embedded whitespace. And
Point-Stat
> should error out and tell you that.
>
> John
>
> On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I keep getting the same error when trying to run Point-Stat with
the HiRA
> > verification logic enabled. The error involves the probability
field
> > thresholds:
> >
> > ERROR:check_prob_thresh() -> When verifying a probability field,
you must
> > select at least 3 thresholds beginning with 0.0 and ending with
1.0.
> >
> > During troubleshooting, I've changed the settings for
prob_cat_thresh in
> > the
> > attached config file several times thinking that the formatting of
my
> > settings was the issue. Maybe I'm on the wrong track. Could you
take a
> look
> > at the config file to check the format or other possible issue
which
> might
> > be causing this issue?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 01 16:00:07 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for taking another look at it and for the more detailed
approach to
explaining how HiRA works which I've never used before. After reading
the
User's Guide, I got the impression that the categorical thresholds
(cat_thresh
array) were being used to compute the fractional coverage values
which are
evaluated as probability forecasts. (Ref the first para of the Section
on HiRA
Framework). So it was a little confusing how the prob_cat_thresh
settings
would fit in. I didn't see any example of what those threshold values
looked
like and came across something somewhere else in the User's Guide
which I
thought might be appropriate.
I need to understand what is meant by "meaningful" thresholds for each
variable. My thought was to just get it to run with one set of
thresholds fits
all approach and then hopefully gain an understanding of the impact of
using
the wrong or inappropriate threshold by looking at the output.
In your statement below: "we could have the code check to see if
cat_thresh in
the obs dictionary is non-empty", if obs = fcst why wouldn't you check
the
fcst dictionary? Just curious about why the obs dictionary would be
different/better for some reason.
>From your recommended HiRA settings, it seems to me that the
cov_thresh is the
critical setting which provides the probability bins which are applied
to the
probability forecasts. If the HiRA dictionary is best linked to the
specific
variable, then your fix to make things easier looks like a nice
improvement.
So, in reality, the cat_thresh array values are identical to the
prob_cat_thresh array values, correct?
If I'm correct in the previous sentence, I think I now understand why
the
prob_cat_thresh threshold values have to be tailored to each variable,
just as
the cat_thresh values are tailored to each variable (i.e >=270 K
doesn't make
sense for the RH variable. Is this what you mean by "meaningful"?
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, July 1, 2020 11:17 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Sorry for the confusion. I led you astray. The HiRA configuration
option
names are not entirely intuitive.
Please take a look at the README file starting on line 1418:
Caution-
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f5d2b0db9a37/met/data/config/README#L1418
I modified your hira entry like this:
hira = {
flag = TRUE; *// set to TRUE/false to turn on/off
this
logic*
width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
size and must be > 1*
vld_thresh = 1.0;
cov_thresh = [ ==0.25 ]; *// this defines the probability bins
to be
used. You only get to pick one setting here (==0.25 is short for
>=0,>=0.25,>=0.5,>=0.75,>=1.0)*
shape = SQUARE;
prob_cat_thresh = [ >290, >300 ];* // this defines the threshold(s)
for
which HiRA logic should be applied* }
So you just had things a bit mis-configured. The key thing to note
here is
that the hira.prob_cat_thresh setting defines the thresholds for which
HiRA
probabilities should be computed. This is different from the
cat_thresh
setting in the fcst and obs dictionaries. Those are used to compute
categorical counts and statistics and are currently separate.
It's nice that you don't have to use the same set of thresholds for
categorical stats and the HiRA logic... it's more configurable that
way.
However, I do see that it makes things a bit inconvenient. Since
you're
applying HiRA to many variables and each of those variables has
different
meaningful thresholds, you'd need to configure the HiRA dictionary
separately
for each and every variable.
That's what is meant by this comment... that the hira dictionary can
be
specified separately for each variable:
*// HiRA verification method// May be set separately in each
"obs.field"
entry*
So you'd need to copy/paste the HiRA dictionary into every single
obs.field
dictionary entry and modify the prob_cat_thresh thresholds
accordingly.
First, please make sure that you can in fact produce HiRA output.
Second, I
can think of a tweak to the logic that might make using HiRA easier.
How
about, if the hira.prob_cat_thresh array is an empty list, we could
have the
code check to see if cat_thresh in the obs dictionary is non-empty.
And if
that's non-empty then we'd simply set prob_cat_thresh = cat_thresh.
I imagine that's what you were expecting in the first place... that
Point-Stat
would just be using the thresholds from cat_thresh to define the HiRA
probability thresholds. It still is nice to have prob_cat_thresh be
configurable separately, but if left unset, defaulting to cat_thresh
seems
pretty reasonable to me. Do you think that would be a worthwhile
change?
Thanks,
John
On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> I forgot to attach the log file.
>
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, June 30, 2020 7:00 PM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> I edited the config file to use the shortcut and corrected the
> whitespace issue, but I get the same error.
> R/
> John
>
>
>
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, June 30, 2020 6:10 PM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> Thanks for taking a look, John. I'll give that a try. Appreciate the
> heads-up on a future error in the "desc".
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, June 30, 2020 5:11 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Please try using this shortcut:
> prob_cat_thresh = [ ==0.25 ];
>
> Does that do it?
>
> Also, your "desc" setting can't contain embedded whitespace. And
> Point-Stat should error out and tell you that.
>
> John
>
> On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I keep getting the same error when trying to run Point-Stat with
the
> > HiRA verification logic enabled. The error involves the
probability
> > field
> > thresholds:
> >
> > ERROR:check_prob_thresh() -> When verifying a probability field,
you
> > must select at least 3 thresholds beginning with 0.0 and ending
with 1.0.
> >
> > During troubleshooting, I've changed the settings for
> > prob_cat_thresh in the attached config file several times thinking
> > that the formatting of my settings was the issue. Maybe I'm on the
> > wrong track. Could you take a
> look
> > at the config file to check the format or other possible issue
which
> might
> > be causing this issue?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> > 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jul 01 16:47:21 2020
John,
Yes, that's exactly what I mean. ">290" makes sense for TMP in Kelvin
but
not RH or wind or precip. By having prob_cat_thresh = cat_thresh
(unless
its otherwise specified) you'd only need to set the cat_thresh once.
I'll need to write up this proposed change as a GitHub issue.
John
On Wed, Jul 1, 2020 at 4:00 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for taking another look at it and for the more detailed
approach to
> explaining how HiRA works which I've never used before. After
reading the
> User's Guide, I got the impression that the categorical thresholds
> (cat_thresh
> array) were being used to compute the fractional coverage values
which
> are
> evaluated as probability forecasts. (Ref the first para of the
Section on
> HiRA
> Framework). So it was a little confusing how the prob_cat_thresh
settings
> would fit in. I didn't see any example of what those threshold
values
> looked
> like and came across something somewhere else in the User's Guide
which I
> thought might be appropriate.
>
> I need to understand what is meant by "meaningful" thresholds for
each
> variable. My thought was to just get it to run with one set of
thresholds
> fits
> all approach and then hopefully gain an understanding of the impact
of
> using
> the wrong or inappropriate threshold by looking at the output.
>
> In your statement below: "we could have the code check to see if
> cat_thresh in
> the obs dictionary is non-empty", if obs = fcst why wouldn't you
check the
> fcst dictionary? Just curious about why the obs dictionary would be
> different/better for some reason.
>
> From your recommended HiRA settings, it seems to me that the
cov_thresh is
> the
> critical setting which provides the probability bins which are
applied to
> the
> probability forecasts. If the HiRA dictionary is best linked to the
> specific
> variable, then your fix to make things easier looks like a nice
> improvement.
> So, in reality, the cat_thresh array values are identical to the
> prob_cat_thresh array values, correct?
>
> If I'm correct in the previous sentence, I think I now understand
why the
> prob_cat_thresh threshold values have to be tailored to each
variable,
> just as
> the cat_thresh values are tailored to each variable (i.e >=270 K
doesn't
> make
> sense for the RH variable. Is this what you mean by "meaningful"?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, July 1, 2020 11:17 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the confusion. I led you astray. The HiRA configuration
option
> names are not entirely intuitive.
>
> Please take a look at the README file starting on line 1418:
> Caution-
>
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f5d2b0db9a37/met/data/config/README#L1418
>
> I modified your hira entry like this:
>
> hira = {
> flag = TRUE; *// set to TRUE/false to turn on/off
this
> logic*
> width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
> size and must be > 1*
> vld_thresh = 1.0;
> cov_thresh = [ ==0.25 ]; *// this defines the probability
bins to be
> used. You only get to pick one setting here (==0.25 is short for
> >=0,>=0.25,>=0.5,>=0.75,>=1.0)*
> shape = SQUARE;
> prob_cat_thresh = [ >290, >300 ];* // this defines the
threshold(s) for
> which HiRA logic should be applied* }
>
> So you just had things a bit mis-configured. The key thing to note
here is
> that the hira.prob_cat_thresh setting defines the thresholds for
which
> HiRA
> probabilities should be computed. This is different from the
cat_thresh
> setting in the fcst and obs dictionaries. Those are used to compute
> categorical counts and statistics and are currently separate.
>
> It's nice that you don't have to use the same set of thresholds for
> categorical stats and the HiRA logic... it's more configurable that
way.
> However, I do see that it makes things a bit inconvenient. Since
you're
> applying HiRA to many variables and each of those variables has
different
> meaningful thresholds, you'd need to configure the HiRA dictionary
> separately
> for each and every variable.
>
> That's what is meant by this comment... that the hira dictionary can
be
> specified separately for each variable:
>
> *// HiRA verification method// May be set separately in each
"obs.field"
> entry*
>
> So you'd need to copy/paste the HiRA dictionary into every single
> obs.field
> dictionary entry and modify the prob_cat_thresh thresholds
accordingly.
>
> First, please make sure that you can in fact produce HiRA output.
Second,
> I
> can think of a tweak to the logic that might make using HiRA easier.
How
> about, if the hira.prob_cat_thresh array is an empty list, we could
have
> the
> code check to see if cat_thresh in the obs dictionary is non-empty.
And if
> that's non-empty then we'd simply set prob_cat_thresh = cat_thresh.
>
> I imagine that's what you were expecting in the first place... that
> Point-Stat
> would just be using the thresholds from cat_thresh to define the
HiRA
> probability thresholds. It still is nice to have prob_cat_thresh be
> configurable separately, but if left unset, defaulting to cat_thresh
seems
> pretty reasonable to me. Do you think that would be a worthwhile
change?
>
> Thanks,
> John
>
> On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> >
> > I forgot to attach the log file.
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, June 30, 2020 7:00 PM
> > To: met_help at ucar.edu
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > I edited the config file to use the shortcut and corrected the
> > whitespace issue, but I get the same error.
> > R/
> > John
> >
> >
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, June 30, 2020 6:10 PM
> > To: met_help at ucar.edu
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > Thanks for taking a look, John. I'll give that a try. Appreciate
the
> > heads-up on a future error in the "desc".
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, June 30, 2020 5:11 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Please try using this shortcut:
> > prob_cat_thresh = [ ==0.25 ];
> >
> > Does that do it?
> >
> > Also, your "desc" setting can't contain embedded whitespace. And
> > Point-Stat should error out and tell you that.
> >
> > John
> >
> > On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I keep getting the same error when trying to run Point-Stat with
the
> > > HiRA verification logic enabled. The error involves the
probability
> > > field
> > > thresholds:
> > >
> > > ERROR:check_prob_thresh() -> When verifying a probability field,
you
> > > must select at least 3 thresholds beginning with 0.0 and ending
with
> 1.0.
> > >
> > > During troubleshooting, I've changed the settings for
> > > prob_cat_thresh in the attached config file several times
thinking
> > > that the formatting of my settings was the issue. Maybe I'm on
the
> > > wrong track. Could you take a
> > look
> > > at the config file to check the format or other possible issue
which
> > might
> > > be causing this issue?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 01 18:09:53 2020
CLASSIFICATION: UNCLASSIFIED
Thanks for confirming this for me. The recommended sw change looks
good to me.
I'll test the chg to the HiRA dictionary and the fcst/ob dictionaries
which
you provided tomorrow and let you know.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, July 1, 2020 4:47 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat
error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Yes, that's exactly what I mean. ">290" makes sense for TMP in Kelvin
but not
RH or wind or precip. By having prob_cat_thresh = cat_thresh (unless
its
otherwise specified) you'd only need to set the cat_thresh once.
I'll need to write up this proposed change as a GitHub issue.
John
On Wed, Jul 1, 2020 at 4:00 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for taking another look at it and for the more detailed
> approach to explaining how HiRA works which I've never used before.
> After reading the User's Guide, I got the impression that the
> categorical thresholds (cat_thresh
> array) were being used to compute the fractional coverage values
> which are evaluated as probability forecasts. (Ref the first para of
> the Section on HiRA Framework). So it was a little confusing how the
> prob_cat_thresh settings would fit in. I didn't see any example of
> what those threshold values looked like and came across something
> somewhere else in the User's Guide which I thought might be
> appropriate.
>
> I need to understand what is meant by "meaningful" thresholds for
each
> variable. My thought was to just get it to run with one set of
> thresholds fits all approach and then hopefully gain an
understanding
> of the impact of using the wrong or inappropriate threshold by
looking
> at the output.
>
> In your statement below: "we could have the code check to see if
> cat_thresh in the obs dictionary is non-empty", if obs = fcst why
> wouldn't you check the fcst dictionary? Just curious about why the
obs
> dictionary would be different/better for some reason.
>
> From your recommended HiRA settings, it seems to me that the
> cov_thresh is the critical setting which provides the probability
bins
> which are applied to the probability forecasts. If the HiRA
dictionary
> is best linked to the specific variable, then your fix to make
things
> easier looks like a nice improvement.
> So, in reality, the cat_thresh array values are identical to the
> prob_cat_thresh array values, correct?
>
> If I'm correct in the previous sentence, I think I now understand
why
> the prob_cat_thresh threshold values have to be tailored to each
> variable, just as the cat_thresh values are tailored to each
variable
> (i.e >=270 K doesn't make sense for the RH variable. Is this what
you
> mean by "meaningful"?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, July 1, 2020 11:17 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the confusion. I led you astray. The HiRA configuration
> option names are not entirely intuitive.
>
> Please take a look at the README file starting on line 1418:
> Caution-
> Caution-
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f
> 5d2b0db9a37/met/data/config/README#L1418
>
> I modified your hira entry like this:
>
> hira = {
> flag = TRUE; *// set to TRUE/false to turn on/off
this
> logic*
> width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
> size and must be > 1*
> vld_thresh = 1.0;
> cov_thresh = [ ==0.25 ]; *// this defines the probability
bins to be
> used. You only get to pick one setting here (==0.25 is short for
> >=0,>=0.25,>=0.5,>=0.75,>=1.0)*
> shape = SQUARE;
> prob_cat_thresh = [ >290, >300 ];* // this defines the
threshold(s)
> for which HiRA logic should be applied* }
>
> So you just had things a bit mis-configured. The key thing to note
> here is that the hira.prob_cat_thresh setting defines the thresholds
> for which HiRA probabilities should be computed. This is different
> from the cat_thresh setting in the fcst and obs dictionaries. Those
> are used to compute categorical counts and statistics and are
> currently separate.
>
> It's nice that you don't have to use the same set of thresholds for
> categorical stats and the HiRA logic... it's more configurable that
way.
> However, I do see that it makes things a bit inconvenient. Since
> you're applying HiRA to many variables and each of those variables
has
> different meaningful thresholds, you'd need to configure the HiRA
> dictionary separately for each and every variable.
>
> That's what is meant by this comment... that the hira dictionary can
> be specified separately for each variable:
>
> *// HiRA verification method// May be set separately in each
"obs.field"
> entry*
>
> So you'd need to copy/paste the HiRA dictionary into every single
> obs.field dictionary entry and modify the prob_cat_thresh thresholds
> accordingly.
>
> First, please make sure that you can in fact produce HiRA output.
> Second, I can think of a tweak to the logic that might make using
HiRA
> easier. How about, if the hira.prob_cat_thresh array is an empty
list,
> we could have the code check to see if cat_thresh in the obs
> dictionary is non-empty. And if that's non-empty then we'd simply
set
> prob_cat_thresh = cat_thresh.
>
> I imagine that's what you were expecting in the first place... that
> Point-Stat would just be using the thresholds from cat_thresh to
> define the HiRA probability thresholds. It still is nice to have
> prob_cat_thresh be configurable separately, but if left unset,
> defaulting to cat_thresh seems pretty reasonable to me. Do you think
> that would be a worthwhile change?
>
> Thanks,
> John
>
> On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > 790 >
> >
> > I forgot to attach the log file.
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, June 30, 2020 7:00 PM
> > To: met_help at ucar.edu
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > I edited the config file to use the shortcut and corrected the
> > whitespace issue, but I get the same error.
> > R/
> > John
> >
> >
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, June 30, 2020 6:10 PM
> > To: met_help at ucar.edu
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > Thanks for taking a look, John. I'll give that a try. Appreciate
the
> > heads-up on a future error in the "desc".
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, June 30, 2020 5:11 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Please try using this shortcut:
> > prob_cat_thresh = [ ==0.25 ];
> >
> > Does that do it?
> >
> > Also, your "desc" setting can't contain embedded whitespace. And
> > Point-Stat should error out and tell you that.
> >
> > John
> >
> > On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > 790 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I keep getting the same error when trying to run Point-Stat with
> > > the HiRA verification logic enabled. The error involves the
> > > probability field
> > > thresholds:
> > >
> > > ERROR:check_prob_thresh() -> When verifying a probability field,
> > > you must select at least 3 thresholds beginning with 0.0 and
> > > ending with
> 1.0.
> > >
> > > During troubleshooting, I've changed the settings for
> > > prob_cat_thresh in the attached config file several times
thinking
> > > that the formatting of my settings was the issue. Maybe I'm on
the
> > > wrong track. Could you take a
> > look
> > > at the config file to check the format or other possible issue
> > > which
> > might
> > > be causing this issue?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jul 01 18:22:26 2020
Sounds good. Thanks.
FYI, here's the GitHub issue I wrote up for this change:
https://github.com/NCAR/MET/issues/1400
I tentatively assigned it to the next release, but can't guarantee
we'll
get it in there.
John
On Wed, Jul 1, 2020 at 6:10 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for confirming this for me. The recommended sw change looks
good to
> me.
> I'll test the chg to the HiRA dictionary and the fcst/ob
dictionaries
> which
> you provided tomorrow and let you know.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, July 1, 2020 4:47 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that's exactly what I mean. ">290" makes sense for TMP in
Kelvin but
> not
> RH or wind or precip. By having prob_cat_thresh = cat_thresh (unless
its
> otherwise specified) you'd only need to set the cat_thresh once.
>
> I'll need to write up this proposed change as a GitHub issue.
>
> John
>
> On Wed, Jul 1, 2020 at 4:00 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for taking another look at it and for the more detailed
> > approach to explaining how HiRA works which I've never used
before.
> > After reading the User's Guide, I got the impression that the
> > categorical thresholds (cat_thresh
> > array) were being used to compute the fractional coverage values
> > which are evaluated as probability forecasts. (Ref the first para
of
> > the Section on HiRA Framework). So it was a little confusing how
the
> > prob_cat_thresh settings would fit in. I didn't see any example of
> > what those threshold values looked like and came across something
> > somewhere else in the User's Guide which I thought might be
> > appropriate.
> >
> > I need to understand what is meant by "meaningful" thresholds for
each
> > variable. My thought was to just get it to run with one set of
> > thresholds fits all approach and then hopefully gain an
understanding
> > of the impact of using the wrong or inappropriate threshold by
looking
> > at the output.
> >
> > In your statement below: "we could have the code check to see if
> > cat_thresh in the obs dictionary is non-empty", if obs = fcst why
> > wouldn't you check the fcst dictionary? Just curious about why the
obs
> > dictionary would be different/better for some reason.
> >
> > From your recommended HiRA settings, it seems to me that the
> > cov_thresh is the critical setting which provides the probability
bins
> > which are applied to the probability forecasts. If the HiRA
dictionary
> > is best linked to the specific variable, then your fix to make
things
> > easier looks like a nice improvement.
> > So, in reality, the cat_thresh array values are identical to the
> > prob_cat_thresh array values, correct?
> >
> > If I'm correct in the previous sentence, I think I now understand
why
> > the prob_cat_thresh threshold values have to be tailored to each
> > variable, just as the cat_thresh values are tailored to each
variable
> > (i.e >=270 K doesn't make sense for the RH variable. Is this what
you
> > mean by "meaningful"?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Wednesday, July 1, 2020 11:17 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry for the confusion. I led you astray. The HiRA configuration
> > option names are not entirely intuitive.
> >
> > Please take a look at the README file starting on line 1418:
> > Caution-
> > Caution-
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f
> > 5d2b0db9a37/met/data/config/README#L1418
> >
> > I modified your hira entry like this:
> >
> > hira = {
> > flag = TRUE; *// set to TRUE/false to turn
on/off this
> > logic*
> > width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
> > size and must be > 1*
> > vld_thresh = 1.0;
> > cov_thresh = [ ==0.25 ]; *// this defines the probability
bins to
> be
> > used. You only get to pick one setting here (==0.25 is short for
> > >=0,>=0.25,>=0.5,>=0.75,>=1.0)*
> > shape = SQUARE;
> > prob_cat_thresh = [ >290, >300 ];* // this defines the
threshold(s)
> > for which HiRA logic should be applied* }
> >
> > So you just had things a bit mis-configured. The key thing to note
> > here is that the hira.prob_cat_thresh setting defines the
thresholds
> > for which HiRA probabilities should be computed. This is different
> > from the cat_thresh setting in the fcst and obs dictionaries.
Those
> > are used to compute categorical counts and statistics and are
> > currently separate.
> >
> > It's nice that you don't have to use the same set of thresholds
for
> > categorical stats and the HiRA logic... it's more configurable
that way.
> > However, I do see that it makes things a bit inconvenient. Since
> > you're applying HiRA to many variables and each of those variables
has
> > different meaningful thresholds, you'd need to configure the HiRA
> > dictionary separately for each and every variable.
> >
> > That's what is meant by this comment... that the hira dictionary
can
> > be specified separately for each variable:
> >
> > *// HiRA verification method// May be set separately in each
"obs.field"
> > entry*
> >
> > So you'd need to copy/paste the HiRA dictionary into every single
> > obs.field dictionary entry and modify the prob_cat_thresh
thresholds
> > accordingly.
> >
> > First, please make sure that you can in fact produce HiRA output.
> > Second, I can think of a tweak to the logic that might make using
HiRA
> > easier. How about, if the hira.prob_cat_thresh array is an empty
list,
> > we could have the code check to see if cat_thresh in the obs
> > dictionary is non-empty. And if that's non-empty then we'd simply
set
> > prob_cat_thresh = cat_thresh.
> >
> > I imagine that's what you were expecting in the first place...
that
> > Point-Stat would just be using the thresholds from cat_thresh to
> > define the HiRA probability thresholds. It still is nice to have
> > prob_cat_thresh be configurable separately, but if left unset,
> > defaulting to cat_thresh seems pretty reasonable to me. Do you
think
> > that would be a worthwhile change?
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > > 790 >
> > >
> > > I forgot to attach the log file.
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, June 30, 2020 7:00 PM
> > > To: met_help at ucar.edu
> > > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > I edited the config file to use the shortcut and corrected the
> > > whitespace issue, but I get the same error.
> > > R/
> > > John
> > >
> > >
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, June 30, 2020 6:10 PM
> > > To: met_help at ucar.edu
> > > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > Thanks for taking a look, John. I'll give that a try. Appreciate
the
> > > heads-up on a future error in the "desc".
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, June 30, 2020 5:11 PM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Please try using this shortcut:
> > > prob_cat_thresh = [ ==0.25 ];
> > >
> > > Does that do it?
> > >
> > > Also, your "desc" setting can't contain embedded whitespace. And
> > > Point-Stat should error out and tell you that.
> > >
> > > John
> > >
> > > On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > > 790 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I keep getting the same error when trying to run Point-Stat
with
> > > > the HiRA verification logic enabled. The error involves the
> > > > probability field
> > > > thresholds:
> > > >
> > > > ERROR:check_prob_thresh() -> When verifying a probability
field,
> > > > you must select at least 3 thresholds beginning with 0.0 and
> > > > ending with
> > 1.0.
> > > >
> > > > During troubleshooting, I've changed the settings for
> > > > prob_cat_thresh in the attached config file several times
thinking
> > > > that the formatting of my settings was the issue. Maybe I'm on
the
> > > > wrong track. Could you take a
> > > look
> > > > at the config file to check the format or other possible issue
> > > > which
> > > might
> > > > be causing this issue?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > Office:(575) 678-2004 DSN 258-2004
> > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > > 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0 Point-Stat error (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 01 18:26:25 2020
Thanks for creating the sw chg request and for the link.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, July 01, 2020 6:22 PM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
Point-Stat error (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Sounds good. Thanks.
FYI, here's the GitHub issue I wrote up for this change:
Caution-https://github.com/NCAR/MET/issues/1400
I tentatively assigned it to the next release, but can't guarantee
we'll
get it in there.
John
On Wed, Jul 1, 2020 at 6:10 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for confirming this for me. The recommended sw change looks
good to
> me.
> I'll test the chg to the HiRA dictionary and the fcst/ob
dictionaries
> which
> you provided tomorrow and let you know.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, July 1, 2020 4:47 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> Point-Stat
> error (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that's exactly what I mean. ">290" makes sense for TMP in
Kelvin but
> not
> RH or wind or precip. By having prob_cat_thresh = cat_thresh (unless
its
> otherwise specified) you'd only need to set the cat_thresh once.
>
> I'll need to write up this proposed change as a GitHub issue.
>
> John
>
> On Wed, Jul 1, 2020 at 4:00 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95790 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for taking another look at it and for the more detailed
> > approach to explaining how HiRA works which I've never used
before.
> > After reading the User's Guide, I got the impression that the
> > categorical thresholds (cat_thresh
> > array) were being used to compute the fractional coverage values
> > which are evaluated as probability forecasts. (Ref the first para
of
> > the Section on HiRA Framework). So it was a little confusing how
the
> > prob_cat_thresh settings would fit in. I didn't see any example of
> > what those threshold values looked like and came across something
> > somewhere else in the User's Guide which I thought might be
> > appropriate.
> >
> > I need to understand what is meant by "meaningful" thresholds for
each
> > variable. My thought was to just get it to run with one set of
> > thresholds fits all approach and then hopefully gain an
understanding
> > of the impact of using the wrong or inappropriate threshold by
looking
> > at the output.
> >
> > In your statement below: "we could have the code check to see if
> > cat_thresh in the obs dictionary is non-empty", if obs = fcst why
> > wouldn't you check the fcst dictionary? Just curious about why the
obs
> > dictionary would be different/better for some reason.
> >
> > From your recommended HiRA settings, it seems to me that the
> > cov_thresh is the critical setting which provides the probability
bins
> > which are applied to the probability forecasts. If the HiRA
dictionary
> > is best linked to the specific variable, then your fix to make
things
> > easier looks like a nice improvement.
> > So, in reality, the cat_thresh array values are identical to the
> > prob_cat_thresh array values, correct?
> >
> > If I'm correct in the previous sentence, I think I now understand
why
> > the prob_cat_thresh threshold values have to be tailored to each
> > variable, just as the cat_thresh values are tailored to each
variable
> > (i.e >=270 K doesn't make sense for the RH variable. Is this what
you
> > mean by "meaningful"?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Wednesday, July 1, 2020 11:17 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > Point-Stat error (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry for the confusion. I led you astray. The HiRA configuration
> > option names are not entirely intuitive.
> >
> > Please take a look at the README file starting on line 1418:
> > Caution-
> > Caution-Caution-
https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f
> > 5d2b0db9a37/met/data/config/README#L1418
> >
> > I modified your hira entry like this:
> >
> > hira = {
> > flag = TRUE; *// set to TRUE/false to turn
on/off this
> > logic*
> > width = [ 3, 5, 7, 9, 11 ]; *// defines the
neighborhood
> > size and must be > 1*
> > vld_thresh = 1.0;
> > cov_thresh = [ ==0.25 ]; *// this defines the probability
bins to
> be
> > used. You only get to pick one setting here (==0.25 is short for
> > >=0,>=0.25,>=0.5,>=0.75,>=1.0)*
> > shape = SQUARE;
> > prob_cat_thresh = [ >290, >300 ];* // this defines the
threshold(s)
> > for which HiRA logic should be applied* }
> >
> > So you just had things a bit mis-configured. The key thing to note
> > here is that the hira.prob_cat_thresh setting defines the
thresholds
> > for which HiRA probabilities should be computed. This is different
> > from the cat_thresh setting in the fcst and obs dictionaries.
Those
> > are used to compute categorical counts and statistics and are
> > currently separate.
> >
> > It's nice that you don't have to use the same set of thresholds
for
> > categorical stats and the HiRA logic... it's more configurable
that way.
> > However, I do see that it makes things a bit inconvenient. Since
> > you're applying HiRA to many variables and each of those variables
has
> > different meaningful thresholds, you'd need to configure the HiRA
> > dictionary separately for each and every variable.
> >
> > That's what is meant by this comment... that the hira dictionary
can
> > be specified separately for each variable:
> >
> > *// HiRA verification method// May be set separately in each
"obs.field"
> > entry*
> >
> > So you'd need to copy/paste the HiRA dictionary into every single
> > obs.field dictionary entry and modify the prob_cat_thresh
thresholds
> > accordingly.
> >
> > First, please make sure that you can in fact produce HiRA output.
> > Second, I can think of a tweak to the logic that might make using
HiRA
> > easier. How about, if the hira.prob_cat_thresh array is an empty
list,
> > we could have the code check to see if cat_thresh in the obs
> > dictionary is non-empty. And if that's non-empty then we'd simply
set
> > prob_cat_thresh = cat_thresh.
> >
> > I imagine that's what you were expecting in the first place...
that
> > Point-Stat would just be using the thresholds from cat_thresh to
> > define the HiRA probability thresholds. It still is nice to have
> > prob_cat_thresh be configurable separately, but if left unset,
> > defaulting to cat_thresh seems pretty reasonable to me. Do you
think
> > that would be a worthwhile change?
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020 at 7:12 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > > 790 >
> > >
> > > I forgot to attach the log file.
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, June 30, 2020 7:00 PM
> > > To: met_help at ucar.edu
> > > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > I edited the config file to use the shortcut and corrected the
> > > whitespace issue, but I get the same error.
> > > R/
> > > John
> > >
> > >
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, June 30, 2020 6:10 PM
> > > To: met_help at ucar.edu
> > > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET
V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > Thanks for taking a look, John. I'll give that a try. Appreciate
the
> > > heads-up on a future error in the "desc".
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, June 30, 2020 5:11 PM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #95790] MET V9.0
> > > Point-Stat error (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Please try using this shortcut:
> > > prob_cat_thresh = [ ==0.25 ];
> > >
> > > Does that do it?
> > >
> > > Also, your "desc" setting can't contain embedded whitespace. And
> > > Point-Stat should error out and tell you that.
> > >
> > > John
> > >
> > > On Tue, Jun 30, 2020 at 4:57 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jun 30 16:56:45 2020: Request 95790 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET V9.0 Point-Stat error (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95
> > > 790 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I keep getting the same error when trying to run Point-Stat
with
> > > > the HiRA verification logic enabled. The error involves the
> > > > probability field
> > > > thresholds:
> > > >
> > > > ERROR:check_prob_thresh() -> When verifying a probability
field,
> > > > you must select at least 3 thresholds beginning with 0.0 and
> > > > ending with
> > 1.0.
> > > >
> > > > During troubleshooting, I've changed the settings for
> > > > prob_cat_thresh in the attached config file several times
thinking
> > > > that the formatting of my settings was the issue. Maybe I'm on
the
> > > > wrong track. Could you take a
> > > look
> > > > at the config file to check the format or other possible issue
> > > > which
> > > might
> > > > be causing this issue?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > Office:(575) 678-2004 DSN 258-2004
> > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > > 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
More information about the Met_help
mailing list