[Met_help] [rt.rap.ucar.edu #78584] History for questions about QC for observation
John Halley Gotway via RT
met_help at ucar.edu
Wed Nov 9 12:20:12 MST 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear Mr. or Mrs.,
Thanks in advance for your help.
I'm wondering what methods can be used with MET to screen the observations
in the verification process.
I found obs_quality can help if the flag parameters are in the obs file.
However, if there are no such flags in my obs and I know the definitions of
missing values or other values like -99 assigned with special meaning,
how can I make those info with MET?
Or , for simplification, if I want to use the obs in a range like -10 <
obs<70.
Can I or ,if I can, How can I invoke such a constraint in the MET process.
Your help is appreciated.
Best,
Ting
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Thu Nov 03 14:31:15 2016
Ting,
OK, there are two things I'd like to point out regarding this issue.
First, the NetCDF point observation files that Point-Stat reads
contain a
slot for quality control information. Specifically, the NetCDF output
from
ascii2nc, madis2nc, and pb2nc contain a variable named "obs_qty" which
is a
string containing quality control info.
If you're running ascii2nc, it's your job to format the input ascii
data
the expected way. If you have quality control info, it should go in
column
10, named "QC_String" in the ascii2nc usage statement. MET does not
do any
QC for you, but it can use the QC strings you've set.
In the Point-Stat config file, you can set the "obs_quality" entry to
a
list of one or more quality control strings to be used. If set, only
observations whose quality control value appears in that list will be
used.
Second, you asked if there's a way to restrict the observations used
by
filtering their values, such as -10<obs<70. The answer is yes, at
least
for continuous statistics, like RMSE and ME. We call this
"conditional
continuous verification" since we are conditioning the analysis on the
input values.
We set this up using the "cnt_thresh" and "cnt_logic" settings in the
Point-Stat config file, such as:
fcst = {
field = [
{ name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
];
}
obs = {
field = [
{ name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-10&&le70 ];
cnt_logic = INTERSECTION; }
];
}
Note that in the forecast I'm setting cnt_thresh to NA, meaning that
no
filtering criteria will be applied to the forecast value. But in the
observation I used "ge-10&&le70" to define the desired range of
observation
values.
You can list as many cnt_thresh values as you'd like in the list but
the
number of fcst settings must match the number of obs settings. For
example...
fcst...
cnt_thresh = [ NA, NA, NA, NA, NA ];
obs...
cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
The above settings will compute continuous statistics 5 ways... using
(1)
no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and (5) obs >=
30
Hope that helps. Just let me know if you have more questions.
Thanks,
John
On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> Transaction: Ticket created by ting.lei at noaa.gov
> Queue: met_help
> Subject: questions about QC for observation
> Owner: Nobody
> Requestors: ting.lei at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
>
> Dear Mr. or Mrs.,
> Thanks in advance for your help.
> I'm wondering what methods can be used with MET to screen the
observations
> in the verification process.
> I found obs_quality can help if the flag parameters are in the obs
file.
> However, if there are no such flags in my obs and I know the
definitions of
> missing values or other values like -99 assigned with special
meaning,
> how can I make those info with MET?
> Or , for simplification, if I want to use the obs in a range like
-10 <
> obs<70.
> Can I or ,if I can, How can I invoke such a constraint in the MET
process.
> Your help is appreciated.
> Best,
> Ting
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Thu Nov 03 14:36:41 2016
John,
Thanks a lot.
I need more time to study the first part about QC in your explanation.
However, it is great that the cnt_thresh can be used. It can
definitely
meet my requirement for being now.
Just a clarification is great. This function is working for both
Met-point and Met-grid, right?
Thanks.
Ting
On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Ting,
>
> OK, there are two things I'd like to point out regarding this issue.
>
> First, the NetCDF point observation files that Point-Stat reads
contain a
> slot for quality control information. Specifically, the NetCDF
output from
> ascii2nc, madis2nc, and pb2nc contain a variable named "obs_qty"
which is a
> string containing quality control info.
>
> If you're running ascii2nc, it's your job to format the input ascii
data
> the expected way. If you have quality control info, it should go in
column
> 10, named "QC_String" in the ascii2nc usage statement. MET does not
do any
> QC for you, but it can use the QC strings you've set.
>
> In the Point-Stat config file, you can set the "obs_quality" entry
to a
> list of one or more quality control strings to be used. If set,
only
> observations whose quality control value appears in that list will
be used.
>
> Second, you asked if there's a way to restrict the observations used
by
> filtering their values, such as -10<obs<70. The answer is yes, at
least
> for continuous statistics, like RMSE and ME. We call this
"conditional
> continuous verification" since we are conditioning the analysis on
the
> input values.
>
> We set this up using the "cnt_thresh" and "cnt_logic" settings in
the
> Point-Stat config file, such as:
>
> fcst = {
>
> field = [
> { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
> ];
>
> }
>
> obs = {
>
> field = [
> { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-10&&le70 ];
> cnt_logic = INTERSECTION; }
> ];
>
> }
>
> Note that in the forecast I'm setting cnt_thresh to NA, meaning that
no
> filtering criteria will be applied to the forecast value. But in
the
> observation I used "ge-10&&le70" to define the desired range of
observation
> values.
>
> You can list as many cnt_thresh values as you'd like in the list but
the
> number of fcst settings must match the number of obs settings. For
> example...
>
> fcst...
> cnt_thresh = [ NA, NA, NA, NA, NA ];
>
> obs...
> cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
>
> The above settings will compute continuous statistics 5 ways...
using (1)
> no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and (5) obs
>= 30
>
> Hope that helps. Just let me know if you have more questions.
>
> Thanks,
> John
>
>
> On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> > Transaction: Ticket created by ting.lei at noaa.gov
> > Queue: met_help
> > Subject: questions about QC for observation
> > Owner: Nobody
> > Requestors: ting.lei at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >
> >
> > Dear Mr. or Mrs.,
> > Thanks in advance for your help.
> > I'm wondering what methods can be used with MET to screen the
> observations
> > in the verification process.
> > I found obs_quality can help if the flag parameters are in the obs
file.
> > However, if there are no such flags in my obs and I know the
definitions
> of
> > missing values or other values like -99 assigned with special
meaning,
> > how can I make those info with MET?
> > Or , for simplification, if I want to use the obs in a range like
-10 <
> > obs<70.
> > Can I or ,if I can, How can I invoke such a constraint in the MET
> process.
> > Your help is appreciated.
> > Best,
> > Ting
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Thu Nov 03 15:12:33 2016
Ting,
Yes, the cnt_thresh filtering option works for both Grid-Stat and
Point-Stat.
The fcst -> cnt_thresh option filters the forecast values. The obs ->
cnt_thresh option filters the observation values. And the cnt_logic
option
can be set to UNION, INTERSECTION, or SYMDIFF to specify how the logic
for
how the forecast and observation thresholds should be combined.
For example... suppose you've defined cnt_thresh for temperature as
>273.
cnt_logic = UNION means the forecast **or** observation value is >
273.
cnt_logic = INTERSECTION means the forecast **and** observation values
are
>273.
cnt_logic = SYMDIFF means the that one is >273 and the other is not.
Thanks,
John
On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
> John,
> Thanks a lot.
> I need more time to study the first part about QC in your
explanation.
> However, it is great that the cnt_thresh can be used. It can
definitely
> meet my requirement for being now.
> Just a clarification is great. This function is working for both
> Met-point and Met-grid, right?
> Thanks.
> Ting
>
> On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Ting,
> >
> > OK, there are two things I'd like to point out regarding this
issue.
> >
> > First, the NetCDF point observation files that Point-Stat reads
contain a
> > slot for quality control information. Specifically, the NetCDF
output
> from
> > ascii2nc, madis2nc, and pb2nc contain a variable named "obs_qty"
which
> is a
> > string containing quality control info.
> >
> > If you're running ascii2nc, it's your job to format the input
ascii data
> > the expected way. If you have quality control info, it should go
in
> column
> > 10, named "QC_String" in the ascii2nc usage statement. MET does
not do
> any
> > QC for you, but it can use the QC strings you've set.
> >
> > In the Point-Stat config file, you can set the "obs_quality" entry
to a
> > list of one or more quality control strings to be used. If set,
only
> > observations whose quality control value appears in that list will
be
> used.
> >
> > Second, you asked if there's a way to restrict the observations
used by
> > filtering their values, such as -10<obs<70. The answer is yes, at
least
> > for continuous statistics, like RMSE and ME. We call this
"conditional
> > continuous verification" since we are conditioning the analysis on
the
> > input values.
> >
> > We set this up using the "cnt_thresh" and "cnt_logic" settings in
the
> > Point-Stat config file, such as:
> >
> > fcst = {
> >
> > field = [
> > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
> > ];
> >
> > }
> >
> > obs = {
> >
> > field = [
> > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-10&&le70 ];
> > cnt_logic = INTERSECTION; }
> > ];
> >
> > }
> >
> > Note that in the forecast I'm setting cnt_thresh to NA, meaning
that no
> > filtering criteria will be applied to the forecast value. But in
the
> > observation I used "ge-10&&le70" to define the desired range of
> observation
> > values.
> >
> > You can list as many cnt_thresh values as you'd like in the list
but the
> > number of fcst settings must match the number of obs settings.
For
> > example...
> >
> > fcst...
> > cnt_thresh = [ NA, NA, NA, NA, NA ];
> >
> > obs...
> > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> >
> > The above settings will compute continuous statistics 5 ways...
using (1)
> > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and (5)
obs >= 30
> >
> > Hope that helps. Just let me know if you have more questions.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> > > Transaction: Ticket created by ting.lei at noaa.gov
> > > Queue: met_help
> > > Subject: questions about QC for observation
> > > Owner: Nobody
> > > Requestors: ting.lei at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
> >
> > >
> > >
> > > Dear Mr. or Mrs.,
> > > Thanks in advance for your help.
> > > I'm wondering what methods can be used with MET to screen the
> > observations
> > > in the verification process.
> > > I found obs_quality can help if the flag parameters are in the
obs
> file.
> > > However, if there are no such flags in my obs and I know the
> definitions
> > of
> > > missing values or other values like -99 assigned with special
meaning,
> > > how can I make those info with MET?
> > > Or , for simplification, if I want to use the obs in a range
like -10 <
> > > obs<70.
> > > Can I or ,if I can, How can I invoke such a constraint in the
MET
> > process.
> > > Your help is appreciated.
> > > Best,
> > > Ting
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Thu Nov 03 16:14:07 2016
John,
Thanks again for your explanation.
That is great.
My question in this email has been perfectly answered.
Best Wishes,
Ting
On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Ting,
>
> Yes, the cnt_thresh filtering option works for both Grid-Stat and
> Point-Stat.
>
> The fcst -> cnt_thresh option filters the forecast values. The obs
->
> cnt_thresh option filters the observation values. And the cnt_logic
option
> can be set to UNION, INTERSECTION, or SYMDIFF to specify how the
logic for
> how the forecast and observation thresholds should be combined.
>
> For example... suppose you've defined cnt_thresh for temperature as
>273.
> cnt_logic = UNION means the forecast **or** observation value is >
273.
> cnt_logic = INTERSECTION means the forecast **and** observation
values are
> >273.
> cnt_logic = SYMDIFF means the that one is >273 and the other is not.
>
> Thanks,
> John
>
> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >
> > John,
> > Thanks a lot.
> > I need more time to study the first part about QC in your
explanation.
> > However, it is great that the cnt_thresh can be used. It can
definitely
> > meet my requirement for being now.
> > Just a clarification is great. This function is working for both
> > Met-point and Met-grid, right?
> > Thanks.
> > Ting
> >
> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Ting,
> > >
> > > OK, there are two things I'd like to point out regarding this
issue.
> > >
> > > First, the NetCDF point observation files that Point-Stat reads
> contain a
> > > slot for quality control information. Specifically, the NetCDF
output
> > from
> > > ascii2nc, madis2nc, and pb2nc contain a variable named "obs_qty"
which
> > is a
> > > string containing quality control info.
> > >
> > > If you're running ascii2nc, it's your job to format the input
ascii
> data
> > > the expected way. If you have quality control info, it should
go in
> > column
> > > 10, named "QC_String" in the ascii2nc usage statement. MET does
not do
> > any
> > > QC for you, but it can use the QC strings you've set.
> > >
> > > In the Point-Stat config file, you can set the "obs_quality"
entry to a
> > > list of one or more quality control strings to be used. If set,
only
> > > observations whose quality control value appears in that list
will be
> > used.
> > >
> > > Second, you asked if there's a way to restrict the observations
used by
> > > filtering their values, such as -10<obs<70. The answer is yes,
at
> least
> > > for continuous statistics, like RMSE and ME. We call this
"conditional
> > > continuous verification" since we are conditioning the analysis
on the
> > > input values.
> > >
> > > We set this up using the "cnt_thresh" and "cnt_logic" settings
in the
> > > Point-Stat config file, such as:
> > >
> > > fcst = {
> > >
> > > field = [
> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
> > > ];
> > >
> > > }
> > >
> > > obs = {
> > >
> > > field = [
> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-10&&le70
];
> > > cnt_logic = INTERSECTION; }
> > > ];
> > >
> > > }
> > >
> > > Note that in the forecast I'm setting cnt_thresh to NA, meaning
that no
> > > filtering criteria will be applied to the forecast value. But
in the
> > > observation I used "ge-10&&le70" to define the desired range of
> > observation
> > > values.
> > >
> > > You can list as many cnt_thresh values as you'd like in the list
but
> the
> > > number of fcst settings must match the number of obs settings.
For
> > > example...
> > >
> > > fcst...
> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > >
> > > obs...
> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > >
> > > The above settings will compute continuous statistics 5 ways...
using
> (1)
> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and (5)
obs >=
> 30
> > >
> > > Hope that helps. Just let me know if you have more questions.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > > > Queue: met_help
> > > > Subject: questions about QC for observation
> > > > Owner: Nobody
> > > > Requestors: ting.lei at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=78584
> > >
> > > >
> > > >
> > > > Dear Mr. or Mrs.,
> > > > Thanks in advance for your help.
> > > > I'm wondering what methods can be used with MET to screen the
> > > observations
> > > > in the verification process.
> > > > I found obs_quality can help if the flag parameters are in the
obs
> > file.
> > > > However, if there are no such flags in my obs and I know the
> > definitions
> > > of
> > > > missing values or other values like -99 assigned with special
> meaning,
> > > > how can I make those info with MET?
> > > > Or , for simplification, if I want to use the obs in a range
like
> -10 <
> > > > obs<70.
> > > > Can I or ,if I can, How can I invoke such a constraint in the
MET
> > > process.
> > > > Your help is appreciated.
> > > > Best,
> > > > Ting
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Tue Nov 08 11:03:17 2016
Hi, John,
Sorry for re-openning this ticket.
I still need some further clarifications.
When using the cnt_thresh to choose the obs/fcst in a range to be used
in
the verification for continuous variables,
will this control those for the categorical verification like CAC?
For example, the cat_thresh will control how the obs/fcst will be
verified
in several categories . Will the cnt_thresh take effect?
Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this make MET
only do
verification like CAC over the whole obs pool with values between -10
and
70 while calculate them in the categories decided by the cat_thresh?
Your further clarification is appreciated.
Ting
On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate
<ting.lei at noaa.gov
> wrote:
> John,
> Thanks again for your explanation.
> That is great.
> My question in this email has been perfectly answered.
> Best Wishes,
> Ting
>
> On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ting,
>>
>> Yes, the cnt_thresh filtering option works for both Grid-Stat and
>> Point-Stat.
>>
>> The fcst -> cnt_thresh option filters the forecast values. The obs
->
>> cnt_thresh option filters the observation values. And the
cnt_logic
>> option
>> can be set to UNION, INTERSECTION, or SYMDIFF to specify how the
logic for
>> how the forecast and observation thresholds should be combined.
>>
>> For example... suppose you've defined cnt_thresh for temperature as
>273.
>> cnt_logic = UNION means the forecast **or** observation value is >
273.
>> cnt_logic = INTERSECTION means the forecast **and** observation
values are
>> >273.
>> cnt_logic = SYMDIFF means the that one is >273 and the other is
not.
>>
>> Thanks,
>> John
>>
>> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>> >
>> > John,
>> > Thanks a lot.
>> > I need more time to study the first part about QC in your
explanation.
>> > However, it is great that the cnt_thresh can be used. It can
definitely
>> > meet my requirement for being now.
>> > Just a clarification is great. This function is working for
both
>> > Met-point and Met-grid, right?
>> > Thanks.
>> > Ting
>> >
>> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
>> > met_help at ucar.edu
>> > > wrote:
>> >
>> > > Ting,
>> > >
>> > > OK, there are two things I'd like to point out regarding this
issue.
>> > >
>> > > First, the NetCDF point observation files that Point-Stat reads
>> contain a
>> > > slot for quality control information. Specifically, the NetCDF
output
>> > from
>> > > ascii2nc, madis2nc, and pb2nc contain a variable named
"obs_qty" which
>> > is a
>> > > string containing quality control info.
>> > >
>> > > If you're running ascii2nc, it's your job to format the input
ascii
>> data
>> > > the expected way. If you have quality control info, it should
go in
>> > column
>> > > 10, named "QC_String" in the ascii2nc usage statement. MET
does not
>> do
>> > any
>> > > QC for you, but it can use the QC strings you've set.
>> > >
>> > > In the Point-Stat config file, you can set the "obs_quality"
entry to
>> a
>> > > list of one or more quality control strings to be used. If
set, only
>> > > observations whose quality control value appears in that list
will be
>> > used.
>> > >
>> > > Second, you asked if there's a way to restrict the observations
used
>> by
>> > > filtering their values, such as -10<obs<70. The answer is yes,
at
>> least
>> > > for continuous statistics, like RMSE and ME. We call this
>> "conditional
>> > > continuous verification" since we are conditioning the analysis
on the
>> > > input values.
>> > >
>> > > We set this up using the "cnt_thresh" and "cnt_logic" settings
in the
>> > > Point-Stat config file, such as:
>> > >
>> > > fcst = {
>> > >
>> > > field = [
>> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
>> > > ];
>> > >
>> > > }
>> > >
>> > > obs = {
>> > >
>> > > field = [
>> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-10&&le70
];
>> > > cnt_logic = INTERSECTION; }
>> > > ];
>> > >
>> > > }
>> > >
>> > > Note that in the forecast I'm setting cnt_thresh to NA, meaning
that
>> no
>> > > filtering criteria will be applied to the forecast value. But
in the
>> > > observation I used "ge-10&&le70" to define the desired range of
>> > observation
>> > > values.
>> > >
>> > > You can list as many cnt_thresh values as you'd like in the
list but
>> the
>> > > number of fcst settings must match the number of obs settings.
For
>> > > example...
>> > >
>> > > fcst...
>> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
>> > >
>> > > obs...
>> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
>> > >
>> > > The above settings will compute continuous statistics 5 ways...
using
>> (1)
>> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and (5)
obs
>> >= 30
>> > >
>> > > Hope that helps. Just let me know if you have more questions.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > >
>> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate via
RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
>> > > > Transaction: Ticket created by ting.lei at noaa.gov
>> > > > Queue: met_help
>> > > > Subject: questions about QC for observation
>> > > > Owner: Nobody
>> > > > Requestors: ting.lei at noaa.gov
>> > > > Status: new
>> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
>> ket/Display.html?id=78584
>> > >
>> > > >
>> > > >
>> > > > Dear Mr. or Mrs.,
>> > > > Thanks in advance for your help.
>> > > > I'm wondering what methods can be used with MET to screen the
>> > > observations
>> > > > in the verification process.
>> > > > I found obs_quality can help if the flag parameters are in
the obs
>> > file.
>> > > > However, if there are no such flags in my obs and I know the
>> > definitions
>> > > of
>> > > > missing values or other values like -99 assigned with special
>> meaning,
>> > > > how can I make those info with MET?
>> > > > Or , for simplification, if I want to use the obs in a range
like
>> -10 <
>> > > > obs<70.
>> > > > Can I or ,if I can, How can I invoke such a constraint in the
MET
>> > > process.
>> > > > Your help is appreciated.
>> > > > Best,
>> > > > Ting
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>
------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Wed Nov 09 10:20:28 2016
Ting,
I see you have questions about the cnt_thresh logic is applied.
The answer to your question is no, the cnt_thresh setting is not
applied to
categorical verification.
Here's how it works...
- The cnt_thresh settings affect the MET output written to the CNT
(continuous statistics) and SL1L2 (partial sums) line types only.
- The cat_thresh settings affect the MET output written to the CTC
(contingency table counts), FHO (another version of the contingency
table
counts), and the CTS (contingency table statistics) line types.
So an observation value of 80 would still show up as an event in the
CTC
and CTS output lines.
I see that really you'd like to limit your analysis (both continuous
and
categorical) to using observations which fall between -10 and 70.
Since
you're running ascii2nc anyway, the easiest way to accomplish this is
to
write a script yourself to filter the point observations prior to
calling
ascii2nc. You could do this either by discarding observation values
outside the desired range, or you could assign them a bad data flag in
the
"QC_String" column.
There is still a way you can do this with the MET tools, but it is
more
difficult than what I described above. I'll outline it here anyway...
(1) Configure Point-Stat so that it *ONLY* writes the matched pair
(MPR)
output line, not statistics or partial sums.
(2) Run the STAT-Analysis tool to read those matched pair lines, apply
the
filtering criteria you'd like, and then compute the desired
statistics.
The STAT-Analysis job would look something like this:
stat_analysis -lookin point_stat_*.stat \
-job aggregate_stat -line_type MPR -out_line_type CTC \
-column_thresh OBS ge-10&&le70 \
-out_fcst_thresh ge35 -out_obs_thresh ge35 \
-out_stat ctc_filtered.stat
The job is "aggregate_stat" since we're reading MPR lines and writing
out a
different output type, CTC. The "-column_thresh" setting applies your
filtering threshold to the data values in the input column named
"OBS".
The "-out_fcst_thresh" and "-out_obs_thresh" settings define the
threshold
for the output contingency table. And lastly, the "-out_stat"
settings
writes the counts to an output .stat file.
So this job reads in all MPR lines, ignores the ones where the OBS
column
is outside the defined range, and uses the remaining lines to compute
a
contingency table.
Running the STAT-Analysis tool on MPR lines like this provides the
most
amount of flexibility that MET has to offer. However, it usually
isn't
feasible to write out the full set of individual matched pair values
for
operational runs!
I can see how we could enhance Point-Stat and Grid-Stat by adding
another
filtering threshold option to do exactly what you're describing. It
wouldn't be all that difficult to do. However, there isn't really a
spot
for writing out that threshold information to the output files.
Perhaps we
call it something like "qc_thresh"? We'd discard any matched pairs
which
don't meet the defined "qc_thresh" criteria so they wouldn't be used
for
continuous or categorical verification. Do you think that type of
functionality would be useful more broadly at NCEP?
Thanks,
John
On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
> Hi, John,
> Sorry for re-openning this ticket.
> I still need some further clarifications.
> When using the cnt_thresh to choose the obs/fcst in a range to be
used in
> the verification for continuous variables,
> will this control those for the categorical verification like CAC?
> For example, the cat_thresh will control how the obs/fcst will be
verified
> in several categories . Will the cnt_thresh take effect?
> Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this make MET
only do
> verification like CAC over the whole obs pool with values between
-10 and
> 70 while calculate them in the categories decided by the cat_thresh?
> Your further clarification is appreciated.
> Ting
>
>
>
> On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate <
> ting.lei at noaa.gov
> > wrote:
>
> > John,
> > Thanks again for your explanation.
> > That is great.
> > My question in this email has been perfectly answered.
> > Best Wishes,
> > Ting
> >
> > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Ting,
> >>
> >> Yes, the cnt_thresh filtering option works for both Grid-Stat and
> >> Point-Stat.
> >>
> >> The fcst -> cnt_thresh option filters the forecast values. The
obs ->
> >> cnt_thresh option filters the observation values. And the
cnt_logic
> >> option
> >> can be set to UNION, INTERSECTION, or SYMDIFF to specify how the
logic
> for
> >> how the forecast and observation thresholds should be combined.
> >>
> >> For example... suppose you've defined cnt_thresh for temperature
as
> >273.
> >> cnt_logic = UNION means the forecast **or** observation value is
> 273.
> >> cnt_logic = INTERSECTION means the forecast **and** observation
values
> are
> >> >273.
> >> cnt_logic = SYMDIFF means the that one is >273 and the other is
not.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via RT
<
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >> >
> >> > John,
> >> > Thanks a lot.
> >> > I need more time to study the first part about QC in your
explanation.
> >> > However, it is great that the cnt_thresh can be used. It can
> definitely
> >> > meet my requirement for being now.
> >> > Just a clarification is great. This function is working for
both
> >> > Met-point and Met-grid, right?
> >> > Thanks.
> >> > Ting
> >> >
> >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu
> >> > > wrote:
> >> >
> >> > > Ting,
> >> > >
> >> > > OK, there are two things I'd like to point out regarding this
issue.
> >> > >
> >> > > First, the NetCDF point observation files that Point-Stat
reads
> >> contain a
> >> > > slot for quality control information. Specifically, the
NetCDF
> output
> >> > from
> >> > > ascii2nc, madis2nc, and pb2nc contain a variable named
"obs_qty"
> which
> >> > is a
> >> > > string containing quality control info.
> >> > >
> >> > > If you're running ascii2nc, it's your job to format the input
ascii
> >> data
> >> > > the expected way. If you have quality control info, it
should go in
> >> > column
> >> > > 10, named "QC_String" in the ascii2nc usage statement. MET
does not
> >> do
> >> > any
> >> > > QC for you, but it can use the QC strings you've set.
> >> > >
> >> > > In the Point-Stat config file, you can set the "obs_quality"
entry
> to
> >> a
> >> > > list of one or more quality control strings to be used. If
set,
> only
> >> > > observations whose quality control value appears in that list
will
> be
> >> > used.
> >> > >
> >> > > Second, you asked if there's a way to restrict the
observations used
> >> by
> >> > > filtering their values, such as -10<obs<70. The answer is
yes, at
> >> least
> >> > > for continuous statistics, like RMSE and ME. We call this
> >> "conditional
> >> > > continuous verification" since we are conditioning the
analysis on
> the
> >> > > input values.
> >> > >
> >> > > We set this up using the "cnt_thresh" and "cnt_logic"
settings in
> the
> >> > > Point-Stat config file, such as:
> >> > >
> >> > > fcst = {
> >> > >
> >> > > field = [
> >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
> >> > > ];
> >> > >
> >> > > }
> >> > >
> >> > > obs = {
> >> > >
> >> > > field = [
> >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-
10&&le70 ];
> >> > > cnt_logic = INTERSECTION; }
> >> > > ];
> >> > >
> >> > > }
> >> > >
> >> > > Note that in the forecast I'm setting cnt_thresh to NA,
meaning that
> >> no
> >> > > filtering criteria will be applied to the forecast value.
But in
> the
> >> > > observation I used "ge-10&&le70" to define the desired range
of
> >> > observation
> >> > > values.
> >> > >
> >> > > You can list as many cnt_thresh values as you'd like in the
list but
> >> the
> >> > > number of fcst settings must match the number of obs
settings. For
> >> > > example...
> >> > >
> >> > > fcst...
> >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> >> > >
> >> > > obs...
> >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> >> > >
> >> > > The above settings will compute continuous statistics 5
ways...
> using
> >> (1)
> >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and
(5) obs
> >> >= 30
> >> > >
> >> > > Hope that helps. Just let me know if you have more
questions.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > >
> >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate
via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> >> > > > Queue: met_help
> >> > > > Subject: questions about QC for observation
> >> > > > Owner: Nobody
> >> > > > Requestors: ting.lei at noaa.gov
> >> > > > Status: new
> >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> >> ket/Display.html?id=78584
> >> > >
> >> > > >
> >> > > >
> >> > > > Dear Mr. or Mrs.,
> >> > > > Thanks in advance for your help.
> >> > > > I'm wondering what methods can be used with MET to screen
the
> >> > > observations
> >> > > > in the verification process.
> >> > > > I found obs_quality can help if the flag parameters are in
the obs
> >> > file.
> >> > > > However, if there are no such flags in my obs and I know
the
> >> > definitions
> >> > > of
> >> > > > missing values or other values like -99 assigned with
special
> >> meaning,
> >> > > > how can I make those info with MET?
> >> > > > Or , for simplification, if I want to use the obs in a
range like
> >> -10 <
> >> > > > obs<70.
> >> > > > Can I or ,if I can, How can I invoke such a constraint in
the MET
> >> > > process.
> >> > > > Your help is appreciated.
> >> > > > Best,
> >> > > > Ting
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Wed Nov 09 11:00:39 2016
Hi, John,
Thanks a lot for your wonderful explanation. I need to study them
more
carefully.
An easy answer is : that, sure, I believe more flexible QC functions
in the
MET will be helpful
for us at EMC. I expect more discussions will be helpful, especially
when
MET is becoming more and more widely used here,
One more question: is the form " cat_thresh = [ ge-10&&le70 ];"
validate to
the MET?
Regards,
Ting
On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ting,
>
> I see you have questions about the cnt_thresh logic is applied.
>
> The answer to your question is no, the cnt_thresh setting is not
applied to
> categorical verification.
>
> Here's how it works...
>
> - The cnt_thresh settings affect the MET output written to the CNT
> (continuous statistics) and SL1L2 (partial sums) line types only.
>
> - The cat_thresh settings affect the MET output written to the CTC
> (contingency table counts), FHO (another version of the contingency
table
> counts), and the CTS (contingency table statistics) line types.
>
> So an observation value of 80 would still show up as an event in the
CTC
> and CTS output lines.
>
> I see that really you'd like to limit your analysis (both continuous
and
> categorical) to using observations which fall between -10 and 70.
Since
> you're running ascii2nc anyway, the easiest way to accomplish this
is to
> write a script yourself to filter the point observations prior to
calling
> ascii2nc. You could do this either by discarding observation values
> outside the desired range, or you could assign them a bad data flag
in the
> "QC_String" column.
>
> There is still a way you can do this with the MET tools, but it is
more
> difficult than what I described above. I'll outline it here
anyway...
>
> (1) Configure Point-Stat so that it *ONLY* writes the matched pair
(MPR)
> output line, not statistics or partial sums.
> (2) Run the STAT-Analysis tool to read those matched pair lines,
apply the
> filtering criteria you'd like, and then compute the desired
statistics.
> The STAT-Analysis job would look something like this:
>
> stat_analysis -lookin point_stat_*.stat \
> -job aggregate_stat -line_type MPR -out_line_type CTC \
> -column_thresh OBS ge-10&&le70 \
> -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> -out_stat ctc_filtered.stat
>
> The job is "aggregate_stat" since we're reading MPR lines and
writing out a
> different output type, CTC. The "-column_thresh" setting applies
your
> filtering threshold to the data values in the input column named
"OBS".
> The "-out_fcst_thresh" and "-out_obs_thresh" settings define the
threshold
> for the output contingency table. And lastly, the "-out_stat"
settings
> writes the counts to an output .stat file.
>
> So this job reads in all MPR lines, ignores the ones where the OBS
column
> is outside the defined range, and uses the remaining lines to
compute a
> contingency table.
>
> Running the STAT-Analysis tool on MPR lines like this provides the
most
> amount of flexibility that MET has to offer. However, it usually
isn't
> feasible to write out the full set of individual matched pair values
for
> operational runs!
>
> I can see how we could enhance Point-Stat and Grid-Stat by adding
another
> filtering threshold option to do exactly what you're describing. It
> wouldn't be all that difficult to do. However, there isn't really a
spot
> for writing out that threshold information to the output files.
Perhaps we
> call it something like "qc_thresh"? We'd discard any matched pairs
which
> don't meet the defined "qc_thresh" criteria so they wouldn't be used
for
> continuous or categorical verification. Do you think that type of
> functionality would be useful more broadly at NCEP?
>
> Thanks,
> John
>
>
>
>
>
> On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >
> > Hi, John,
> > Sorry for re-openning this ticket.
> > I still need some further clarifications.
> > When using the cnt_thresh to choose the obs/fcst in a range to be
used in
> > the verification for continuous variables,
> > will this control those for the categorical verification like CAC?
> > For example, the cat_thresh will control how the obs/fcst will be
> verified
> > in several categories . Will the cnt_thresh take effect?
> > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this make MET
only
> do
> > verification like CAC over the whole obs pool with values between
-10 and
> > 70 while calculate them in the categories decided by the
cat_thresh?
> > Your further clarification is appreciated.
> > Ting
> >
> >
> >
> > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate <
> > ting.lei at noaa.gov
> > > wrote:
> >
> > > John,
> > > Thanks again for your explanation.
> > > That is great.
> > > My question in this email has been perfectly answered.
> > > Best Wishes,
> > > Ting
> > >
> > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Ting,
> > >>
> > >> Yes, the cnt_thresh filtering option works for both Grid-Stat
and
> > >> Point-Stat.
> > >>
> > >> The fcst -> cnt_thresh option filters the forecast values. The
obs ->
> > >> cnt_thresh option filters the observation values. And the
cnt_logic
> > >> option
> > >> can be set to UNION, INTERSECTION, or SYMDIFF to specify how
the logic
> > for
> > >> how the forecast and observation thresholds should be combined.
> > >>
> > >> For example... suppose you've defined cnt_thresh for
temperature as
> > >273.
> > >> cnt_logic = UNION means the forecast **or** observation value
is >
> 273.
> > >> cnt_logic = INTERSECTION means the forecast **and** observation
values
> > are
> > >> >273.
> > >> cnt_logic = SYMDIFF means the that one is >273 and the other is
not.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via
RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
>
> > >> >
> > >> > John,
> > >> > Thanks a lot.
> > >> > I need more time to study the first part about QC in your
> explanation.
> > >> > However, it is great that the cnt_thresh can be used. It can
> > definitely
> > >> > meet my requirement for being now.
> > >> > Just a clarification is great. This function is working for
both
> > >> > Met-point and Met-grid, right?
> > >> > Thanks.
> > >> > Ting
> > >> >
> > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
> > >> > met_help at ucar.edu
> > >> > > wrote:
> > >> >
> > >> > > Ting,
> > >> > >
> > >> > > OK, there are two things I'd like to point out regarding
this
> issue.
> > >> > >
> > >> > > First, the NetCDF point observation files that Point-Stat
reads
> > >> contain a
> > >> > > slot for quality control information. Specifically, the
NetCDF
> > output
> > >> > from
> > >> > > ascii2nc, madis2nc, and pb2nc contain a variable named
"obs_qty"
> > which
> > >> > is a
> > >> > > string containing quality control info.
> > >> > >
> > >> > > If you're running ascii2nc, it's your job to format the
input
> ascii
> > >> data
> > >> > > the expected way. If you have quality control info, it
should go
> in
> > >> > column
> > >> > > 10, named "QC_String" in the ascii2nc usage statement. MET
does
> not
> > >> do
> > >> > any
> > >> > > QC for you, but it can use the QC strings you've set.
> > >> > >
> > >> > > In the Point-Stat config file, you can set the
"obs_quality" entry
> > to
> > >> a
> > >> > > list of one or more quality control strings to be used. If
set,
> > only
> > >> > > observations whose quality control value appears in that
list will
> > be
> > >> > used.
> > >> > >
> > >> > > Second, you asked if there's a way to restrict the
observations
> used
> > >> by
> > >> > > filtering their values, such as -10<obs<70. The answer is
yes, at
> > >> least
> > >> > > for continuous statistics, like RMSE and ME. We call this
> > >> "conditional
> > >> > > continuous verification" since we are conditioning the
analysis on
> > the
> > >> > > input values.
> > >> > >
> > >> > > We set this up using the "cnt_thresh" and "cnt_logic"
settings in
> > the
> > >> > > Point-Stat config file, such as:
> > >> > >
> > >> > > fcst = {
> > >> > >
> > >> > > field = [
> > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ]; }
> > >> > > ];
> > >> > >
> > >> > > }
> > >> > >
> > >> > > obs = {
> > >> > >
> > >> > > field = [
> > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-
10&&le70 ];
> > >> > > cnt_logic = INTERSECTION; }
> > >> > > ];
> > >> > >
> > >> > > }
> > >> > >
> > >> > > Note that in the forecast I'm setting cnt_thresh to NA,
meaning
> that
> > >> no
> > >> > > filtering criteria will be applied to the forecast value.
But in
> > the
> > >> > > observation I used "ge-10&&le70" to define the desired
range of
> > >> > observation
> > >> > > values.
> > >> > >
> > >> > > You can list as many cnt_thresh values as you'd like in the
list
> but
> > >> the
> > >> > > number of fcst settings must match the number of obs
settings.
> For
> > >> > > example...
> > >> > >
> > >> > > fcst...
> > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > >> > >
> > >> > > obs...
> > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > >> > >
> > >> > > The above settings will compute continuous statistics 5
ways...
> > using
> > >> (1)
> > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20 and
(5)
> obs
> > >> >= 30
> > >> > >
> > >> > > Hope that helps. Just let me know if you have more
questions.
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > >
> > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA Affiliate
via RT
> <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> > >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > >> > > > Queue: met_help
> > >> > > > Subject: questions about QC for observation
> > >> > > > Owner: Nobody
> > >> > > > Requestors: ting.lei at noaa.gov
> > >> > > > Status: new
> > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > >> ket/Display.html?id=78584
> > >> > >
> > >> > > >
> > >> > > >
> > >> > > > Dear Mr. or Mrs.,
> > >> > > > Thanks in advance for your help.
> > >> > > > I'm wondering what methods can be used with MET to screen
the
> > >> > > observations
> > >> > > > in the verification process.
> > >> > > > I found obs_quality can help if the flag parameters are
in the
> obs
> > >> > file.
> > >> > > > However, if there are no such flags in my obs and I know
the
> > >> > definitions
> > >> > > of
> > >> > > > missing values or other values like -99 assigned with
special
> > >> meaning,
> > >> > > > how can I make those info with MET?
> > >> > > > Or , for simplification, if I want to use the obs in a
range
> like
> > >> -10 <
> > >> > > > obs<70.
> > >> > > > Can I or ,if I can, How can I invoke such a constraint in
the
> MET
> > >> > > process.
> > >> > > > Your help is appreciated.
> > >> > > > Best,
> > >> > > > Ting
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Wed Nov 09 11:40:29 2016
Ting,
Yes, cat_thresh = [ ge-10&&le70 ]; should work to define a threshold
for
categorical verification. However, that's a rather odd threshold to
define. That means that anything between -10 and 70 is an "event"
while
anything outside that range is a "non-event". The difference here is
that
the "cnt_thresh" *FILTERS* the matched pair data prior to computing
continuous statistics. The "cat_thresh" defines the event-threshold
for
populating a contingency table.
It's probably confusing that we call them both "_thresh"... instead,
perhaps we should have named "cnt_thresh" as "cnt_filter" instead.
If you use that categorical event threshold, you'll end up, for
example,
with a CSI (critical success index) statistics that quantifies how
well
your forecast did predicting values between -10 and 70. Instead, I
suspect
you'd want something like:
cat_thresh = [ >=20, >=30, >=40 ];
With this setting you'd get 3 CSI values in the output describing how
well
your forecast predicts reflectivity exceeding those thresholds.
Make sense?
Thanks,
John
On Wed, Nov 9, 2016 at 11:00 AM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
> Hi, John,
> Thanks a lot for your wonderful explanation. I need to study them
more
> carefully.
> An easy answer is : that, sure, I believe more flexible QC functions
in the
> MET will be helpful
> for us at EMC. I expect more discussions will be helpful, especially
when
> MET is becoming more and more widely used here,
> One more question: is the form " cat_thresh = [ ge-10&&le70 ];"
validate to
> the MET?
> Regards,
> Ting
>
>
> On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ting,
> >
> > I see you have questions about the cnt_thresh logic is applied.
> >
> > The answer to your question is no, the cnt_thresh setting is not
applied
> to
> > categorical verification.
> >
> > Here's how it works...
> >
> > - The cnt_thresh settings affect the MET output written to the CNT
> > (continuous statistics) and SL1L2 (partial sums) line types only.
> >
> > - The cat_thresh settings affect the MET output written to the CTC
> > (contingency table counts), FHO (another version of the
contingency table
> > counts), and the CTS (contingency table statistics) line types.
> >
> > So an observation value of 80 would still show up as an event in
the CTC
> > and CTS output lines.
> >
> > I see that really you'd like to limit your analysis (both
continuous and
> > categorical) to using observations which fall between -10 and 70.
Since
> > you're running ascii2nc anyway, the easiest way to accomplish this
is to
> > write a script yourself to filter the point observations prior to
calling
> > ascii2nc. You could do this either by discarding observation
values
> > outside the desired range, or you could assign them a bad data
flag in
> the
> > "QC_String" column.
> >
> > There is still a way you can do this with the MET tools, but it is
more
> > difficult than what I described above. I'll outline it here
anyway...
> >
> > (1) Configure Point-Stat so that it *ONLY* writes the matched pair
(MPR)
> > output line, not statistics or partial sums.
> > (2) Run the STAT-Analysis tool to read those matched pair lines,
apply
> the
> > filtering criteria you'd like, and then compute the desired
statistics.
> > The STAT-Analysis job would look something like this:
> >
> > stat_analysis -lookin point_stat_*.stat \
> > -job aggregate_stat -line_type MPR -out_line_type CTC \
> > -column_thresh OBS ge-10&&le70 \
> > -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> > -out_stat ctc_filtered.stat
> >
> > The job is "aggregate_stat" since we're reading MPR lines and
writing
> out a
> > different output type, CTC. The "-column_thresh" setting applies
your
> > filtering threshold to the data values in the input column named
"OBS".
> > The "-out_fcst_thresh" and "-out_obs_thresh" settings define the
> threshold
> > for the output contingency table. And lastly, the "-out_stat"
settings
> > writes the counts to an output .stat file.
> >
> > So this job reads in all MPR lines, ignores the ones where the OBS
column
> > is outside the defined range, and uses the remaining lines to
compute a
> > contingency table.
> >
> > Running the STAT-Analysis tool on MPR lines like this provides the
most
> > amount of flexibility that MET has to offer. However, it usually
isn't
> > feasible to write out the full set of individual matched pair
values for
> > operational runs!
> >
> > I can see how we could enhance Point-Stat and Grid-Stat by adding
another
> > filtering threshold option to do exactly what you're describing.
It
> > wouldn't be all that difficult to do. However, there isn't really
a spot
> > for writing out that threshold information to the output files.
Perhaps
> we
> > call it something like "qc_thresh"? We'd discard any matched
pairs which
> > don't meet the defined "qc_thresh" criteria so they wouldn't be
used for
> > continuous or categorical verification. Do you think that type of
> > functionality would be useful more broadly at NCEP?
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > >
> > > Hi, John,
> > > Sorry for re-openning this ticket.
> > > I still need some further clarifications.
> > > When using the cnt_thresh to choose the obs/fcst in a range to
be used
> in
> > > the verification for continuous variables,
> > > will this control those for the categorical verification like
CAC?
> > > For example, the cat_thresh will control how the obs/fcst will
be
> > verified
> > > in several categories . Will the cnt_thresh take effect?
> > > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this make
MET only
> > do
> > > verification like CAC over the whole obs pool with values
between -10
> and
> > > 70 while calculate them in the categories decided by the
cat_thresh?
> > > Your further clarification is appreciated.
> > > Ting
> > >
> > >
> > >
> > > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate <
> > > ting.lei at noaa.gov
> > > > wrote:
> > >
> > > > John,
> > > > Thanks again for your explanation.
> > > > That is great.
> > > > My question in this email has been perfectly answered.
> > > > Best Wishes,
> > > > Ting
> > > >
> > > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Ting,
> > > >>
> > > >> Yes, the cnt_thresh filtering option works for both Grid-Stat
and
> > > >> Point-Stat.
> > > >>
> > > >> The fcst -> cnt_thresh option filters the forecast values.
The obs
> ->
> > > >> cnt_thresh option filters the observation values. And the
cnt_logic
> > > >> option
> > > >> can be set to UNION, INTERSECTION, or SYMDIFF to specify how
the
> logic
> > > for
> > > >> how the forecast and observation thresholds should be
combined.
> > > >>
> > > >> For example... suppose you've defined cnt_thresh for
temperature as
> > > >273.
> > > >> cnt_logic = UNION means the forecast **or** observation value
is >
> > 273.
> > > >> cnt_logic = INTERSECTION means the forecast **and**
observation
> values
> > > are
> > > >> >273.
> > > >> cnt_logic = SYMDIFF means the that one is >273 and the other
is not.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate via
RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > > >> >
> > > >> > John,
> > > >> > Thanks a lot.
> > > >> > I need more time to study the first part about QC in your
> > explanation.
> > > >> > However, it is great that the cnt_thresh can be used. It
can
> > > definitely
> > > >> > meet my requirement for being now.
> > > >> > Just a clarification is great. This function is working
for both
> > > >> > Met-point and Met-grid, right?
> > > >> > Thanks.
> > > >> > Ting
> > > >> >
> > > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT <
> > > >> > met_help at ucar.edu
> > > >> > > wrote:
> > > >> >
> > > >> > > Ting,
> > > >> > >
> > > >> > > OK, there are two things I'd like to point out regarding
this
> > issue.
> > > >> > >
> > > >> > > First, the NetCDF point observation files that Point-Stat
reads
> > > >> contain a
> > > >> > > slot for quality control information. Specifically, the
NetCDF
> > > output
> > > >> > from
> > > >> > > ascii2nc, madis2nc, and pb2nc contain a variable named
"obs_qty"
> > > which
> > > >> > is a
> > > >> > > string containing quality control info.
> > > >> > >
> > > >> > > If you're running ascii2nc, it's your job to format the
input
> > ascii
> > > >> data
> > > >> > > the expected way. If you have quality control info, it
should
> go
> > in
> > > >> > column
> > > >> > > 10, named "QC_String" in the ascii2nc usage statement.
MET does
> > not
> > > >> do
> > > >> > any
> > > >> > > QC for you, but it can use the QC strings you've set.
> > > >> > >
> > > >> > > In the Point-Stat config file, you can set the
"obs_quality"
> entry
> > > to
> > > >> a
> > > >> > > list of one or more quality control strings to be used.
If set,
> > > only
> > > >> > > observations whose quality control value appears in that
list
> will
> > > be
> > > >> > used.
> > > >> > >
> > > >> > > Second, you asked if there's a way to restrict the
observations
> > used
> > > >> by
> > > >> > > filtering their values, such as -10<obs<70. The answer
is yes,
> at
> > > >> least
> > > >> > > for continuous statistics, like RMSE and ME. We call
this
> > > >> "conditional
> > > >> > > continuous verification" since we are conditioning the
analysis
> on
> > > the
> > > >> > > input values.
> > > >> > >
> > > >> > > We set this up using the "cnt_thresh" and "cnt_logic"
settings
> in
> > > the
> > > >> > > Point-Stat config file, such as:
> > > >> > >
> > > >> > > fcst = {
> > > >> > >
> > > >> > > field = [
> > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA ];
}
> > > >> > > ];
> > > >> > >
> > > >> > > }
> > > >> > >
> > > >> > > obs = {
> > > >> > >
> > > >> > > field = [
> > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ ge-
10&&le70
> ];
> > > >> > > cnt_logic = INTERSECTION; }
> > > >> > > ];
> > > >> > >
> > > >> > > }
> > > >> > >
> > > >> > > Note that in the forecast I'm setting cnt_thresh to NA,
meaning
> > that
> > > >> no
> > > >> > > filtering criteria will be applied to the forecast value.
But
> in
> > > the
> > > >> > > observation I used "ge-10&&le70" to define the desired
range of
> > > >> > observation
> > > >> > > values.
> > > >> > >
> > > >> > > You can list as many cnt_thresh values as you'd like in
the list
> > but
> > > >> the
> > > >> > > number of fcst settings must match the number of obs
settings.
> > For
> > > >> > > example...
> > > >> > >
> > > >> > > fcst...
> > > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > > >> > >
> > > >> > > obs...
> > > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > > >> > >
> > > >> > > The above settings will compute continuous statistics 5
ways...
> > > using
> > > >> (1)
> > > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20
and (5)
> > obs
> > > >> >= 30
> > > >> > >
> > > >> > > Hope that helps. Just let me know if you have more
questions.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA
Affiliate via
> RT
> > <
> > > >> > > met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted upon.
> > > >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > > >> > > > Queue: met_help
> > > >> > > > Subject: questions about QC for observation
> > > >> > > > Owner: Nobody
> > > >> > > > Requestors: ting.lei at noaa.gov
> > > >> > > > Status: new
> > > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > > >> ket/Display.html?id=78584
> > > >> > >
> > > >> > > >
> > > >> > > >
> > > >> > > > Dear Mr. or Mrs.,
> > > >> > > > Thanks in advance for your help.
> > > >> > > > I'm wondering what methods can be used with MET to
screen the
> > > >> > > observations
> > > >> > > > in the verification process.
> > > >> > > > I found obs_quality can help if the flag parameters are
in the
> > obs
> > > >> > file.
> > > >> > > > However, if there are no such flags in my obs and I
know the
> > > >> > definitions
> > > >> > > of
> > > >> > > > missing values or other values like -99 assigned with
special
> > > >> meaning,
> > > >> > > > how can I make those info with MET?
> > > >> > > > Or , for simplification, if I want to use the obs in a
range
> > like
> > > >> -10 <
> > > >> > > > obs<70.
> > > >> > > > Can I or ,if I can, How can I invoke such a constraint
in the
> > MET
> > > >> > > process.
> > > >> > > > Your help is appreciated.
> > > >> > > > Best,
> > > >> > > > Ting
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Wed Nov 09 11:45:37 2016
John,
Thanks a lot for your explanation.
I want to use cat_thresh like ge30&&le 70 rather than ge30 is because
I presume the values larger than 70 (like 999) will be a flag for bad
values.
So, by using the flexibility of MET here, I can enforce a QC restraint
that
values >70 will not be used.
Hope this is not a abuse of MET related functions:).
Regards,
Ting
On Wed, Nov 9, 2016 at 1:40 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Ting,
>
> Yes, cat_thresh = [ ge-10&&le70 ]; should work to define a threshold
for
> categorical verification. However, that's a rather odd threshold to
> define. That means that anything between -10 and 70 is an "event"
while
> anything outside that range is a "non-event". The difference here
is that
> the "cnt_thresh" *FILTERS* the matched pair data prior to computing
> continuous statistics. The "cat_thresh" defines the event-threshold
for
> populating a contingency table.
>
> It's probably confusing that we call them both "_thresh"... instead,
> perhaps we should have named "cnt_thresh" as "cnt_filter" instead.
>
> If you use that categorical event threshold, you'll end up, for
example,
> with a CSI (critical success index) statistics that quantifies how
well
> your forecast did predicting values between -10 and 70. Instead, I
suspect
> you'd want something like:
> cat_thresh = [ >=20, >=30, >=40 ];
>
> With this setting you'd get 3 CSI values in the output describing
how well
> your forecast predicts reflectivity exceeding those thresholds.
>
> Make sense?
>
> Thanks,
> John
>
> On Wed, Nov 9, 2016 at 11:00 AM, Ting Lei - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >
> > Hi, John,
> > Thanks a lot for your wonderful explanation. I need to study them
more
> > carefully.
> > An easy answer is : that, sure, I believe more flexible QC
functions in
> the
> > MET will be helpful
> > for us at EMC. I expect more discussions will be helpful,
especially when
> > MET is becoming more and more widely used here,
> > One more question: is the form " cat_thresh = [ ge-10&&le70 ];"
validate
> to
> > the MET?
> > Regards,
> > Ting
> >
> >
> > On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ting,
> > >
> > > I see you have questions about the cnt_thresh logic is applied.
> > >
> > > The answer to your question is no, the cnt_thresh setting is not
> applied
> > to
> > > categorical verification.
> > >
> > > Here's how it works...
> > >
> > > - The cnt_thresh settings affect the MET output written to the
CNT
> > > (continuous statistics) and SL1L2 (partial sums) line types
only.
> > >
> > > - The cat_thresh settings affect the MET output written to the
CTC
> > > (contingency table counts), FHO (another version of the
contingency
> table
> > > counts), and the CTS (contingency table statistics) line types.
> > >
> > > So an observation value of 80 would still show up as an event in
the
> CTC
> > > and CTS output lines.
> > >
> > > I see that really you'd like to limit your analysis (both
continuous
> and
> > > categorical) to using observations which fall between -10 and
70.
> Since
> > > you're running ascii2nc anyway, the easiest way to accomplish
this is
> to
> > > write a script yourself to filter the point observations prior
to
> calling
> > > ascii2nc. You could do this either by discarding observation
values
> > > outside the desired range, or you could assign them a bad data
flag in
> > the
> > > "QC_String" column.
> > >
> > > There is still a way you can do this with the MET tools, but it
is more
> > > difficult than what I described above. I'll outline it here
anyway...
> > >
> > > (1) Configure Point-Stat so that it *ONLY* writes the matched
pair
> (MPR)
> > > output line, not statistics or partial sums.
> > > (2) Run the STAT-Analysis tool to read those matched pair lines,
apply
> > the
> > > filtering criteria you'd like, and then compute the desired
statistics.
> > > The STAT-Analysis job would look something like this:
> > >
> > > stat_analysis -lookin point_stat_*.stat \
> > > -job aggregate_stat -line_type MPR -out_line_type CTC \
> > > -column_thresh OBS ge-10&&le70 \
> > > -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> > > -out_stat ctc_filtered.stat
> > >
> > > The job is "aggregate_stat" since we're reading MPR lines and
writing
> > out a
> > > different output type, CTC. The "-column_thresh" setting
applies your
> > > filtering threshold to the data values in the input column named
"OBS".
> > > The "-out_fcst_thresh" and "-out_obs_thresh" settings define the
> > threshold
> > > for the output contingency table. And lastly, the "-out_stat"
settings
> > > writes the counts to an output .stat file.
> > >
> > > So this job reads in all MPR lines, ignores the ones where the
OBS
> column
> > > is outside the defined range, and uses the remaining lines to
compute a
> > > contingency table.
> > >
> > > Running the STAT-Analysis tool on MPR lines like this provides
the most
> > > amount of flexibility that MET has to offer. However, it
usually isn't
> > > feasible to write out the full set of individual matched pair
values
> for
> > > operational runs!
> > >
> > > I can see how we could enhance Point-Stat and Grid-Stat by
adding
> another
> > > filtering threshold option to do exactly what you're describing.
It
> > > wouldn't be all that difficult to do. However, there isn't
really a
> spot
> > > for writing out that threshold information to the output files.
> Perhaps
> > we
> > > call it something like "qc_thresh"? We'd discard any matched
pairs
> which
> > > don't meet the defined "qc_thresh" criteria so they wouldn't be
used
> for
> > > continuous or categorical verification. Do you think that type
of
> > > functionality would be useful more broadly at NCEP?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
>
> > > >
> > > > Hi, John,
> > > > Sorry for re-openning this ticket.
> > > > I still need some further clarifications.
> > > > When using the cnt_thresh to choose the obs/fcst in a range to
be
> used
> > in
> > > > the verification for continuous variables,
> > > > will this control those for the categorical verification like
CAC?
> > > > For example, the cat_thresh will control how the obs/fcst will
be
> > > verified
> > > > in several categories . Will the cnt_thresh take effect?
> > > > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this make
MET
> only
> > > do
> > > > verification like CAC over the whole obs pool with values
between -10
> > and
> > > > 70 while calculate them in the categories decided by the
cat_thresh?
> > > > Your further clarification is appreciated.
> > > > Ting
> > > >
> > > >
> > > >
> > > > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate <
> > > > ting.lei at noaa.gov
> > > > > wrote:
> > > >
> > > > > John,
> > > > > Thanks again for your explanation.
> > > > > That is great.
> > > > > My question in this email has been perfectly answered.
> > > > > Best Wishes,
> > > > > Ting
> > > > >
> > > > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Ting,
> > > > >>
> > > > >> Yes, the cnt_thresh filtering option works for both Grid-
Stat and
> > > > >> Point-Stat.
> > > > >>
> > > > >> The fcst -> cnt_thresh option filters the forecast values.
The
> obs
> > ->
> > > > >> cnt_thresh option filters the observation values. And the
> cnt_logic
> > > > >> option
> > > > >> can be set to UNION, INTERSECTION, or SYMDIFF to specify
how the
> > logic
> > > > for
> > > > >> how the forecast and observation thresholds should be
combined.
> > > > >>
> > > > >> For example... suppose you've defined cnt_thresh for
temperature
> as
> > > > >273.
> > > > >> cnt_logic = UNION means the forecast **or** observation
value is >
> > > 273.
> > > > >> cnt_logic = INTERSECTION means the forecast **and**
observation
> > values
> > > > are
> > > > >> >273.
> > > > >> cnt_logic = SYMDIFF means the that one is >273 and the
other is
> not.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate
via RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > > > >> >
> > > > >> > John,
> > > > >> > Thanks a lot.
> > > > >> > I need more time to study the first part about QC in your
> > > explanation.
> > > > >> > However, it is great that the cnt_thresh can be used. It
can
> > > > definitely
> > > > >> > meet my requirement for being now.
> > > > >> > Just a clarification is great. This function is working
for
> both
> > > > >> > Met-point and Met-grid, right?
> > > > >> > Thanks.
> > > > >> > Ting
> > > > >> >
> > > > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via RT
<
> > > > >> > met_help at ucar.edu
> > > > >> > > wrote:
> > > > >> >
> > > > >> > > Ting,
> > > > >> > >
> > > > >> > > OK, there are two things I'd like to point out
regarding this
> > > issue.
> > > > >> > >
> > > > >> > > First, the NetCDF point observation files that Point-
Stat
> reads
> > > > >> contain a
> > > > >> > > slot for quality control information. Specifically,
the
> NetCDF
> > > > output
> > > > >> > from
> > > > >> > > ascii2nc, madis2nc, and pb2nc contain a variable named
> "obs_qty"
> > > > which
> > > > >> > is a
> > > > >> > > string containing quality control info.
> > > > >> > >
> > > > >> > > If you're running ascii2nc, it's your job to format the
input
> > > ascii
> > > > >> data
> > > > >> > > the expected way. If you have quality control info, it
should
> > go
> > > in
> > > > >> > column
> > > > >> > > 10, named "QC_String" in the ascii2nc usage statement.
MET
> does
> > > not
> > > > >> do
> > > > >> > any
> > > > >> > > QC for you, but it can use the QC strings you've set.
> > > > >> > >
> > > > >> > > In the Point-Stat config file, you can set the
"obs_quality"
> > entry
> > > > to
> > > > >> a
> > > > >> > > list of one or more quality control strings to be used.
If
> set,
> > > > only
> > > > >> > > observations whose quality control value appears in
that list
> > will
> > > > be
> > > > >> > used.
> > > > >> > >
> > > > >> > > Second, you asked if there's a way to restrict the
> observations
> > > used
> > > > >> by
> > > > >> > > filtering their values, such as -10<obs<70. The answer
is
> yes,
> > at
> > > > >> least
> > > > >> > > for continuous statistics, like RMSE and ME. We call
this
> > > > >> "conditional
> > > > >> > > continuous verification" since we are conditioning the
> analysis
> > on
> > > > the
> > > > >> > > input values.
> > > > >> > >
> > > > >> > > We set this up using the "cnt_thresh" and "cnt_logic"
settings
> > in
> > > > the
> > > > >> > > Point-Stat config file, such as:
> > > > >> > >
> > > > >> > > fcst = {
> > > > >> > >
> > > > >> > > field = [
> > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [ NA
]; }
> > > > >> > > ];
> > > > >> > >
> > > > >> > > }
> > > > >> > >
> > > > >> > > obs = {
> > > > >> > >
> > > > >> > > field = [
> > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [
> ge-10&&le70
> > ];
> > > > >> > > cnt_logic = INTERSECTION; }
> > > > >> > > ];
> > > > >> > >
> > > > >> > > }
> > > > >> > >
> > > > >> > > Note that in the forecast I'm setting cnt_thresh to NA,
> meaning
> > > that
> > > > >> no
> > > > >> > > filtering criteria will be applied to the forecast
value. But
> > in
> > > > the
> > > > >> > > observation I used "ge-10&&le70" to define the desired
range
> of
> > > > >> > observation
> > > > >> > > values.
> > > > >> > >
> > > > >> > > You can list as many cnt_thresh values as you'd like in
the
> list
> > > but
> > > > >> the
> > > > >> > > number of fcst settings must match the number of obs
settings.
> > > For
> > > > >> > > example...
> > > > >> > >
> > > > >> > > fcst...
> > > > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > > > >> > >
> > > > >> > > obs...
> > > > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > > > >> > >
> > > > >> > > The above settings will compute continuous statistics 5
> ways...
> > > > using
> > > > >> (1)
> > > > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >= 20
and
> (5)
> > > obs
> > > > >> >= 30
> > > > >> > >
> > > > >> > > Hope that helps. Just let me know if you have more
questions.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > John
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA
Affiliate via
> > RT
> > > <
> > > > >> > > met_help at ucar.edu> wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted
upon.
> > > > >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > > > >> > > > Queue: met_help
> > > > >> > > > Subject: questions about QC for observation
> > > > >> > > > Owner: Nobody
> > > > >> > > > Requestors: ting.lei at noaa.gov
> > > > >> > > > Status: new
> > > > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > >> ket/Display.html?id=78584
> > > > >> > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Dear Mr. or Mrs.,
> > > > >> > > > Thanks in advance for your help.
> > > > >> > > > I'm wondering what methods can be used with MET to
screen
> the
> > > > >> > > observations
> > > > >> > > > in the verification process.
> > > > >> > > > I found obs_quality can help if the flag parameters
are in
> the
> > > obs
> > > > >> > file.
> > > > >> > > > However, if there are no such flags in my obs and I
know the
> > > > >> > definitions
> > > > >> > > of
> > > > >> > > > missing values or other values like -99 assigned with
> special
> > > > >> meaning,
> > > > >> > > > how can I make those info with MET?
> > > > >> > > > Or , for simplification, if I want to use the obs in
a range
> > > like
> > > > >> -10 <
> > > > >> > > > obs<70.
> > > > >> > > > Can I or ,if I can, How can I invoke such a
constraint in
> the
> > > MET
> > > > >> > > process.
> > > > >> > > > Your help is appreciated.
> > > > >> > > > Best,
> > > > >> > > > Ting
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Wed Nov 09 11:57:11 2016
Ting,
You're welcome to set up the config file however you'd like. And it's
not
an "abuse" of the functionality as long as you understand what it's
doing.
Let's say you have a matched pair of FCST = 55 and OBS = 80.
Setting cat_thresh = [ ge30&&le 70 ] in both the forecast and
observation
dictionaries would result in that point being an "event" in the
forecast
and a "non-event" in the observation. So that would be counted as a
false
alarm. FCST = 80 and OBS = 80 would be counted as a correct negative.
FCST = 80 and OBS = 55 would be counted as a miss.
So the observations > 70 won't be thrown out, it just affects what
category
they are put into. If the statistics still make sense to you defined
in
this way, then that's fine.
Another option would be setting the cat_thresh differently in the
forecast
and observation dictionaries...
fcst -> cat_thresh = [ ge30 ];
obs -> cat_thresh = [ ge30&&le 70 ];
That would change the how the events are defined in each field.
John
On Wed, Nov 9, 2016 at 11:45 AM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
> John,
> Thanks a lot for your explanation.
> I want to use cat_thresh like ge30&&le 70 rather than ge30 is
because
> I presume the values larger than 70 (like 999) will be a flag for
bad
> values.
> So, by using the flexibility of MET here, I can enforce a QC
restraint that
> values >70 will not be used.
> Hope this is not a abuse of MET related functions:).
> Regards,
> Ting
>
>
> On Wed, Nov 9, 2016 at 1:40 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Ting,
> >
> > Yes, cat_thresh = [ ge-10&&le70 ]; should work to define a
threshold for
> > categorical verification. However, that's a rather odd threshold
to
> > define. That means that anything between -10 and 70 is an "event"
while
> > anything outside that range is a "non-event". The difference here
is
> that
> > the "cnt_thresh" *FILTERS* the matched pair data prior to
computing
> > continuous statistics. The "cat_thresh" defines the event-
threshold for
> > populating a contingency table.
> >
> > It's probably confusing that we call them both "_thresh"...
instead,
> > perhaps we should have named "cnt_thresh" as "cnt_filter" instead.
> >
> > If you use that categorical event threshold, you'll end up, for
example,
> > with a CSI (critical success index) statistics that quantifies how
well
> > your forecast did predicting values between -10 and 70. Instead,
I
> suspect
> > you'd want something like:
> > cat_thresh = [ >=20, >=30, >=40 ];
> >
> > With this setting you'd get 3 CSI values in the output describing
how
> well
> > your forecast predicts reflectivity exceeding those thresholds.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> > On Wed, Nov 9, 2016 at 11:00 AM, Ting Lei - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > >
> > > Hi, John,
> > > Thanks a lot for your wonderful explanation. I need to study
them more
> > > carefully.
> > > An easy answer is : that, sure, I believe more flexible QC
functions in
> > the
> > > MET will be helpful
> > > for us at EMC. I expect more discussions will be helpful,
especially
> when
> > > MET is becoming more and more widely used here,
> > > One more question: is the form " cat_thresh = [ ge-10&&le70 ];"
> validate
> > to
> > > the MET?
> > > Regards,
> > > Ting
> > >
> > >
> > > On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Ting,
> > > >
> > > > I see you have questions about the cnt_thresh logic is
applied.
> > > >
> > > > The answer to your question is no, the cnt_thresh setting is
not
> > applied
> > > to
> > > > categorical verification.
> > > >
> > > > Here's how it works...
> > > >
> > > > - The cnt_thresh settings affect the MET output written to the
CNT
> > > > (continuous statistics) and SL1L2 (partial sums) line types
only.
> > > >
> > > > - The cat_thresh settings affect the MET output written to the
CTC
> > > > (contingency table counts), FHO (another version of the
contingency
> > table
> > > > counts), and the CTS (contingency table statistics) line
types.
> > > >
> > > > So an observation value of 80 would still show up as an event
in the
> > CTC
> > > > and CTS output lines.
> > > >
> > > > I see that really you'd like to limit your analysis (both
continuous
> > and
> > > > categorical) to using observations which fall between -10 and
70.
> > Since
> > > > you're running ascii2nc anyway, the easiest way to accomplish
this is
> > to
> > > > write a script yourself to filter the point observations prior
to
> > calling
> > > > ascii2nc. You could do this either by discarding observation
values
> > > > outside the desired range, or you could assign them a bad data
flag
> in
> > > the
> > > > "QC_String" column.
> > > >
> > > > There is still a way you can do this with the MET tools, but
it is
> more
> > > > difficult than what I described above. I'll outline it here
> anyway...
> > > >
> > > > (1) Configure Point-Stat so that it *ONLY* writes the matched
pair
> > (MPR)
> > > > output line, not statistics or partial sums.
> > > > (2) Run the STAT-Analysis tool to read those matched pair
lines,
> apply
> > > the
> > > > filtering criteria you'd like, and then compute the desired
> statistics.
> > > > The STAT-Analysis job would look something like this:
> > > >
> > > > stat_analysis -lookin point_stat_*.stat \
> > > > -job aggregate_stat -line_type MPR -out_line_type CTC \
> > > > -column_thresh OBS ge-10&&le70 \
> > > > -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> > > > -out_stat ctc_filtered.stat
> > > >
> > > > The job is "aggregate_stat" since we're reading MPR lines and
writing
> > > out a
> > > > different output type, CTC. The "-column_thresh" setting
applies
> your
> > > > filtering threshold to the data values in the input column
named
> "OBS".
> > > > The "-out_fcst_thresh" and "-out_obs_thresh" settings define
the
> > > threshold
> > > > for the output contingency table. And lastly, the "-out_stat"
> settings
> > > > writes the counts to an output .stat file.
> > > >
> > > > So this job reads in all MPR lines, ignores the ones where the
OBS
> > column
> > > > is outside the defined range, and uses the remaining lines to
> compute a
> > > > contingency table.
> > > >
> > > > Running the STAT-Analysis tool on MPR lines like this provides
the
> most
> > > > amount of flexibility that MET has to offer. However, it
usually
> isn't
> > > > feasible to write out the full set of individual matched pair
values
> > for
> > > > operational runs!
> > > >
> > > > I can see how we could enhance Point-Stat and Grid-Stat by
adding
> > another
> > > > filtering threshold option to do exactly what you're
describing. It
> > > > wouldn't be all that difficult to do. However, there isn't
really a
> > spot
> > > > for writing out that threshold information to the output
files.
> > Perhaps
> > > we
> > > > call it something like "qc_thresh"? We'd discard any matched
pairs
> > which
> > > > don't meet the defined "qc_thresh" criteria so they wouldn't
be used
> > for
> > > > continuous or categorical verification. Do you think that type
of
> > > > functionality would be useful more broadly at NCEP?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > > > >
> > > > > Hi, John,
> > > > > Sorry for re-openning this ticket.
> > > > > I still need some further clarifications.
> > > > > When using the cnt_thresh to choose the obs/fcst in a range
to be
> > used
> > > in
> > > > > the verification for continuous variables,
> > > > > will this control those for the categorical verification
like CAC?
> > > > > For example, the cat_thresh will control how the obs/fcst
will be
> > > > verified
> > > > > in several categories . Will the cnt_thresh take effect?
> > > > > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this
make MET
> > only
> > > > do
> > > > > verification like CAC over the whole obs pool with values
between
> -10
> > > and
> > > > > 70 while calculate them in the categories decided by the
> cat_thresh?
> > > > > Your further clarification is appreciated.
> > > > > Ting
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate <
> > > > > ting.lei at noaa.gov
> > > > > > wrote:
> > > > >
> > > > > > John,
> > > > > > Thanks again for your explanation.
> > > > > > That is great.
> > > > > > My question in this email has been perfectly answered.
> > > > > > Best Wishes,
> > > > > > Ting
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >> Ting,
> > > > > >>
> > > > > >> Yes, the cnt_thresh filtering option works for both Grid-
Stat
> and
> > > > > >> Point-Stat.
> > > > > >>
> > > > > >> The fcst -> cnt_thresh option filters the forecast
values. The
> > obs
> > > ->
> > > > > >> cnt_thresh option filters the observation values. And
the
> > cnt_logic
> > > > > >> option
> > > > > >> can be set to UNION, INTERSECTION, or SYMDIFF to specify
how the
> > > logic
> > > > > for
> > > > > >> how the forecast and observation thresholds should be
combined.
> > > > > >>
> > > > > >> For example... suppose you've defined cnt_thresh for
temperature
> > as
> > > > > >273.
> > > > > >> cnt_logic = UNION means the forecast **or** observation
value
> is >
> > > > 273.
> > > > > >> cnt_logic = INTERSECTION means the forecast **and**
observation
> > > values
> > > > > are
> > > > > >> >273.
> > > > > >> cnt_logic = SYMDIFF means the that one is >273 and the
other is
> > not.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA Affiliate
via
> RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
> >
> > > > > >> >
> > > > > >> > John,
> > > > > >> > Thanks a lot.
> > > > > >> > I need more time to study the first part about QC in
your
> > > > explanation.
> > > > > >> > However, it is great that the cnt_thresh can be used.
It can
> > > > > definitely
> > > > > >> > meet my requirement for being now.
> > > > > >> > Just a clarification is great. This function is
working for
> > both
> > > > > >> > Met-point and Met-grid, right?
> > > > > >> > Thanks.
> > > > > >> > Ting
> > > > > >> >
> > > > > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway via
RT <
> > > > > >> > met_help at ucar.edu
> > > > > >> > > wrote:
> > > > > >> >
> > > > > >> > > Ting,
> > > > > >> > >
> > > > > >> > > OK, there are two things I'd like to point out
regarding
> this
> > > > issue.
> > > > > >> > >
> > > > > >> > > First, the NetCDF point observation files that Point-
Stat
> > reads
> > > > > >> contain a
> > > > > >> > > slot for quality control information. Specifically,
the
> > NetCDF
> > > > > output
> > > > > >> > from
> > > > > >> > > ascii2nc, madis2nc, and pb2nc contain a variable
named
> > "obs_qty"
> > > > > which
> > > > > >> > is a
> > > > > >> > > string containing quality control info.
> > > > > >> > >
> > > > > >> > > If you're running ascii2nc, it's your job to format
the
> input
> > > > ascii
> > > > > >> data
> > > > > >> > > the expected way. If you have quality control info,
it
> should
> > > go
> > > > in
> > > > > >> > column
> > > > > >> > > 10, named "QC_String" in the ascii2nc usage
statement. MET
> > does
> > > > not
> > > > > >> do
> > > > > >> > any
> > > > > >> > > QC for you, but it can use the QC strings you've set.
> > > > > >> > >
> > > > > >> > > In the Point-Stat config file, you can set the
"obs_quality"
> > > entry
> > > > > to
> > > > > >> a
> > > > > >> > > list of one or more quality control strings to be
used. If
> > set,
> > > > > only
> > > > > >> > > observations whose quality control value appears in
that
> list
> > > will
> > > > > be
> > > > > >> > used.
> > > > > >> > >
> > > > > >> > > Second, you asked if there's a way to restrict the
> > observations
> > > > used
> > > > > >> by
> > > > > >> > > filtering their values, such as -10<obs<70. The
answer is
> > yes,
> > > at
> > > > > >> least
> > > > > >> > > for continuous statistics, like RMSE and ME. We call
this
> > > > > >> "conditional
> > > > > >> > > continuous verification" since we are conditioning
the
> > analysis
> > > on
> > > > > the
> > > > > >> > > input values.
> > > > > >> > >
> > > > > >> > > We set this up using the "cnt_thresh" and "cnt_logic"
> settings
> > > in
> > > > > the
> > > > > >> > > Point-Stat config file, such as:
> > > > > >> > >
> > > > > >> > > fcst = {
> > > > > >> > >
> > > > > >> > > field = [
> > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [
NA ]; }
> > > > > >> > > ];
> > > > > >> > >
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > obs = {
> > > > > >> > >
> > > > > >> > > field = [
> > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [
> > ge-10&&le70
> > > ];
> > > > > >> > > cnt_logic = INTERSECTION; }
> > > > > >> > > ];
> > > > > >> > >
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > Note that in the forecast I'm setting cnt_thresh to
NA,
> > meaning
> > > > that
> > > > > >> no
> > > > > >> > > filtering criteria will be applied to the forecast
value.
> But
> > > in
> > > > > the
> > > > > >> > > observation I used "ge-10&&le70" to define the
desired range
> > of
> > > > > >> > observation
> > > > > >> > > values.
> > > > > >> > >
> > > > > >> > > You can list as many cnt_thresh values as you'd like
in the
> > list
> > > > but
> > > > > >> the
> > > > > >> > > number of fcst settings must match the number of obs
> settings.
> > > > For
> > > > > >> > > example...
> > > > > >> > >
> > > > > >> > > fcst...
> > > > > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > > > > >> > >
> > > > > >> > > obs...
> > > > > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > > > > >> > >
> > > > > >> > > The above settings will compute continuous statistics
5
> > ways...
> > > > > using
> > > > > >> (1)
> > > > > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs >=
20 and
> > (5)
> > > > obs
> > > > > >> >= 30
> > > > > >> > >
> > > > > >> > > Hope that helps. Just let me know if you have more
> questions.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > John
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > >> > > met_help at ucar.edu> wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted
upon.
> > > > > >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > > > > >> > > > Queue: met_help
> > > > > >> > > > Subject: questions about QC for observation
> > > > > >> > > > Owner: Nobody
> > > > > >> > > > Requestors: ting.lei at noaa.gov
> > > > > >> > > > Status: new
> > > > > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > > >> ket/Display.html?id=78584
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Dear Mr. or Mrs.,
> > > > > >> > > > Thanks in advance for your help.
> > > > > >> > > > I'm wondering what methods can be used with MET to
screen
> > the
> > > > > >> > > observations
> > > > > >> > > > in the verification process.
> > > > > >> > > > I found obs_quality can help if the flag parameters
are in
> > the
> > > > obs
> > > > > >> > file.
> > > > > >> > > > However, if there are no such flags in my obs and I
know
> the
> > > > > >> > definitions
> > > > > >> > > of
> > > > > >> > > > missing values or other values like -99 assigned
with
> > special
> > > > > >> meaning,
> > > > > >> > > > how can I make those info with MET?
> > > > > >> > > > Or , for simplification, if I want to use the obs
in a
> range
> > > > like
> > > > > >> -10 <
> > > > > >> > > > obs<70.
> > > > > >> > > > Can I or ,if I can, How can I invoke such a
constraint in
> > the
> > > > MET
> > > > > >> > > process.
> > > > > >> > > > Your help is appreciated.
> > > > > >> > > > Best,
> > > > > >> > > > Ting
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: Ting Lei - NOAA Affiliate
Time: Wed Nov 09 12:12:56 2016
Hi, John,
Thanks lot for pointing out my mistakes. Sure, I will be careful on
how MET
will react to those input parameters.
I will see if I can use the option you suggested to attain my goal
with
dumping some resulted "fake counts".
Really appreciate your help.
Ting
On Wed, Nov 9, 2016 at 1:57 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Ting,
>
> You're welcome to set up the config file however you'd like. And
it's not
> an "abuse" of the functionality as long as you understand what it's
doing.
>
> Let's say you have a matched pair of FCST = 55 and OBS = 80.
>
> Setting cat_thresh = [ ge30&&le 70 ] in both the forecast and
observation
> dictionaries would result in that point being an "event" in the
forecast
> and a "non-event" in the observation. So that would be counted as a
false
> alarm. FCST = 80 and OBS = 80 would be counted as a correct
negative.
> FCST = 80 and OBS = 55 would be counted as a miss.
>
> So the observations > 70 won't be thrown out, it just affects what
category
> they are put into. If the statistics still make sense to you
defined in
> this way, then that's fine.
>
> Another option would be setting the cat_thresh differently in the
forecast
> and observation dictionaries...
> fcst -> cat_thresh = [ ge30 ];
> obs -> cat_thresh = [ ge30&&le 70 ];
>
> That would change the how the events are defined in each field.
>
> John
>
> On Wed, Nov 9, 2016 at 11:45 AM, Ting Lei - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> >
> > John,
> > Thanks a lot for your explanation.
> > I want to use cat_thresh like ge30&&le 70 rather than ge30 is
because
> > I presume the values larger than 70 (like 999) will be a flag for
bad
> > values.
> > So, by using the flexibility of MET here, I can enforce a QC
restraint
> that
> > values >70 will not be used.
> > Hope this is not a abuse of MET related functions:).
> > Regards,
> > Ting
> >
> >
> > On Wed, Nov 9, 2016 at 1:40 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Ting,
> > >
> > > Yes, cat_thresh = [ ge-10&&le70 ]; should work to define a
threshold
> for
> > > categorical verification. However, that's a rather odd
threshold to
> > > define. That means that anything between -10 and 70 is an
"event"
> while
> > > anything outside that range is a "non-event". The difference
here is
> > that
> > > the "cnt_thresh" *FILTERS* the matched pair data prior to
computing
> > > continuous statistics. The "cat_thresh" defines the event-
threshold
> for
> > > populating a contingency table.
> > >
> > > It's probably confusing that we call them both "_thresh"...
instead,
> > > perhaps we should have named "cnt_thresh" as "cnt_filter"
instead.
> > >
> > > If you use that categorical event threshold, you'll end up, for
> example,
> > > with a CSI (critical success index) statistics that quantifies
how well
> > > your forecast did predicting values between -10 and 70.
Instead, I
> > suspect
> > > you'd want something like:
> > > cat_thresh = [ >=20, >=30, >=40 ];
> > >
> > > With this setting you'd get 3 CSI values in the output
describing how
> > well
> > > your forecast predicts reflectivity exceeding those thresholds.
> > >
> > > Make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Nov 9, 2016 at 11:00 AM, Ting Lei - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
>
> > > >
> > > > Hi, John,
> > > > Thanks a lot for your wonderful explanation. I need to study
them
> more
> > > > carefully.
> > > > An easy answer is : that, sure, I believe more flexible QC
functions
> in
> > > the
> > > > MET will be helpful
> > > > for us at EMC. I expect more discussions will be helpful,
especially
> > when
> > > > MET is becoming more and more widely used here,
> > > > One more question: is the form " cat_thresh = [ ge-10&&le70
];"
> > validate
> > > to
> > > > the MET?
> > > > Regards,
> > > > Ting
> > > >
> > > >
> > > > On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Ting,
> > > > >
> > > > > I see you have questions about the cnt_thresh logic is
applied.
> > > > >
> > > > > The answer to your question is no, the cnt_thresh setting is
not
> > > applied
> > > > to
> > > > > categorical verification.
> > > > >
> > > > > Here's how it works...
> > > > >
> > > > > - The cnt_thresh settings affect the MET output written to
the CNT
> > > > > (continuous statistics) and SL1L2 (partial sums) line types
only.
> > > > >
> > > > > - The cat_thresh settings affect the MET output written to
the CTC
> > > > > (contingency table counts), FHO (another version of the
contingency
> > > table
> > > > > counts), and the CTS (contingency table statistics) line
types.
> > > > >
> > > > > So an observation value of 80 would still show up as an
event in
> the
> > > CTC
> > > > > and CTS output lines.
> > > > >
> > > > > I see that really you'd like to limit your analysis (both
> continuous
> > > and
> > > > > categorical) to using observations which fall between -10
and 70.
> > > Since
> > > > > you're running ascii2nc anyway, the easiest way to
accomplish this
> is
> > > to
> > > > > write a script yourself to filter the point observations
prior to
> > > calling
> > > > > ascii2nc. You could do this either by discarding
observation
> values
> > > > > outside the desired range, or you could assign them a bad
data flag
> > in
> > > > the
> > > > > "QC_String" column.
> > > > >
> > > > > There is still a way you can do this with the MET tools, but
it is
> > more
> > > > > difficult than what I described above. I'll outline it here
> > anyway...
> > > > >
> > > > > (1) Configure Point-Stat so that it *ONLY* writes the
matched pair
> > > (MPR)
> > > > > output line, not statistics or partial sums.
> > > > > (2) Run the STAT-Analysis tool to read those matched pair
lines,
> > apply
> > > > the
> > > > > filtering criteria you'd like, and then compute the desired
> > statistics.
> > > > > The STAT-Analysis job would look something like this:
> > > > >
> > > > > stat_analysis -lookin point_stat_*.stat \
> > > > > -job aggregate_stat -line_type MPR -out_line_type CTC \
> > > > > -column_thresh OBS ge-10&&le70 \
> > > > > -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> > > > > -out_stat ctc_filtered.stat
> > > > >
> > > > > The job is "aggregate_stat" since we're reading MPR lines
and
> writing
> > > > out a
> > > > > different output type, CTC. The "-column_thresh" setting
applies
> > your
> > > > > filtering threshold to the data values in the input column
named
> > "OBS".
> > > > > The "-out_fcst_thresh" and "-out_obs_thresh" settings define
the
> > > > threshold
> > > > > for the output contingency table. And lastly, the "-
out_stat"
> > settings
> > > > > writes the counts to an output .stat file.
> > > > >
> > > > > So this job reads in all MPR lines, ignores the ones where
the OBS
> > > column
> > > > > is outside the defined range, and uses the remaining lines
to
> > compute a
> > > > > contingency table.
> > > > >
> > > > > Running the STAT-Analysis tool on MPR lines like this
provides the
> > most
> > > > > amount of flexibility that MET has to offer. However, it
usually
> > isn't
> > > > > feasible to write out the full set of individual matched
pair
> values
> > > for
> > > > > operational runs!
> > > > >
> > > > > I can see how we could enhance Point-Stat and Grid-Stat by
adding
> > > another
> > > > > filtering threshold option to do exactly what you're
describing.
> It
> > > > > wouldn't be all that difficult to do. However, there isn't
really
> a
> > > spot
> > > > > for writing out that threshold information to the output
files.
> > > Perhaps
> > > > we
> > > > > call it something like "qc_thresh"? We'd discard any
matched pairs
> > > which
> > > > > don't meet the defined "qc_thresh" criteria so they wouldn't
be
> used
> > > for
> > > > > continuous or categorical verification. Do you think that
type of
> > > > > functionality would be useful more broadly at NCEP?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > > > > >
> > > > > > Hi, John,
> > > > > > Sorry for re-openning this ticket.
> > > > > > I still need some further clarifications.
> > > > > > When using the cnt_thresh to choose the obs/fcst in a
range to be
> > > used
> > > > in
> > > > > > the verification for continuous variables,
> > > > > > will this control those for the categorical verification
like
> CAC?
> > > > > > For example, the cat_thresh will control how the obs/fcst
will be
> > > > > verified
> > > > > > in several categories . Will the cnt_thresh take effect?
> > > > > > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will this
make
> MET
> > > only
> > > > > do
> > > > > > verification like CAC over the whole obs pool with values
between
> > -10
> > > > and
> > > > > > 70 while calculate them in the categories decided by the
> > cat_thresh?
> > > > > > Your further clarification is appreciated.
> > > > > > Ting
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA Affiliate
<
> > > > > > ting.lei at noaa.gov
> > > > > > > wrote:
> > > > > >
> > > > > > > John,
> > > > > > > Thanks again for your explanation.
> > > > > > > That is great.
> > > > > > > My question in this email has been perfectly answered.
> > > > > > > Best Wishes,
> > > > > > > Ting
> > > > > > >
> > > > > > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >> Ting,
> > > > > > >>
> > > > > > >> Yes, the cnt_thresh filtering option works for both
Grid-Stat
> > and
> > > > > > >> Point-Stat.
> > > > > > >>
> > > > > > >> The fcst -> cnt_thresh option filters the forecast
values.
> The
> > > obs
> > > > ->
> > > > > > >> cnt_thresh option filters the observation values. And
the
> > > cnt_logic
> > > > > > >> option
> > > > > > >> can be set to UNION, INTERSECTION, or SYMDIFF to
specify how
> the
> > > > logic
> > > > > > for
> > > > > > >> how the forecast and observation thresholds should be
> combined.
> > > > > > >>
> > > > > > >> For example... suppose you've defined cnt_thresh for
> temperature
> > > as
> > > > > > >273.
> > > > > > >> cnt_logic = UNION means the forecast **or** observation
value
> > is >
> > > > > 273.
> > > > > > >> cnt_logic = INTERSECTION means the forecast **and**
> observation
> > > > values
> > > > > > are
> > > > > > >> >273.
> > > > > > >> cnt_logic = SYMDIFF means the that one is >273 and the
other
> is
> > > not.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA
Affiliate via
> > RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=78584
> > >
> > > > > > >> >
> > > > > > >> > John,
> > > > > > >> > Thanks a lot.
> > > > > > >> > I need more time to study the first part about QC in
your
> > > > > explanation.
> > > > > > >> > However, it is great that the cnt_thresh can be used.
It can
> > > > > > definitely
> > > > > > >> > meet my requirement for being now.
> > > > > > >> > Just a clarification is great. This function is
working
> for
> > > both
> > > > > > >> > Met-point and Met-grid, right?
> > > > > > >> > Thanks.
> > > > > > >> > Ting
> > > > > > >> >
> > > > > > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway
via RT <
> > > > > > >> > met_help at ucar.edu
> > > > > > >> > > wrote:
> > > > > > >> >
> > > > > > >> > > Ting,
> > > > > > >> > >
> > > > > > >> > > OK, there are two things I'd like to point out
regarding
> > this
> > > > > issue.
> > > > > > >> > >
> > > > > > >> > > First, the NetCDF point observation files that
Point-Stat
> > > reads
> > > > > > >> contain a
> > > > > > >> > > slot for quality control information.
Specifically, the
> > > NetCDF
> > > > > > output
> > > > > > >> > from
> > > > > > >> > > ascii2nc, madis2nc, and pb2nc contain a variable
named
> > > "obs_qty"
> > > > > > which
> > > > > > >> > is a
> > > > > > >> > > string containing quality control info.
> > > > > > >> > >
> > > > > > >> > > If you're running ascii2nc, it's your job to format
the
> > input
> > > > > ascii
> > > > > > >> data
> > > > > > >> > > the expected way. If you have quality control
info, it
> > should
> > > > go
> > > > > in
> > > > > > >> > column
> > > > > > >> > > 10, named "QC_String" in the ascii2nc usage
statement.
> MET
> > > does
> > > > > not
> > > > > > >> do
> > > > > > >> > any
> > > > > > >> > > QC for you, but it can use the QC strings you've
set.
> > > > > > >> > >
> > > > > > >> > > In the Point-Stat config file, you can set the
> "obs_quality"
> > > > entry
> > > > > > to
> > > > > > >> a
> > > > > > >> > > list of one or more quality control strings to be
used.
> If
> > > set,
> > > > > > only
> > > > > > >> > > observations whose quality control value appears in
that
> > list
> > > > will
> > > > > > be
> > > > > > >> > used.
> > > > > > >> > >
> > > > > > >> > > Second, you asked if there's a way to restrict the
> > > observations
> > > > > used
> > > > > > >> by
> > > > > > >> > > filtering their values, such as -10<obs<70. The
answer is
> > > yes,
> > > > at
> > > > > > >> least
> > > > > > >> > > for continuous statistics, like RMSE and ME. We
call this
> > > > > > >> "conditional
> > > > > > >> > > continuous verification" since we are conditioning
the
> > > analysis
> > > > on
> > > > > > the
> > > > > > >> > > input values.
> > > > > > >> > >
> > > > > > >> > > We set this up using the "cnt_thresh" and
"cnt_logic"
> > settings
> > > > in
> > > > > > the
> > > > > > >> > > Point-Stat config file, such as:
> > > > > > >> > >
> > > > > > >> > > fcst = {
> > > > > > >> > >
> > > > > > >> > > field = [
> > > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [
NA ];
> }
> > > > > > >> > > ];
> > > > > > >> > >
> > > > > > >> > > }
> > > > > > >> > >
> > > > > > >> > > obs = {
> > > > > > >> > >
> > > > > > >> > > field = [
> > > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh = [
> > > ge-10&&le70
> > > > ];
> > > > > > >> > > cnt_logic = INTERSECTION; }
> > > > > > >> > > ];
> > > > > > >> > >
> > > > > > >> > > }
> > > > > > >> > >
> > > > > > >> > > Note that in the forecast I'm setting cnt_thresh to
NA,
> > > meaning
> > > > > that
> > > > > > >> no
> > > > > > >> > > filtering criteria will be applied to the forecast
value.
> > But
> > > > in
> > > > > > the
> > > > > > >> > > observation I used "ge-10&&le70" to define the
desired
> range
> > > of
> > > > > > >> > observation
> > > > > > >> > > values.
> > > > > > >> > >
> > > > > > >> > > You can list as many cnt_thresh values as you'd
like in
> the
> > > list
> > > > > but
> > > > > > >> the
> > > > > > >> > > number of fcst settings must match the number of
obs
> > settings.
> > > > > For
> > > > > > >> > > example...
> > > > > > >> > >
> > > > > > >> > > fcst...
> > > > > > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > > > > > >> > >
> > > > > > >> > > obs...
> > > > > > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > > > > > >> > >
> > > > > > >> > > The above settings will compute continuous
statistics 5
> > > ways...
> > > > > > using
> > > > > > >> (1)
> > > > > > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs
>= 20
> and
> > > (5)
> > > > > obs
> > > > > > >> >= 30
> > > > > > >> > >
> > > > > > >> > > Hope that helps. Just let me know if you have more
> > questions.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > John
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > >> > > met_help at ucar.edu> wrote:
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was acted
upon.
> > > > > > >> > > > Transaction: Ticket created by ting.lei at noaa.gov
> > > > > > >> > > > Queue: met_help
> > > > > > >> > > > Subject: questions about QC for observation
> > > > > > >> > > > Owner: Nobody
> > > > > > >> > > > Requestors: ting.lei at noaa.gov
> > > > > > >> > > > Status: new
> > > > > > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > > > >> ket/Display.html?id=78584
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Dear Mr. or Mrs.,
> > > > > > >> > > > Thanks in advance for your help.
> > > > > > >> > > > I'm wondering what methods can be used with MET
to
> screen
> > > the
> > > > > > >> > > observations
> > > > > > >> > > > in the verification process.
> > > > > > >> > > > I found obs_quality can help if the flag
parameters are
> in
> > > the
> > > > > obs
> > > > > > >> > file.
> > > > > > >> > > > However, if there are no such flags in my obs and
I know
> > the
> > > > > > >> > definitions
> > > > > > >> > > of
> > > > > > >> > > > missing values or other values like -99 assigned
with
> > > special
> > > > > > >> meaning,
> > > > > > >> > > > how can I make those info with MET?
> > > > > > >> > > > Or , for simplification, if I want to use the obs
in a
> > range
> > > > > like
> > > > > > >> -10 <
> > > > > > >> > > > obs<70.
> > > > > > >> > > > Can I or ,if I can, How can I invoke such a
constraint
> in
> > > the
> > > > > MET
> > > > > > >> > > process.
> > > > > > >> > > > Your help is appreciated.
> > > > > > >> > > > Best,
> > > > > > >> > > > Ting
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: questions about QC for observation
From: John Halley Gotway
Time: Wed Nov 09 12:19:19 2016
Ting,
Sure, and I did create a MET development task to add that first level
filtering capability (e.g. qc_thresh) to do what you're after more
easily
in the MET tools. But I can't make any promises on when we'll be able
to
add it.
We do have a monthly telecon with NCEP MET/METViewer users and could
discuss the priority of that task during the next telecon.
But I'll go ahead and resolve this ticket now.
Thanks,
John
On Wed, Nov 9, 2016 at 12:12 PM, Ting Lei - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
>
> Hi, John,
> Thanks lot for pointing out my mistakes. Sure, I will be careful on
how MET
> will react to those input parameters.
> I will see if I can use the option you suggested to attain my goal
with
> dumping some resulted "fake counts".
> Really appreciate your help.
> Ting
>
>
> On Wed, Nov 9, 2016 at 1:57 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Ting,
> >
> > You're welcome to set up the config file however you'd like. And
it's
> not
> > an "abuse" of the functionality as long as you understand what
it's
> doing.
> >
> > Let's say you have a matched pair of FCST = 55 and OBS = 80.
> >
> > Setting cat_thresh = [ ge30&&le 70 ] in both the forecast and
observation
> > dictionaries would result in that point being an "event" in the
forecast
> > and a "non-event" in the observation. So that would be counted as
a
> false
> > alarm. FCST = 80 and OBS = 80 would be counted as a correct
negative.
> > FCST = 80 and OBS = 55 would be counted as a miss.
> >
> > So the observations > 70 won't be thrown out, it just affects what
> category
> > they are put into. If the statistics still make sense to you
defined in
> > this way, then that's fine.
> >
> > Another option would be setting the cat_thresh differently in the
> forecast
> > and observation dictionaries...
> > fcst -> cat_thresh = [ ge30 ];
> > obs -> cat_thresh = [ ge30&&le 70 ];
> >
> > That would change the how the events are defined in each field.
> >
> > John
> >
> > On Wed, Nov 9, 2016 at 11:45 AM, Ting Lei - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > >
> > > John,
> > > Thanks a lot for your explanation.
> > > I want to use cat_thresh like ge30&&le 70 rather than ge30 is
because
> > > I presume the values larger than 70 (like 999) will be a flag
for bad
> > > values.
> > > So, by using the flexibility of MET here, I can enforce a QC
restraint
> > that
> > > values >70 will not be used.
> > > Hope this is not a abuse of MET related functions:).
> > > Regards,
> > > Ting
> > >
> > >
> > > On Wed, Nov 9, 2016 at 1:40 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Ting,
> > > >
> > > > Yes, cat_thresh = [ ge-10&&le70 ]; should work to define a
threshold
> > for
> > > > categorical verification. However, that's a rather odd
threshold to
> > > > define. That means that anything between -10 and 70 is an
"event"
> > while
> > > > anything outside that range is a "non-event". The difference
here is
> > > that
> > > > the "cnt_thresh" *FILTERS* the matched pair data prior to
computing
> > > > continuous statistics. The "cat_thresh" defines the event-
threshold
> > for
> > > > populating a contingency table.
> > > >
> > > > It's probably confusing that we call them both "_thresh"...
instead,
> > > > perhaps we should have named "cnt_thresh" as "cnt_filter"
instead.
> > > >
> > > > If you use that categorical event threshold, you'll end up,
for
> > example,
> > > > with a CSI (critical success index) statistics that quantifies
how
> well
> > > > your forecast did predicting values between -10 and 70.
Instead, I
> > > suspect
> > > > you'd want something like:
> > > > cat_thresh = [ >=20, >=30, >=40 ];
> > > >
> > > > With this setting you'd get 3 CSI values in the output
describing how
> > > well
> > > > your forecast predicts reflectivity exceeding those
thresholds.
> > > >
> > > > Make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Nov 9, 2016 at 11:00 AM, Ting Lei - NOAA Affiliate via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584 >
> > > > >
> > > > > Hi, John,
> > > > > Thanks a lot for your wonderful explanation. I need to
study them
> > more
> > > > > carefully.
> > > > > An easy answer is : that, sure, I believe more flexible QC
> functions
> > in
> > > > the
> > > > > MET will be helpful
> > > > > for us at EMC. I expect more discussions will be helpful,
> especially
> > > when
> > > > > MET is becoming more and more widely used here,
> > > > > One more question: is the form " cat_thresh = [ ge-10&&le70
];"
> > > validate
> > > > to
> > > > > the MET?
> > > > > Regards,
> > > > > Ting
> > > > >
> > > > >
> > > > > On Wed, Nov 9, 2016 at 12:20 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Ting,
> > > > > >
> > > > > > I see you have questions about the cnt_thresh logic is
applied.
> > > > > >
> > > > > > The answer to your question is no, the cnt_thresh setting
is not
> > > > applied
> > > > > to
> > > > > > categorical verification.
> > > > > >
> > > > > > Here's how it works...
> > > > > >
> > > > > > - The cnt_thresh settings affect the MET output written to
the
> CNT
> > > > > > (continuous statistics) and SL1L2 (partial sums) line
types only.
> > > > > >
> > > > > > - The cat_thresh settings affect the MET output written to
the
> CTC
> > > > > > (contingency table counts), FHO (another version of the
> contingency
> > > > table
> > > > > > counts), and the CTS (contingency table statistics) line
types.
> > > > > >
> > > > > > So an observation value of 80 would still show up as an
event in
> > the
> > > > CTC
> > > > > > and CTS output lines.
> > > > > >
> > > > > > I see that really you'd like to limit your analysis (both
> > continuous
> > > > and
> > > > > > categorical) to using observations which fall between -10
and 70.
> > > > Since
> > > > > > you're running ascii2nc anyway, the easiest way to
accomplish
> this
> > is
> > > > to
> > > > > > write a script yourself to filter the point observations
prior to
> > > > calling
> > > > > > ascii2nc. You could do this either by discarding
observation
> > values
> > > > > > outside the desired range, or you could assign them a bad
data
> flag
> > > in
> > > > > the
> > > > > > "QC_String" column.
> > > > > >
> > > > > > There is still a way you can do this with the MET tools,
but it
> is
> > > more
> > > > > > difficult than what I described above. I'll outline it
here
> > > anyway...
> > > > > >
> > > > > > (1) Configure Point-Stat so that it *ONLY* writes the
matched
> pair
> > > > (MPR)
> > > > > > output line, not statistics or partial sums.
> > > > > > (2) Run the STAT-Analysis tool to read those matched pair
lines,
> > > apply
> > > > > the
> > > > > > filtering criteria you'd like, and then compute the
desired
> > > statistics.
> > > > > > The STAT-Analysis job would look something like this:
> > > > > >
> > > > > > stat_analysis -lookin point_stat_*.stat \
> > > > > > -job aggregate_stat -line_type MPR -out_line_type CTC \
> > > > > > -column_thresh OBS ge-10&&le70 \
> > > > > > -out_fcst_thresh ge35 -out_obs_thresh ge35 \
> > > > > > -out_stat ctc_filtered.stat
> > > > > >
> > > > > > The job is "aggregate_stat" since we're reading MPR lines
and
> > writing
> > > > > out a
> > > > > > different output type, CTC. The "-column_thresh" setting
applies
> > > your
> > > > > > filtering threshold to the data values in the input column
named
> > > "OBS".
> > > > > > The "-out_fcst_thresh" and "-out_obs_thresh" settings
define the
> > > > > threshold
> > > > > > for the output contingency table. And lastly, the "-
out_stat"
> > > settings
> > > > > > writes the counts to an output .stat file.
> > > > > >
> > > > > > So this job reads in all MPR lines, ignores the ones where
the
> OBS
> > > > column
> > > > > > is outside the defined range, and uses the remaining lines
to
> > > compute a
> > > > > > contingency table.
> > > > > >
> > > > > > Running the STAT-Analysis tool on MPR lines like this
provides
> the
> > > most
> > > > > > amount of flexibility that MET has to offer. However, it
usually
> > > isn't
> > > > > > feasible to write out the full set of individual matched
pair
> > values
> > > > for
> > > > > > operational runs!
> > > > > >
> > > > > > I can see how we could enhance Point-Stat and Grid-Stat by
adding
> > > > another
> > > > > > filtering threshold option to do exactly what you're
describing.
> > It
> > > > > > wouldn't be all that difficult to do. However, there
isn't
> really
> > a
> > > > spot
> > > > > > for writing out that threshold information to the output
files.
> > > > Perhaps
> > > > > we
> > > > > > call it something like "qc_thresh"? We'd discard any
matched
> pairs
> > > > which
> > > > > > don't meet the defined "qc_thresh" criteria so they
wouldn't be
> > used
> > > > for
> > > > > > continuous or categorical verification. Do you think that
type of
> > > > > > functionality would be useful more broadly at NCEP?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 8, 2016 at 11:03 AM, Ting Lei - NOAA Affiliate
via
> RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78584
> >
> > > > > > >
> > > > > > > Hi, John,
> > > > > > > Sorry for re-openning this ticket.
> > > > > > > I still need some further clarifications.
> > > > > > > When using the cnt_thresh to choose the obs/fcst in a
range to
> be
> > > > used
> > > > > in
> > > > > > > the verification for continuous variables,
> > > > > > > will this control those for the categorical verification
like
> > CAC?
> > > > > > > For example, the cat_thresh will control how the
obs/fcst will
> be
> > > > > > verified
> > > > > > > in several categories . Will the cnt_thresh take
effect?
> > > > > > > Namely, if I have cnt_thresh = [ ge-10&&le70 ]; will
this make
> > MET
> > > > only
> > > > > > do
> > > > > > > verification like CAC over the whole obs pool with
values
> between
> > > -10
> > > > > and
> > > > > > > 70 while calculate them in the categories decided by the
> > > cat_thresh?
> > > > > > > Your further clarification is appreciated.
> > > > > > > Ting
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 3, 2016 at 6:14 PM, Ting Lei - NOAA
Affiliate <
> > > > > > > ting.lei at noaa.gov
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > John,
> > > > > > > > Thanks again for your explanation.
> > > > > > > > That is great.
> > > > > > > > My question in this email has been perfectly answered.
> > > > > > > > Best Wishes,
> > > > > > > > Ting
> > > > > > > >
> > > > > > > > On Thu, Nov 3, 2016 at 5:12 PM, John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >> Ting,
> > > > > > > >>
> > > > > > > >> Yes, the cnt_thresh filtering option works for both
> Grid-Stat
> > > and
> > > > > > > >> Point-Stat.
> > > > > > > >>
> > > > > > > >> The fcst -> cnt_thresh option filters the forecast
values.
> > The
> > > > obs
> > > > > ->
> > > > > > > >> cnt_thresh option filters the observation values.
And the
> > > > cnt_logic
> > > > > > > >> option
> > > > > > > >> can be set to UNION, INTERSECTION, or SYMDIFF to
specify how
> > the
> > > > > logic
> > > > > > > for
> > > > > > > >> how the forecast and observation thresholds should be
> > combined.
> > > > > > > >>
> > > > > > > >> For example... suppose you've defined cnt_thresh for
> > temperature
> > > > as
> > > > > > > >273.
> > > > > > > >> cnt_logic = UNION means the forecast **or**
observation
> value
> > > is >
> > > > > > 273.
> > > > > > > >> cnt_logic = INTERSECTION means the forecast **and**
> > observation
> > > > > values
> > > > > > > are
> > > > > > > >> >273.
> > > > > > > >> cnt_logic = SYMDIFF means the that one is >273 and
the other
> > is
> > > > not.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Thu, Nov 3, 2016 at 2:36 PM, Ting Lei - NOAA
Affiliate
> via
> > > RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=78584
> > > >
> > > > > > > >> >
> > > > > > > >> > John,
> > > > > > > >> > Thanks a lot.
> > > > > > > >> > I need more time to study the first part about QC
in your
> > > > > > explanation.
> > > > > > > >> > However, it is great that the cnt_thresh can be
used. It
> can
> > > > > > > definitely
> > > > > > > >> > meet my requirement for being now.
> > > > > > > >> > Just a clarification is great. This function is
working
> > for
> > > > both
> > > > > > > >> > Met-point and Met-grid, right?
> > > > > > > >> > Thanks.
> > > > > > > >> > Ting
> > > > > > > >> >
> > > > > > > >> > On Thu, Nov 3, 2016 at 4:31 PM, John Halley Gotway
via RT
> <
> > > > > > > >> > met_help at ucar.edu
> > > > > > > >> > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Ting,
> > > > > > > >> > >
> > > > > > > >> > > OK, there are two things I'd like to point out
regarding
> > > this
> > > > > > issue.
> > > > > > > >> > >
> > > > > > > >> > > First, the NetCDF point observation files that
> Point-Stat
> > > > reads
> > > > > > > >> contain a
> > > > > > > >> > > slot for quality control information.
Specifically, the
> > > > NetCDF
> > > > > > > output
> > > > > > > >> > from
> > > > > > > >> > > ascii2nc, madis2nc, and pb2nc contain a variable
named
> > > > "obs_qty"
> > > > > > > which
> > > > > > > >> > is a
> > > > > > > >> > > string containing quality control info.
> > > > > > > >> > >
> > > > > > > >> > > If you're running ascii2nc, it's your job to
format the
> > > input
> > > > > > ascii
> > > > > > > >> data
> > > > > > > >> > > the expected way. If you have quality control
info, it
> > > should
> > > > > go
> > > > > > in
> > > > > > > >> > column
> > > > > > > >> > > 10, named "QC_String" in the ascii2nc usage
statement.
> > MET
> > > > does
> > > > > > not
> > > > > > > >> do
> > > > > > > >> > any
> > > > > > > >> > > QC for you, but it can use the QC strings you've
set.
> > > > > > > >> > >
> > > > > > > >> > > In the Point-Stat config file, you can set the
> > "obs_quality"
> > > > > entry
> > > > > > > to
> > > > > > > >> a
> > > > > > > >> > > list of one or more quality control strings to be
used.
> > If
> > > > set,
> > > > > > > only
> > > > > > > >> > > observations whose quality control value appears
in that
> > > list
> > > > > will
> > > > > > > be
> > > > > > > >> > used.
> > > > > > > >> > >
> > > > > > > >> > > Second, you asked if there's a way to restrict
the
> > > > observations
> > > > > > used
> > > > > > > >> by
> > > > > > > >> > > filtering their values, such as -10<obs<70. The
answer
> is
> > > > yes,
> > > > > at
> > > > > > > >> least
> > > > > > > >> > > for continuous statistics, like RMSE and ME. We
call
> this
> > > > > > > >> "conditional
> > > > > > > >> > > continuous verification" since we are
conditioning the
> > > > analysis
> > > > > on
> > > > > > > the
> > > > > > > >> > > input values.
> > > > > > > >> > >
> > > > > > > >> > > We set this up using the "cnt_thresh" and
"cnt_logic"
> > > settings
> > > > > in
> > > > > > > the
> > > > > > > >> > > Point-Stat config file, such as:
> > > > > > > >> > >
> > > > > > > >> > > fcst = {
> > > > > > > >> > >
> > > > > > > >> > > field = [
> > > > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh =
[ NA
> ];
> > }
> > > > > > > >> > > ];
> > > > > > > >> > >
> > > > > > > >> > > }
> > > > > > > >> > >
> > > > > > > >> > > obs = {
> > > > > > > >> > >
> > > > > > > >> > > field = [
> > > > > > > >> > > { name = "REFC"; level=[ "L0" ]; cnt_thresh =
[
> > > > ge-10&&le70
> > > > > ];
> > > > > > > >> > > cnt_logic = INTERSECTION; }
> > > > > > > >> > > ];
> > > > > > > >> > >
> > > > > > > >> > > }
> > > > > > > >> > >
> > > > > > > >> > > Note that in the forecast I'm setting cnt_thresh
to NA,
> > > > meaning
> > > > > > that
> > > > > > > >> no
> > > > > > > >> > > filtering criteria will be applied to the
forecast
> value.
> > > But
> > > > > in
> > > > > > > the
> > > > > > > >> > > observation I used "ge-10&&le70" to define the
desired
> > range
> > > > of
> > > > > > > >> > observation
> > > > > > > >> > > values.
> > > > > > > >> > >
> > > > > > > >> > > You can list as many cnt_thresh values as you'd
like in
> > the
> > > > list
> > > > > > but
> > > > > > > >> the
> > > > > > > >> > > number of fcst settings must match the number of
obs
> > > settings.
> > > > > > For
> > > > > > > >> > > example...
> > > > > > > >> > >
> > > > > > > >> > > fcst...
> > > > > > > >> > > cnt_thresh = [ NA, NA, NA, NA, NA ];
> > > > > > > >> > >
> > > > > > > >> > > obs...
> > > > > > > >> > > cnt_thresh = [ NA, ge-10, ge10, ge20, ge30 ];
> > > > > > > >> > >
> > > > > > > >> > > The above settings will compute continuous
statistics 5
> > > > ways...
> > > > > > > using
> > > > > > > >> (1)
> > > > > > > >> > > no filtering (2) obs >= -10 (3) obs >= 10 (4) obs
>= 20
> > and
> > > > (5)
> > > > > > obs
> > > > > > > >> >= 30
> > > > > > > >> > >
> > > > > > > >> > > Hope that helps. Just let me know if you have
more
> > > questions.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > John
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Nov 3, 2016 at 11:10 AM, Ting Lei - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > > >> > > met_help at ucar.edu> wrote:
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > Thu Nov 03 11:10:27 2016: Request 78584 was
acted
> upon.
> > > > > > > >> > > > Transaction: Ticket created by
ting.lei at noaa.gov
> > > > > > > >> > > > Queue: met_help
> > > > > > > >> > > > Subject: questions about QC for
observation
> > > > > > > >> > > > Owner: Nobody
> > > > > > > >> > > > Requestors: ting.lei at noaa.gov
> > > > > > > >> > > > Status: new
> > > > > > > >> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > > > > >> ket/Display.html?id=78584
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > Dear Mr. or Mrs.,
> > > > > > > >> > > > Thanks in advance for your help.
> > > > > > > >> > > > I'm wondering what methods can be used with MET
to
> > screen
> > > > the
> > > > > > > >> > > observations
> > > > > > > >> > > > in the verification process.
> > > > > > > >> > > > I found obs_quality can help if the flag
parameters
> are
> > in
> > > > the
> > > > > > obs
> > > > > > > >> > file.
> > > > > > > >> > > > However, if there are no such flags in my obs
and I
> know
> > > the
> > > > > > > >> > definitions
> > > > > > > >> > > of
> > > > > > > >> > > > missing values or other values like -99
assigned with
> > > > special
> > > > > > > >> meaning,
> > > > > > > >> > > > how can I make those info with MET?
> > > > > > > >> > > > Or , for simplification, if I want to use the
obs in a
> > > range
> > > > > > like
> > > > > > > >> -10 <
> > > > > > > >> > > > obs<70.
> > > > > > > >> > > > Can I or ,if I can, How can I invoke such a
constraint
> > in
> > > > the
> > > > > > MET
> > > > > > > >> > > process.
> > > > > > > >> > > > Your help is appreciated.
> > > > > > > >> > > > Best,
> > > > > > > >> > > > Ting
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list