[Met_help] [rt.rap.ucar.edu #68198] History for Multi-category statistical verification measure

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

Thanks for the wonderful verification tool. I would like to enquire about
multi-category verification measures pls.

There is a paragraph (4.2.2) in the documentation which states:

Measures for categorical variables
*Categorical verification statistics are used to evaluate forecasts that
are in the form of a discrete set of categories rather than on a continuous
scale. Currently, Point-Stat computes categorical statistics for variables
in two categories. In future versions, MET will include the capability to
compute measures for multi-category forecasts.*

At the moment I understand one can only create 'yes' or 'no' categories.

Would like to know if the feature underlined will come soon or later, or
maybe already in the latest version of the product?

Thanks.


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Multi-category statistical verification measure
From: John Halley Gotway
Time: Tue Jul 15 10:33:01 2014

Zan,

Good news.  The software is more up to date than the documentation!
The
current release of MET, v4.1, does already support multi-category
contingency tables.  They can be produced by the point_stat and
grid_stat
tools.  The output is stored in the multi-category contingency table
line
type (MCTC) and the multi-category contingency table statistics line
type
(MCTS).

Actually, there is an example of producing this output in one of the
test
scripts distributed with MET.  Look in
METv4.1/scripts/test_grid_stat.sh at
the 3rd of the 4 calls to grid_stat.  Here's an excerpt from the
config
file it uses:
   METv4.1/scripts/config/GridStatConfig_APCP_24:
...
   field = [
      {
        name       = "APCP";
        level      = [ "A24" ];
        cat_thresh = [ >0.0, >5.0, >10.0 ];
      }
   ];
...
   mctc   = BOTH;
   mcts   = BOTH;

Notice that we've specified 3 entries for the categorical threshold
(cat_thresh) and they're all of the same type (>).  Also, the mctc and
mcts
output types are turned on.

These 3 thresholds are used to define 4 categories.  Less than 0,
between 0
and 5, between 5 and 10, and greater than 10.  Grid-Stat then
constructs a
4x4 contingency table, and you can see it's output in this file:
   METv4.1/out/grid_stat/grid_stat_APCP_24_240000L_20050808_000000V_mctc.txt

There are only a handful of statistics that can be computed on a 4x4
contingency table (accuracy, hanssen-kuipers discriminant, heidke
skill
score, and the gerrity score), but they can be found this this file:
   METv4.1/out/grid_stat/grid_stat_APCP_24_240000L_20050808_000000V_mcts.txt

Just like stat_analysis can read and aggregate CTC lines together, it
can
do the same for MCTC lines.

Hope that helps.

Thanks,
John Halley Gotway
met_help at ucar.edu




On Tue, Jul 15, 2014 at 2:26 AM, Zan Rahmat via RT <met_help at ucar.edu>
wrote:

>
> Tue Jul 15 02:26:05 2014: Request 68198 was acted upon.
> Transaction: Ticket created by raizan.rahmat at gmail.com
>        Queue: met_help
>      Subject: Multi-category statistical verification measure
>        Owner: Nobody
>   Requestors: raizan.rahmat at gmail.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68198 >
>
>
> Hi,
>
> Thanks for the wonderful verification tool. I would like to enquire
about
> multi-category verification measures pls.
>
> There is a paragraph (4.2.2) in the documentation which states:
>
> Measures for categorical variables
> *Categorical verification statistics are used to evaluate forecasts
that
> are in the form of a discrete set of categories rather than on a
continuous
> scale. Currently, Point-Stat computes categorical statistics for
variables
> in two categories. In future versions, MET will include the
capability to
> compute measures for multi-category forecasts.*
>
> At the moment I understand one can only create 'yes' or 'no'
categories.
>
> Would like to know if the feature underlined will come soon or
later, or
> maybe already in the latest version of the product?
>
> Thanks.
>
>

------------------------------------------------