[Met_help] [rt.rap.ucar.edu #88819] History for HIRA Model Ob Pair Files

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

John, yesterday I asked the question on should the ob field against a certain threshold be a 0 or 1 or the ob value in point stat HIRA?  Through some testing, it appears stat_analysis interprets the ob value as a 0 or 1.  If the ob value is one or greater, it is a 1 in the Brier Score calculation and if it is less than 1 it is a 0 in the brier score calculation.  I am sending my configuration file via ARL Safe.  I have thresholded the observation data in the configuration file so shouldn't a 1 or 0 appear in the model ob pair files observation column?  Is my syntax correct?  Since all my mdlob_pair files have the actual ob values in the ob column, the brier scores are coming out way too high.  I also included a test model ob pair file I used to determine this (it was cut down from my output stat file).  I also included a typical stat file created by the HIRA process.

Bob


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: HIRA Model Ob Pair Files
From: John Halley Gotway
Time: Fri Feb 08 12:37:11 2019

Bob,

Thanks for sending sample data via ARL SAFE.  I was able to grab the
sample
.stat files and your Point-Stat config file.

One point of clarification.  You've listed the same variable 3 times,
changing the single categorical threshold for each task:




*   field = [      { name = "VIS"; level = "Z2"; cat_thresh = [<=.5];
},      { name = "VIS"; level = "Z2"; cat_thresh = [<=1]; },      {
name =
"VIS"; level = "Z2"; cat_thresh = [<=3]; }   ];*

This tells Point-Stat to process 3 separate verification tasks.  Since
you're requesting MPR output (mpr = BOTH;) you get the same set of 23
matched pairs dumped out 3 times.  If you instead define it as a
single
task with 3 categorical thresholds, the raw MPR values should only be
dumped once:


*   field = [      { name = "VIS"; level = "Z2"; cat_thresh = [ <=.5,
<=1,
<=3 ]; }   ];*

Looking in the MPR lines you sent, here's some relevant columns:
FCST_VAR VX_MASK INTERP_MTHD  INTERP_PNTS FCST_THRESH OBS_THRESH
LINE_TYPE
OBS_SID FCST     OBS
VIS      FULL    NEAREST      1           NA          NA         MPR
5470124 15.29708 3.72902
VIS      FULL    NBRHD_SQUARE 1           <=.5        <=.5       MPR
5470124 0        3.72902
VIS      FULL    NBRHD_SQUARE 25          <=.5        <=.5       MPR
    5470124
0        3.72902
VIS      FULL    NBRHD_SQUARE 1           <=1         <=1        MPR
5470124 0        3.72902
VIS      FULL    NBRHD_SQUARE 25          <=1         <=1        MPR
5470124 0        3.72902
VIS      FULL    NBRHD_SQUARE 1           <=3         <=3        MPR
5470124 0        3.72902
VIS      FULL    NBRHD_SQUARE 25          <=3         <=3        MPR
5470124 0        3.72902

For the station named "5470124", we have a forecast/obs matched pair
of
(15.297, 3.729).  When computing the fractional coverage over a
neighborhood of size 1, we should get a value of either 0 or 1.  For a
neighborhood of size 25 (5x5) box, we should get a forecast value
between 0
and 1, indicating how often the event occurred.  In this data, that
forecast fractional coverage value is always 0.  That indicates that
the
2-m visibility was never below 3, 1, or 0.5 in any of the forecast
grid
points nearby this observation station.

The OBS column contains the actual observation value... not 0 or 1.
The
HiRA methodology is used to convert a scalar field into probabilities
(by
applying a threshold and a neighborhood size).  When Point-Stat dumps
the
MPR line type for probabilities, it writes FCST = forecast probability
and
OBS = actual observation value.

For example, you can run a Stat-Analysis job to read MPR input lines
and do
probabilistic verification, like this:
   -job aggregate_stat \
   -line_type MPR \
   -out_line_type PCT \ (Nx2 Probabilistic contingency table)
   -out_fcst_thresh ==0.1 \ (defines forecast probability bins)
   -out_obs_thresh lt3 \ (defines the event threshold to the
observation
values)

So it looks to me like Point-Stat is producing the output data it was
designed to compute.

Hope this helps clarify.

Thanks,
John


On Thu, Feb 7, 2019 at 10:28 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Thu Feb 07 10:27:39 2019: Request 88819 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: HIRA Model Ob Pair Files
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88819 >
>
>
> John, yesterday I asked the question on should the ob field against
a
> certain threshold be a 0 or 1 or the ob value in point stat HIRA?
Through
> some testing, it appears stat_analysis interprets the ob value as a
0 or
> 1.  If the ob value is one or greater, it is a 1 in the Brier Score
> calculation and if it is less than 1 it is a 0 in the brier score
> calculation.  I am sending my configuration file via ARL Safe.  I
have
> thresholded the observation data in the configuration file so
shouldn't a 1
> or 0 appear in the model ob pair files observation column?  Is my
syntax
> correct?  Since all my mdlob_pair files have the actual ob values in
the ob
> column, the brier scores are coming out way too high.  I also
included a
> test model ob pair file I used to determine this (it was cut down
from my
> output stat file).  I also included a typical stat file created by
the HIRA
> process.
>
> Bob
>
>

------------------------------------------------