[Met_help] [rt.rap.ucar.edu #46857] History for sanity check on continuous stats

Fri Jun 17 14:11:18 MDT 2011

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello.

Just wanting a sanity check on how continuous statistics are computed.  E.g., fbar and obar are computed after determining the pairs that mutually satisfy the thresholds and masking set out in the config file, right?  Such that, if I have file with -9999 as bad values, a mask over just land of CONUS, and ask to compare obs and forecast with a forecast threshold of gt0.0, only points that are *both* obs > 0 and fcst > 0, not -9999, and within the confines of the mask will be counted towards the sum of fbar and obar and the count, n, right?

Thanks,
Mike

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #46857] sanity check on continuous stats
From: John Halley Gotway
Time: Fri May 13 14:47:51 2011

Mike,

I created a met_help ticket for this so that we can track the MET-
related support we provide.

To answer your question, actually no, that is not how the continuous
statistics are handled.  The "fcst_thresh" and "obs_thresh" settings
are applied when computing categorical statistics - not
continuous ones.  For a given verification task, Grid-Stat and Point-
Stat build up a number of matched fcst/obs pairs for some variable
over some verification region.  The continuous statistics are
computed over ALL of those matched pairs.  No filtering criteria is
applied.  That's why the fcst_thresh and obs_thresh columns in the
output CNT line type contain NA.

fcst_thresh and obs_thresh are used to define contingency tables from
which contingency table statistics are derived.

Doing the type of filtering of matched pairs your mentioning would be
a reasonable thing to do.  And that type of functionality has been
requested in the past, but it is not available in the current
version of MET.

If you're using Point-Stat, there is one work-around you could try for
a couple of cases, but it would be cumbersome to do in a systematic
way.  You could:
(1) Run Point-Stat and dump out the matched pair (MPR) line type.
(2) Run a STAT-Analysis job something like the following:
  -job aggregate_stat -line_type MPR -out_line_type CNT -column_min
fcst 0.01 -column_min obs 0.01

That would have the effect of looking through the MPR lines, only keep
ones who's forecast and observation values are at least 0.01, and then
use those to recompute continuous statistics.

Would adding this sort of filtering functionality directly to the
Point-Stat and Grid-Stat tools be useful for what you guys are doing
at AFWA?

Hope that helps clarify.

John Halley Gotway
met_help at ucar.edu

On 05/13/2011 02:33 PM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> Fri May 13 14:33:55 2011: Request 46857 was acted upon.
> Transaction: Ticket created by johnhg
>        Queue: met_help
>      Subject: sanity check on continuous stats
>        Owner: johnhg
>   Requestors: Michael.Shaw.Ctr at offutt.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=46857 >
>
>
> Hello.
>
> Just wanting a sanity check on how continuous statistics are
computed.  E.g., fbar and obar are computed after determining the
pairs that mutually satisfy the thresholds and masking set out in the
config file, right?  Such that, if I have file with -9999 as bad
values, a mask over just land of CONUS, and ask to compare obs and
forecast with a forecast threshold of gt0.0, only points that are
*both* obs > 0 and fcst > 0, not -9999, and within the confines of the
mask will be counted towards the sum of fbar and obar and the count,
n, right?
>
> Thanks,
> Mike

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #46857] sanity check on continuous stats
From: John Halley Gotway
Time: Fri May 13 15:01:03 2011

Mike,

If either the forecast or observed value is bad data, that matched
pair is thrown out.  So to be more precise, the continuous statistics
are computed over all matched pairs falling the verification
region where both the forecast and observed values contain valid data.

John

On 05/13/2011 02:57 PM, Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
> rything outside of prescribed masks, etc.  E.g., if I had 1440x721
.25 degree global arrays of both forecast and obs fields of some 2d
variable each containing -9999s, and I asked grid_stat to compute
stats for just conus and had specified thresholds and a mask in the
config file, you're saying that there should be 1440x721=1038240 pairs
used to compute the continuous stats?  I thought the masking and bad
number might be able to get handled (and were) via continuous stats in
grid_stat.
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #46857] sanity check on continuous stats
From: John Halley Gotway
Time: Fri May 13 15:09:23 2011

Mike,

Yes, that's correct.  By verification region, I mean the spatial area
you've defined in the config file using the "mask_grid" and/or
"mask_poly" settings.  The ability to *filter* the matched pairs by
their actual values is not possible without code changes.

To me, a logical extension of the functionality would be to use the
fcst_thresh and obs_thresh settings that are currently used to define
contingency tables and also use them in this filtering
capacity.  So for each verification task, the number of output CNT
lines (and SL1L2 lines) would increase from 1 to the number of
thresholds you've specified.  And the contents of the fcst_thresh and
obs_thresh columns in the output would reflect the filtering
thresholds that were applied.  Am I correct in thinking that's what
you were assuming was happening?

John

On 05/13/2011 03:03 PM, Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
> "Verification region" never being defined by the mask, while
currently impossible (without code changes) to use the thresholds to
further confine the included pairs to "hits" (e.g., obs and fcst > 0
or something like that)?
>
> ______________________________________________
>
> Michael Shaw, Contractor
> SAIC/NASA Support Scientist
> michael.shaw.ctr at offutt.af.mil
> 16WS Environmental Characterization
> HQ AFWA-SAIC
> 101 Nelson Drive
> Offutt AFB, NE 68113-1023
> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Friday, May 13, 2011 4:01 PM
> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
> Subject: Re: [rt.rap.ucar.edu #46857] sanity check on continuous
stats
>
> Mike,
>
> If either the forecast or observed value is bad data, that matched
pair is thrown out.  So to be more precise, the continuous statistics
are computed over all matched pairs falling the verification
> region where both the forecast and observed values contain valid
data.
>
> John
>
> On 05/13/2011 02:57 PM, Shaw, Michael J CTR USAF AFWA 16 WS/WXE
wrote:
>> rything outside of prescribed masks, etc.  E.g., if I had 1440x721
.25 degree global arrays of both forecast and obs fields of some 2d
variable each containing -9999s, and I asked grid_stat to compute
stats for just conus and had specified thresholds and a mask in the
config file, you're saying that there should be 1440x721=1038240 pairs
used to compute the continuous stats?  I thought the masking and bad
number might be able to get handled (and were) via continuous stats in
grid_stat.
>>

------------------------------------------------