[Met_help] [rt.rap.ucar.edu #45901] History for Re:point_stat

Tue Apr 19 08:53:47 MDT 2011

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear Sir/Madam,

I would like to inquire about the forecast threshold and observation threshold
settings in the configuration file for the point_stat utility. If I were to use
point_stat to compute the Gilbert Skill Score for precipitation threshold of a
particular range 10 - 20 mm, how would I set the range? I can set for discreet
ranges like le10 for less than or equal 10mm and so forth but how do I set the
config file to calculate the GSS for a particular range .

The second question is I tried calculating the GSS for rainfall > 100mm , for
48 hour accumulated rainfall forecast assuming the initial date is on the 24th
of a particular month , 48 hours will be verified against the 26th rainfall and
it yields a particular value but when I verified rainfall at initial time 25th
for 24 hours accumulated rainfall the GSS score is NA, is this because there
was no rainfall amount in the observation? If so, the 48 hour forecast on the
24 should also say NA. Please comment.

Thank you very much,

Regards,

Kumar
NWP Developement Section
Malaysian Meteorological Department

�

----------------------
MMD WebMail
http://www.met.gov.my

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #45901] Re:point_stat
From: John Halley Gotway
Time: Thu Apr 07 11:51:48 2011

Kumar,

I gave your question some thought, and really I think you're asking
two different types of questions:
   (1) First, you're asking a technical question about how to
configure MET to compute the statistics in the way you want.
   (2) Second, you may be looking for some statistical advice on how
you should be performing your analysis.

I'll answer the first part, and for the second part, I'll refer you to
Tressa Fowler, our resident statistician for MET.

Regarding your first question, about computing GSS... Gilbert Skill
Score is a statistic that's computed from a 2x2 contingency table.
And in MET, the only way of computing a 2x2 contingency table is
by picking a single threshold to define your event.  You'd like to
define an "event" using two thresholds, rather than one, and that
isn't supported directly in the Point-Stat tool.  Your question is
a very reasonable one, and this issue has been raised in the past, but
has never been a high enough priority to require enhancing the code.

That being said, you could consider 2 options:
  (1) First, MET does include support for multi-category contingency
tables.  For example, suppose you set "fcst_thresh[] = [ "ge10.0",
"ge20.0"];".  If you requested MCTC or MCTS output in the
"output_flag" parameter, MET will use those thresholds to define a 3x3
contingency table, and will dump out the appropriate stats for that
table.  Perhaps those statistics would be of interest to you.
  (2) Second, you can always dump out the raw matched pairs from
Point-Stat (MPR line type).  Then, you could read those matched pairs
into another tool to perform further analysis.  Where I work, we
use the R package for this type of statistical analysis.

For your second question about GSS for APCP > 100mm, I'll need a
little more detail to understand your situation.  When you see an
output CTS line and that line contains an NA value for GSS, that
usually means that it was not mathematically possible to compute GSS.
If you look closely at the definition of GSS, you'll see that it's
computed using n, hits, misses, and false alarms.  I believe
that if hits = misses = false alarms = 0, then GSS ends up being NA
since you'd be dividing by zero.  Please take a look at the
corresponding CTC line and see if you're in that case.  If all of the
matched pairs fall into the correct negative (FN_ON) column, then GSS
will be undefined.

If that doesn't answer the question, I'd need to look at some of your
data to really understand what's going on.  I'd need to see the
forecast, observation, and configuration files that demonstrate
the problem your having.

Hopefully that helps.  Please let us know if you have further
questions about how to run MET and/or need advice about what
statistics to look at.

Thanks,
John Halley Gotway
met_help at ucar.edu

On 04/07/2011 10:27 AM, RAL HelpDesk {for Kumarenthiran} wrote:
>
> Thu Apr 07 10:27:56 2011: Request 45901 was acted upon.
> Transaction: Ticket created by kumar at met.gov.my
>        Queue: met_help
>      Subject: Re:point_stat
>        Owner: Nobody
>   Requestors: kumar at met.gov.my
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45901 >
>
>
> Dear Sir/Madam,
>
> I would like to inquire about the forecast threshold and observation
threshold
> settings in the configuration file for the point_stat utility. If I
were to use
> point_stat to compute the Gilbert Skill Score for precipitation
threshold of a
> particular range 10 - 20 mm, how would I set the range? I can set
for discreet
> ranges like le10 for less than or equal 10mm and so forth but how do
I set the
> config file to calculate the GSS for a particular range .
>
> The second question is I tried calculating the GSS for rainfall >
100mm , for
> 48 hour accumulated rainfall forecast assuming the initial date is
on the 24th
> of a particular month , 48 hours will be verified against the 26th
rainfall and
> it yields a particular value but when I verified rainfall at initial
time 25th
> for 24 hours accumulated rainfall the GSS score is NA, is this
because there
> was no rainfall amount in the observation? If so, the 48 hour
forecast on the
> 24 should also say NA. Please comment.
>
>
> Thank you very much,
>
> Regards,
>
> Kumar
> NWP Developement Section
> Malaysian Meteorological Department
>
> �
>
> ----------------------
> MMD WebMail
> http://www.met.gov.my
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.

------------------------------------------------
Subject: Re:point_stat
From: Tressa Fowler
Time: Thu Apr 07 14:47:06 2011

Hi Kumar,

I would urge you to use caution in applying categorical statistics to
event ranges such as this. Certainly, it makes sense to do a
conditional analysis, and only look at cases with precip between 10
and 20 mm using continuous statistics. For categorical statistics,
however, you may end up with some nonsense. Consider the following
example, for 'events' defined as precip between 10 and 20 mm with non-
events being all other amounts of precip (<10 or >20). If you have a
forecast of no precip and an observation of 25 mm of precip, your
contingency table will count this event as a correctly forecast non-
event. Most people would instead consider this a 'miss'. I am not sure
that skill scores based on these counts would be informative.

Please let me know if you have any further questions.

Tressa

On Thu Apr 07 10:27:56 2011, kumar at met.gov.my wrote:
> Dear Sir/Madam,
>
> I would like to inquire about the forecast threshold and observation
> threshold
> settings in the configuration file for the point_stat utility. If I
> were to use
> point_stat to compute the Gilbert Skill Score for precipitation
> threshold of a
> particular range 10 - 20 mm, how would I set the range? I can set
for
> discreet
> ranges like le10 for less than or equal 10mm and so forth but how do
I
> set the
> config file to calculate the GSS for a particular range .
>
> The second question is I tried calculating the GSS for rainfall >
> 100mm , for
> 48 hour accumulated rainfall forecast assuming the initial date is
on
> the 24th
> of a particular month , 48 hours will be verified against the 26th
> rainfall and
> it yields a particular value but when I verified rainfall at initial
> time 25th
> for 24 hours accumulated rainfall the GSS score is NA, is this
because
> there
> was no rainfall amount in the observation? If so, the 48 hour
forecast
> on the
> 24 should also say NA. Please comment.
>
>
> Thank you very much,
>
> Regards,
>
> Kumar
> NWP Developement Section
> Malaysian Meteorological Department
>
> �
>
> ----------------------
> MMD WebMail
> http://www.met.gov.my
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.

------------------------------------------------