[Met_help] [rt.rap.ucar.edu #45009] History for Object Count Weighting
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Mon Mar 7 14:03:01 MST 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi guys, In the documentation for METv3.0 on page 11-2, there is a discussion of the need to weight mode statistics by area size. I am currently calculating Hits and Misses using the proscribed method of counting matched and unmatched observation and forecast objects. What method do you recommend to weight these counts by the object area size? I am currently using a modified version of a R program you provided to get the object counts.
Again, please email directly so I can get the response.
Thanks Again for Great Support
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45009] Object Count Weighting
From: John Halley Gotway
Time: Wed Mar 02 14:26:41 2011
Bob,
Good question. Unfortunately, I don't have a definitive answer for
you. When we've generated derived statistics from the MODE objects in
our past work, we've usually done it both ways - based on
object counts and also on object areas.
Conceptually, I think the idea of weighting by area rather than counts
is preferable - to me it seems more intuitive and more similar to a
traditional grid-to-grid approach in that way. Practically
speaking though, we've found that the large area of the large objects
tend to swamp the statistics, and we end up with nearly perfect
scores. For example, we often see probability of detection values
very close to 1 when we weight by area and much lower when we weight
by object count. So ultimately, it's up to you to crunch the numbers
and make sense of the results.
As for *HOW* to actually compute this, I believe the numbers you need
are already being dumped out by that Rscript. Here's the analysis
scripts page for MET:
http://www.dtcenter.org/met/users/downloads/analysis_scripts.php
I assume you're using a variant of the "mode_summary.R" script. On
that page, you should see a link for "sample_output" from that script
(http://www.dtcenter.org/met/users/downloads/Rscripts/mode_summary.out).
Presumably, you're deriving statistics using the counts of
matched/unmatched objects:
Number of Single Fcst Objects = 8
Number of Matched Single Fcst Objects = 8
Number of Unmatched Single Fcst Objects = 0
Number of Single Obs Objects = 12
Number of Matched Single Obs Objects = 8
Number of Unmatched Single Obs Objects = 4
You could repeat those computations for the areas by just substituting
in the corresponding area measures:
Area of Single Fcst Objects = 622
Area of Matched Single Fcst Objects = 622
Area of Unmatched Single Fcst Objects = 0
Area of Single Obs Objects = 2623
Area of Matched Single Obs Objects = 2380
Area of Unmatched Single Obs Objects = 243
I've copied Tressa on this message in case she has any additional
suggestions for you.
Hope that helps.
Thanks,
John
On 03/02/2011 01:58 PM, RAL HelpDesk {for Craig, Robert J Civ USAF
AFWA 16 WS/WXN} wrote:
>
> Wed Mar 02 13:58:12 2011: Request 45009 was acted upon.
> Transaction: Ticket created by craigr at offutt.af.mil
> Queue: met_help
> Subject: Object Count Weighting
> Owner: Nobody
> Requestors: craigr at offutt.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45009 >
>
>
> Hi guys, In the documentation for METv3.0 on page 11-2, there is a
discussion of the need to weight mode statistics by area size. I am
currently calculating Hits and Misses using the proscribed method of
counting matched and unmatched observation and forecast objects. What
method do you recommend to weight these counts by the object area
size? I am currently using a modified version of a R program you
provided to get the object counts.
>
> Again, please email directly so I can get the response.
>
> Thanks Again for Great Support
>
> Bob Craig
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45009] Object Count Weighting
From: John Halley Gotway
Time: Mon Mar 07 13:50:53 2011
Bob,
Just following up on this met-help question you had. Was the
explanation clear, or did more questions arise?
Thanks,
John
On 03/02/2011 02:26 PM, John Halley Gotway wrote:
> Bob,
>
> Good question. Unfortunately, I don't have a definitive answer for
you. When we've generated derived statistics from the MODE objects in
our past work, we've usually done it both ways - based on
> object counts and also on object areas.
>
> Conceptually, I think the idea of weighting by area rather than
counts is preferable - to me it seems more intuitive and more similar
to a traditional grid-to-grid approach in that way. Practically
> speaking though, we've found that the large area of the large
objects tend to swamp the statistics, and we end up with nearly
perfect scores. For example, we often see probability of detection
values
> very close to 1 when we weight by area and much lower when we weight
by object count. So ultimately, it's up to you to crunch the numbers
and make sense of the results.
>
> As for *HOW* to actually compute this, I believe the numbers you
need are already being dumped out by that Rscript. Here's the
analysis scripts page for MET:
> http://www.dtcenter.org/met/users/downloads/analysis_scripts.php
> I assume you're using a variant of the "mode_summary.R" script. On
that page, you should see a link for "sample_output" from that script
>
(http://www.dtcenter.org/met/users/downloads/Rscripts/mode_summary.out).
>
> Presumably, you're deriving statistics using the counts of
matched/unmatched objects:
> Number of Single Fcst Objects = 8
> Number of Matched Single Fcst Objects = 8
> Number of Unmatched Single Fcst Objects = 0
> Number of Single Obs Objects = 12
> Number of Matched Single Obs Objects = 8
> Number of Unmatched Single Obs Objects = 4
>
> You could repeat those computations for the areas by just
substituting in the corresponding area measures:
> Area of Single Fcst Objects = 622
> Area of Matched Single Fcst Objects = 622
> Area of Unmatched Single Fcst Objects = 0
> Area of Single Obs Objects = 2623
> Area of Matched Single Obs Objects = 2380
> Area of Unmatched Single Obs Objects = 243
>
> I've copied Tressa on this message in case she has any additional
suggestions for you.
>
> Hope that helps.
>
> Thanks,
> John
>
> On 03/02/2011 01:58 PM, RAL HelpDesk {for Craig, Robert J Civ USAF
AFWA 16 WS/WXN} wrote:
>>
>> Wed Mar 02 13:58:12 2011: Request 45009 was acted upon.
>> Transaction: Ticket created by craigr at offutt.af.mil
>> Queue: met_help
>> Subject: Object Count Weighting
>> Owner: Nobody
>> Requestors: craigr at offutt.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45009 >
>>
>>
>> Hi guys, In the documentation for METv3.0 on page 11-2, there is a
discussion of the need to weight mode statistics by area size. I am
currently calculating Hits and Misses using the proscribed method of
counting matched and unmatched observation and forecast objects. What
method do you recommend to weight these counts by the object area
size? I am currently using a modified version of a R program you
provided to get the object counts.
>>
>> Again, please email directly so I can get the response.
>>
>> Thanks Again for Great Support
>>
>> Bob Craig
>>
>>
------------------------------------------------
More information about the Met_help
mailing list