[Met_help] Asking for a little help

[John Halley Gotway]

Kefeng,

No, nothing is moved around.  Please take a look at the image I've attached to this message.  This is the first page of the PostScript output from one of the MODE test cases you ran after you
installed MET.

In this example, we're running MODE on a fcst and obs of 24-hourly accumulated precipitation.  Please take a look at the text below the images at the lines for "Simple/M/U", "Area", and "Area M/U".
These lines give information about the matched and unmatched object counts and areas.

In this case, we see that there are 4 forecast objects and 6 observation objects.  Of the 4 forecast objects, 2 of them are matched (red and green), and 2 of them are unmatched (blue).  Of the 6
observation objects, 2 are matched (red and green), and 4 of them are unmatched.

Computing CSI for this one case using object counts would be:
hits         = (#fcst matched + #obs matched)/2 = (2 + 2)/2 = 2
misses       =  #unmatched obs = 4
false alarms =  #unmatched fcst = 2
CSI = (2)/(2 + 4 + 2) = 0.25

However, we could instead look at the matched/unmatched areas which are in the lines for "Area" and "Area M/U" (the M/U is for matched/unmatched).

Computing CSI for this one case using object areas would be:
hits         = (area fcst matched + area matched)/2 = (779 + 730)/2 = 754.5
misses       =  area unmatched obs = 300
false alarms =  area unmatched fcst = 41
CSI = (754.5)/(754.5 + 300 + 41) = 0.6887

Does that make sense how it's being computed?  Here we've computed the CSI for a single case.  You could also compute it over multiple cases by keeping a running total of the hits, misses, and false
alarms, and then use the sums to compute CSI.

John

Zhu wrote:
> Hi, John:
> 
>      Thanks very much! Maybe i can took some time to familar with the R package. A more question is how to define 'count of fcst matches' or 'area of fcst matches'?  Does it move the object to the other (that what i saw from the 'ps' file of mode output) then count the number of matches or calculate the area of matches?
>  
>      Thanks!
> 
> Kefeng
> 
> 
> 2009-06-16 
> 
> 
> 
> Zhu,Kefeng 
> 
> 
> 
> ???: John Halley Gotway 
> ????: 2009-06-16  08:46:15 
> ???: Zhu, Kefeng 
> ??: met_help 
> ??: Re: Asking for a little help 
>  
> Kefeng,
> Good questions.
> I have a question for you.  Do you by any chance have "R" available on your system?  It's an open-source statistical scripting language.  If so, try running the R code I've attached on some of your
> MODE output files.  Here's how you'd run it:
> Rscript mode_summary.R mode_*_obj.txt
> (where "mode_*_obj.txt" grabs one or more MODE output object stats files)
> This Rscript summarizes one or more MODE output files.  I've also attached an ASCII file named "mode_summary_example.txt" that gives sample output from this script.
> OK, getting to your question... here's how CSI is defined:
> CSI = (hits)/(hits + misses + false alarms)
> You can actually compute CSI on MODE output one of two ways, with regards to object counts or object areas:
> (1) With regards to object counts:
> hits         = (count of fcst matches + count of obs matches)/2
> misses       = (count of unmatched obs objects)
> false alarms = (count of unmatched fcst objects)
> (2) With regards to object areas:
> hits         = (area of fcst matches + area of obs matches)/2
> misses       = (area of unmatched obs objects)
> false alarms = (area of unmatched fcst objects)
> Hopefully, somewhere in the MWR article, it'll say which method was used for computing this type of object-based CSI.  All of these counts and areas should show up in the output of "mode_summary.R".
> If you're comfortable using R, I'd suggest modifying the script to compute these CSI values for you.
> Personally, I like computing CSI using areas, but I think you'll find that this leads to very large values of CSI that are very often near 1.
> Hope that helps.
> John
> Zhu,Kefeng wrote:
>> Hi, John:
>>
>>
>>       Do you have any detail documents about how to calculate the Object-based scores, like CSI? Although i have read through the MET user guider and the article of objected-based published in Monthly Weatherly Review, I'm still confused about the concept of matches, misses and false alarms of the two objects. Since the two objects are not in the same location and differ in size and grid numbers, I'm wondering how to define matches, misses and false alarms?
>>
>>       Thanks a lot!
>>
>> Kefeng  
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mode_page1.gif
Type: image/gif
Size: 57404 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/met_help/attachments/20090616/a1d4422e/attachment-0001.gif