[Met_help] mode_analysis question

[John Halley Gotway]

Jonathan,

Actually, the counts in those files have nothing to do with the matching that was performed by MODE.  The intent of the file is to give you an easy way of seeing what you'd get with a traditional
verification of the input fields as opposed to an object-based verification.  Let me point you to the first paragraph of section 6.3.3 of the MET User's Guide for a description of these lines.  Here's
some more detail based on the contents of the "FIELD" column.

FIELD Column...
(1) RAW: Apply any masking of bad data, grid masking, or polygon matching to the raw fields.  Threshold the raw fields using the "fcst_conv_thresh" and "obs_conv_thresh" config values to define 0/1
fields.  Compute the contingency table counts by comparing these 0/1 fields grid point by grid point.

(2) FILTER: Apply any masking of bad data, grid masking, or polygon matching to the raw fields.  In addition, apply the "fcst_raw_thresh" and "obs_raw_thresh" to filter out any additional values.
Then use the "fcst_conv_thresh" and "obs_conv_thresh" config values to define 0/1 fields, and compute a contingency table from them.

(3) OBJECT: Once objects have been defined in the forecast and observation fields, consider any grid point inside of an object to have a value of 1 and any grid point outside of an object to have a
value of 0.  Compute the contingency table counts by comparing these 0/1 fields grid point by grid point.

So really object matching has nothing to do with it.

Make sense?

John


For the RAW line:

Case, Jonathan (MSFC-VP61)[Other] wrote:
> Hi John,
> 
> You hit the nail on the head with #1.  I saw the counts in the MODE contingency files (*_cts.txt) and was wondering if we could summarize those in mode_analysis (to which the answer is no as you said).  
> 
> It seems like we could fairly easily read in the contingency counts in shell script, with a little management of filenames.
> 
> I also wonder how those contingency files are calculated in the MODE output?  Is any forecast object that matches an observed object considered a "hit", and then the individual grid points within that object are added to the total of FY_OY?  And similarly with the unmatched forecast/observed objects, are their grid points summed to be FY_ON and FN_OY?  It seems like it could get a bit unclear when dealing with FCST/OBS objects that don't overlap when trying to determine FY_ON, FN_OY, and FN_ON.  Perhaps you could embellish for me so I can make more sense about the MODE contingency stat files?
> 
> Thanks for your help,
> Jonathan
> 
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Monday, April 27, 2009 8:43 AM
>> To: Case, Jonathan (MSFC-VP61)[Other]
>> Cc: met_help at mailman.ucar.edu
>> Subject: Re: [Met_help] mode_analysis question
>>
>> Jonathan,
>>
>> I'm a little unclear exactly what you're asking here.  It could be one
>> of two things... but the answer to both is no.
>>
>> The MODE-Analysis tool currently performs 2 types of jobs by reading
>> the MODE object statistics files:
>> (1) A "summary" job in which you select one or more MODE output columns
>> of interest and it summarizes the data in those columns.
>> (2) A "bycase" job that produces summary information for each MODE run
>> consisting of counts and areas of matched and unmatched objects.
>>
>> Let me try to understand exactly what you're asking though.  Are you
>> asking:
>> (1) Can MODE-Analysis read those MODE contingency table statistics
>> files (*_cts.txt) output and aggregate them across many cases?
>> (2) Can MODE-Analysis read the MODE object statistics file, treat the
>> matched/unmatched object counts or areas as the elements of a
>> contingency table and derive some contingency table statistics based
>> on that "object matching" contingency table?
>>
>> Like I said, the answer to both is no.  People here have done (2) here
>> before, but it's still kind of an open question how best to use the
>> object counts or areas to populate the elements of a
>> contingency table.  You have to decide how to define the hits, misses,
>> false alarms, and correct nulls.  And doing so gets a bit messy -
>> especially defining the correct nulls.
>>
>> Thanks,
>> John
>>
>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>> Dear Met_help,
>>>
>>> Does mode_analysis let the user summarize contingency statistics of
>> the paired objects, or can it only summarize the various object
>> attributes?
>>> Thanks for the help,
>>> Jonathan
>>>
>>> ***********************************************************
>>> Jonathan Case, ENSCO, Inc.
>>> Aerospace Sciences & Engineering Division
>>> Short-term Prediction Research and Transition Center
>>> 320 Sparkman Drive, Room 3062
>>> Huntsville, AL 35805-1912
>>> Voice: (256) 961-7504   Fax: (256) 961-7788
>>> Emails: Jonathan.Case-1 at nasa.gov
>>>              case.jonathan at ensco.com
>>> ***********************************************************
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>> ---
>>> _______________________________________________
>>> Met_help mailing list
>>> Met_help at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/met_help