[Met_help] MODE Advice: [Fwd: Re: Measuring model skill in mode_analysis]
John Halley Gotway
johnhg at rap.ucar.edu
Wed May 13 12:16:27 MDT 2009
Jonathan,
For MODE-Analysis the "P50" column gives you the median of the valid values in that column of data. So when summarizing the "INTEREST" column, you're getting the median value of the data in the
interest column. You're NOT get the median of the MAXIMUM interest by object. You're just getting the median of all the values. Make sense?
Let me mention one other thing. The MODE-Analysis and Stat-Analysis tools were intentionally designed to not be very sophisticated. They just do exactly as they're instructed by the user without
making many assumptions. So they'll use all of the MODE lines that meet the filtering criteria you've set for the job. But since in METv2.0, we're computing interest values for pairs of simple
objects AND pairs of composites, that median you're getting is computed over ALL of those - which I don't think you'd want.
Instead I'd suggest running your job twice, once with "-pair -simple" and once with "-pair -cluster" arguments. The first will summarize the interest values for pairs of simple objects while the
second will do so for pairs of clusters.
We encourage users to make utilize the "-dump_row" option to look at the lines that went into their analysis jobs to verify that the job really is being run over the set of data they want.
Thanks,
John
Case, Jonathan (MSFC-VP61)[Other] wrote:
> Thanks John,
>
> I will look into installing "R" on our system and running your R script.
>
> Out of curiosity, what is the INTEREST distribution that I'm looking at when I run "mode_analysis" on the INTEREST column stored in the object text files? If I pull out the "P50" value, then isn't this the Median Maximum Interest (MMI) for combined forecast/observed objects, or is this another summary parameter altogether? I suspect the latter because I think the same forecast or observed object could be compared to multiple objects in a single MODE object text file.
>
> As always, thanks for your explanations!
> Jonathan
>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Wednesday, May 13, 2009 12:36 PM
>> To: Case, Jonathan (MSFC-VP61)[Other]
>> Cc: Barbara Brown; met_help
>> Subject: Re: MODE Advice: [Fwd: Re: Measuring model skill in
>> mode_analysis]
>>
>> Jonathan and Barb,
>>
>> As of METv2.0, the MMI, MMIF, and MMIO values are being written on the
>> first page of the MODE PostScript output in the bottom-middle of the
>> page. But they are not dumped out in the any of the ASCII
>> output files.
>>
>> The MODE-Analysis tool does not currently compute MMI directly.
>> However, we have posted an Rscript to the MET website that will compute
>> MMI (as well as other summary info):
>> http://www.dtcenter.org/met/users/downloads/Rscripts/mode_summary.R
>>
>> This Rscript may be run on the output of one or more MODE runs.
>>
>> John
>>
>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>> Hi Barbara,
>>>
>>> I finished reading the draft MET article you sent me. It seems like
>> the interest matrix is key in order to summarize model skill based on
>> MMI, MMIF, and MMIO.
>>> Given the standard output from MODE and the mode_analysis tool, how
>> would I go about calculating the MMI, MMIF, and MMIO stats? I have
>> figured out how to output the interest summary, but I don't see how
>> this can result in the matrix of output alluded to in the paper.
>>> Here is a sample of output I've produced:
>>> ********************************************
>>> lc2:/raid1/casejl/MET/MODE > mode_analysis -lookin
>> ./seus_control_ge10mm/JUN/2008060103 -summary -column INTEREST -
>> fcst_accum 01
>>>
>>>
>>> Total mode lines read = 397
>>> Total mode lines kept = 16
>>>
>>> Field N Min Max Mean StdDev P10 P25
>> P50 P75 P90 Sum
>>> -------- -- ----- ----- ----- ------ ----- ----- ----
>> - ----- ----- ------
>>> interest 16 0.525 0.963 0.729 0.131 0.574 0.626
>> 0.728 0.857 0.885 11.668
>>> ********************************************
>>>
>>> Thanks for the help,
>>> Jonathan
>>>
>>>> -----Original Message-----
>>>> From: Barbara Brown [mailto:bgb at ucar.edu]
>>>> Sent: Saturday, May 09, 2009 2:08 PM
>>>> To: Case, Jonathan (MSFC-VP61)[Other]
>>>> Cc: met_help
>>>> Subject: Re: [Met_help] MODE Advice: [Fwd: Re: Measuring model skill
>> in
>>>> mode_analysis]
>>>>
>>>>
>>>> Hi Jonathan,
>>>>
>>>> John has done a good job giving you some insights into
>> interpretation of
>>>> MODE output. As he said, this is an area where we are still
>> learning what
>>>> works best. And probably the best things to do will always be user-
>>>> dependent. Nevertheless, I'll try to give you a few more thoughts
>> about
>>>> interpretation of MODE output. Let me know if any of them are
>> helpful!
>>>> First, I've attached a recent paper that Chris Davis, Randy Bullock,
>> and I
>>>> just submitted for 2nd reviews. This paper talks more about the MMI
>> as
>>>> well as an object-based Gilbert Skill Score (i.e., ETS). Of course,
>> these
>>>> scores are like other single-valued skill scores - they can't tell
>> the
>>>> whole story by themselves, so should be used in conjunction with
>> other
>>>> types of metrics.
>>>>
>>>> A type of object-based CSI that we've also used is based on area-
>> weighting
>>>> (called "Area-Weighted CSI" or AWCSI). It works something like the
>>>> following:
>>>>
>>>> AWCSI = [(hit area weight) * #hits ] / [(hit area weight * # hits) +
>> (miss
>>>> area weight * # misses) + (false alarm area weight * # false alarms)
>> ]
>>>> Each area weight is the ratio of size of the (hit, miss, or false
>> alarm)
>>>> objects to the total area of all objects and # hits = number of
>> matched
>>>> objects; # misses = # unmatched observed objects; and # false alarms
>> = #
>>>> unmatched forecast objects. This measure answers the question: How
>> well
>>>> did the forecast "yes" objects correspond to the observed "yes"
>> objects?
>>>> The three things I've mentioned above are summary measures. It also
>> might
>>>> be useful to look at distributions of values of some of the
>> individual
>>>> object measures, such as
>>>> o Intersection area divided by union area for matched objects
>>>> o Ratios of - or differences between - forecast and observed
>> intensity
>>>> percentiles
>>>> for matched pairs of objects - the 50th and 90th percentiles
>> are the
>>>> ones
>>>> we normally focus on
>>>> o Differences in any of the other attributes
>>>> o Attributes associated with the missed and false alarm objects
>>>>
>>>> It seems like it would be useful to look at these distributions as a
>>>> function of time of day as well as level of activity (e.g., measured
>> by the
>>>> total area of observed objects).
>>>>
>>>> We would very much like your feedback on things that you find to be
>> useful.
>>>> In fact, it is our hope to post some new analysis approaches on the
>> MET
>>>> website.
>>>>
>>>> Also - please let us know if there are other attributes that you
>> would like
>>>> to be able to look at, that might be particularly useful for your
>>>> application. We're always looking for ways to improve the methods
>> in MET.
>>>> I apologize for taking awhile to respond to your email. Please let
>> me know
>>>> if you have any questions or comments.
>>>>
>>>> Barb Brown
>>>>
>>>>
>>>>
>>>>> -------- Original Message --------
>>>>> Subject: Re: Measuring model skill in mode_analysis
>>>>> Date: Thu, 30 Apr 2009 08:10:08 -0600
>>>>> From: John Halley Gotway <johnhg at rap.ucar.edu>
>>>>> To: Case, Jonathan (MSFC-VP61)[Other] <jonathan.case-1 at nasa.gov>
>>>>> References:
>>>>> <C1333A631BF21841864F141194ED3D4A1522011D2F at NDMSSCC02.ndc.nasa.gov>
>>>>> <49F5B659.1000904 at rap.ucar.edu>
>>>>> <C1333A631BF21841864F141194ED3D4A199DDAEDE2 at NDMSSCC02.ndc.nasa.gov>
>>>>> <49F5C810.5030603 at rap.ucar.edu>
>>>>> <C1333A631BF21841864F141194ED3D4A199E064EC7 at NDMSSCC02.ndc.nasa.gov>
>>>>>
>>>>> Jonathan,
>>>>>
>>>>> Exactly - Those contingency table stats are just the traditional
>>>>> verification method. They're meant to serve as a point of
>> reference
>>>>> for the MODE object output and answer the question "How would this
>>>> forecast have performed using a traditional verification approach?"
>> To get
>>>> at what you're after, we could consider a 4th line of output that
>> contains
>>>> contingency table counts computed in a different way based on the
>> results
>>>> of the object matching. The difficulty is defining how exactly to
>>>> construct that contingency table.
>>>>> Unfortunately, I don't have answer for you about how "BEST" to
>>>>> interpret the MODE output. It's an open area of research, and it's
>> one
>>>> reason why we wanted to distribute MODE to the community - to see
>> how
>>>> people use it. And I sympathize with you that by simply averaging,
>> most
>>>> the differences in object attributes get smeared away. We saw the
>> same
>>>> thing when applying MODE to NCWF2 convective cells.
>>>>> So while I don't know the best way of interpreting the results, I
>> can
>>>> tell you what people have done in the past:
>>>>> (1) Chris Davis (NCAR) has used MODE to define objects without even
>>>> paying attention to the matching. Instead, he looked at the
>> distributions
>>>> of the object areas and intensities based on time of day.
>>>>> By comparing the forecast object distribution to the observation
>> object
>>>> distribution, he was able to diagnose some timing and intensity
>> errors in
>>>> the model.
>>>>> (2) Dave Ahijevych (NCAR) has used the MODE matching output to
>> construct
>>>> contingency tables as you've suggested.
>>>>> (3) There are several metrics you could look at for a single case.
>>>> Please take a look at the sample R script I posted recently for
>> post-
>>>> processing the MODE output files:
>>>>> http://www.dtcenter.org/met/users/downloads/analysis_scripts.php.
>> The
>>>>> "mode_summary.R" script computes counts and areas of
>> matched/unmatched
>>>> objects and computes a metric we're calling MMI (Median of the
>> Maximum
>>>> Interest). Listed at the bottom of this message is a description of
>> MMI.
>>>>> (4) The issue of scale (radius and threshold) is a very important
>> one.
>>>>> You may want to use MODE to analyze your data at multiple scales to
>>>>> determine at which scale the forecast is useful (you can use the
>>>> neighborhood methods Fractions Skill Score for this as well). On
>> that same
>>>> scripts page, take a look at the "mode_quilt_plot.R" Rscript and the
>>>> "sample_PDF_file" that it creates. One thing you can do is use many
>>>> different choices of radii and thresholds and plot metrics for each
>> scale.
>>>>> Hopefully that'll help. I'm forwarding this message to the
>> scientists in
>>>> our group to see if they have any additional advice.
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> MMI stands for "Median of the Maximum Interest". Take a look at
>> the
>>>> example PostScript file here:
>> .../METv2.0/out/mode/mode_RH_ISBL_500_vs_RH_ISBL_500_120000L_20050807_
>>>>> 120000V_000000A.ps
>>>>>
>>>>> In this example, MMI is computed as follows:
>>>>> - There are 8 simple forecast objects and 12 simple observation
>> objects.
>>>>> - For each one of those objects, the maximum interest is computed
>> by
>>>> looking at the interest values for all simple object pairs in which
>> it's a
>>>> member.
>>>>> - So we have one maximum interest value for each object: 8 forecast
>>>> values and 12 observation values.
>>>>> - If an object is way off by itself and wasn't compared to any
>> other
>>>> objects because it's so far away, it's maximum interest value is 0.
>>>>> - Next we compute MMI 3 ways:
>>>>> (1) MMI with respect to forecast objects is the median of those
>> 8
>>>> forecast maximum interest values (MMI w.r.t fcst = 0.8800).
>>>>> (2) MMI with respect to observation objects is the median of
>> those 12
>>>> observation maximum interest values (MMI w.r.t. obs = 0.8723).
>>>>> (3) MMI with respect to all objects is the median of all 20
>> forecast
>>>> and observation maximum interest values (MMI w.r.t all = 0.8765).
>>>>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> I'm still not entirely sure what can be done with these
>> contingency
>>>> stats in MODE output.
>>>>>> It seems from your description below that the FCST/OBS objects
>> must
>>>> overlap at a particular grid point in order to receive a "hit". At
>> face
>>>> value, these contingency stats do not appear to provide much added
>> value
>>>> beyond traditional or neighborhood stats.
>>>>>> So, is there a recommended way to measure the "skill" of these
>>>> matched/unmatched objects besides summarizing and comparing
>> relatively
>>>> obscure object attributes such as complexity ratio, convex hull
>> distance,
>>>> etc.? I have found that the mean object attributes are VERY similar
>> to one
>>>> another when computed over a series of many forecasts. The most
>> obvious
>>>> method (which I've done so far) is to simply compare the matched and
>>>> unmatched object areas using -bycase, in order to see which one has
>> more
>>>> hits and fewer false alarms.
>>>>>> Also, some object attributes are inherently dependent on the
>>>>>> configurable parameters set prior to running MODE. For example, I
>>>>>> found that by reducing the obs/fcst_conv_radius (I ended up using
>> a
>>>>>> value of "3"), MODE produced a more stringent, realistic matching
>> of
>>>>>> the objects. When the radii are reduced, the mean CENTROID_DIST
>> is
>>>>>> subsequently reduced as well because objects must be closer to one
>>>>>> another in order to be matched. [Motivation: If these radii are
>> too
>>>>>> large, then convective precipitation systems on the west coast of
>>>>>> Florida would be paired up with convection on the east coast of
>> the
>>>>>> Florida peninsula, which to a forecaster wouldn't be considered a
>>>>>> "hit".]
>>>>>>
>>>>>> So, as you can see, I'm still trying to get my hands around how
>> best to
>>>> summarize the MODE output.
>>>>>> Thanks for your insight, as always!
>>>>>> Jonathan
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>>>>> Sent: Monday, April 27, 2009 9:58 AM
>>>>>>> To: Case, Jonathan (MSFC-VP61)[Other]
>>>>>>> Cc: met_help at ucar.edu
>>>>>>> Subject: Re: [Met_help] mode_analysis question
>>>>>>>
>>>>>>> Jonathan,
>>>>>>>
>>>>>>> Actually, the counts in those files have nothing to do with the
>>>>>>> matching that was performed by MODE. The intent of the file is
>> to
>>>>>>> give you an easy way of seeing what you'd get with a traditional
>>>>>>> verification of the input fields as opposed to an object-based
>>>>>>> verification. Let me point you to the first paragraph of section
>>>>>>> 6.3.3 of the MET User's Guide for a description of these lines.
>>>>>>> Here's some more detail based on the contents of the "FIELD"
>> column.
>>>>>>> FIELD Column...
>>>>>>> (1) RAW: Apply any masking of bad data, grid masking, or polygon
>>>>>>> matching to the raw fields. Threshold the raw fields using the
>>>>>>> "fcst_conv_thresh" and "obs_conv_thresh" config values to define
>> 0/1
>>>>>>> fields. Compute the contingency table counts by comparing these
>> 0/1
>>>>>>> fields grid point by grid point.
>>>>>>>
>>>>>>> (2) FILTER: Apply any masking of bad data, grid masking, or
>> polygon
>>>>>>> matching to the raw fields. In addition, apply the
>> "fcst_raw_thresh"
>>>>>>> and "obs_raw_thresh" to filter out any additional values.
>>>>>>> Then use the "fcst_conv_thresh" and "obs_conv_thresh" config
>> values
>>>>>>> to define 0/1 fields, and compute a contingency table from them.
>>>>>>>
>>>>>>> (3) OBJECT: Once objects have been defined in the forecast and
>>>>>>> observation fields, consider any grid point inside of an object
>> to
>>>>>>> have a value of 1 and any grid point outside of an object to have
>> a
>>>>>>> value of 0. Compute the contingency table counts by comparing
>> these
>>>>>>> 0/1 fields grid point by grid point.
>>>>>>>
>>>>>>> So really object matching has nothing to do with it.
>>>>>>>
>>>>>>> Make sense?
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>>
>>>>>>> For the RAW line:
>>>>>>>
>>>>>>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> You hit the nail on the head with #1. I saw the counts in the
>> MODE
>>>>>>> contingency files (*_cts.txt) and was wondering if we could
>>>>>>> summarize those in mode_analysis (to which the answer is no as
>> you
>>>> said).
>>>>>>>> It seems like we could fairly easily read in the contingency
>> counts
>>>>>>> in shell script, with a little management of filenames.
>>>>>>>
>>>>>>>> I also wonder how those contingency files are calculated in the
>>>>>>>> MODE
>>>>>>>>
>>>>>>> output? Is any forecast object that matches an observed object
>>>>>>> considered a "hit", and then the individual grid points within
>> that
>>>>>>> object are added to the total of FY_OY? And similarly with the
>>>>>>> unmatched forecast/observed objects, are their grid points summed
>> to
>>>>>>> be FY_ON and FN_OY? It seems like it could get a bit unclear
>> when
>>>>>>> dealing with FCST/OBS objects that don't overlap when trying to
>>>>>>> determine FY_ON, FN_OY, and FN_ON. Perhaps you could embellish
>> for
>>>>>>> me so I can make more sense about the MODE contingency stat
>> files?
>>>>>>>> Thanks for your help,
>>>>>>>> Jonathan
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>>>>>>> Sent: Monday, April 27, 2009 8:43 AM
>>>>>>>>> To: Case, Jonathan (MSFC-VP61)[Other]
>>>>>>>>> Cc: met_help at mailman.ucar.edu
>>>>>>>>> Subject: Re: [Met_help] mode_analysis question
>>>>>>>>>
>>>>>>>>> Jonathan,
>>>>>>>>>
>>>>>>>>> I'm a little unclear exactly what you're asking here. It could
>> be
>>>>>>> one
>>>>>>>
>>>>>>>>> of two things... but the answer to both is no.
>>>>>>>>>
>>>>>>>>> The MODE-Analysis tool currently performs 2 types of jobs by
>>>>>>>>> reading the MODE object statistics files:
>>>>>>>>> (1) A "summary" job in which you select one or more MODE output
>>>>>>>>>
>>>>>>> columns
>>>>>>>
>>>>>>>>> of interest and it summarizes the data in those columns.
>>>>>>>>> (2) A "bycase" job that produces summary information for each
>> MODE
>>>>>>> run
>>>>>>>
>>>>>>>>> consisting of counts and areas of matched and unmatched
>> objects.
>>>>>>>>> Let me try to understand exactly what you're asking though.
>> Are
>>>>>>>>> you
>>>>>>>>> asking:
>>>>>>>>> (1) Can MODE-Analysis read those MODE contingency table
>> statistics
>>>>>>>>> files (*_cts.txt) output and aggregate them across many cases?
>>>>>>>>> (2) Can MODE-Analysis read the MODE object statistics file,
>> treat
>>>>>>> the
>>>>>>>
>>>>>>>>> matched/unmatched object counts or areas as the elements of a
>>>>>>>>> contingency table and derive some contingency table statistics
>>>>>>>>> based on that "object matching" contingency table?
>>>>>>>>>
>>>>>>>>> Like I said, the answer to both is no. People here have done
>> (2)
>>>>>>> here
>>>>>>>
>>>>>>>>> before, but it's still kind of an open question how best to use
>>>>>>>>> the object counts or areas to populate the elements of a
>>>>>>>>> contingency table. You have to decide how to define the hits,
>>>>>>>>>
>>>>>>> misses,
>>>>>>>
>>>>>>>>> false alarms, and correct nulls. And doing so gets a bit messy
>> -
>>>>>>>>> especially defining the correct nulls.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>>>>>>>>
>>>>>>>>>> Dear Met_help,
>>>>>>>>>>
>>>>>>>>>> Does mode_analysis let the user summarize contingency
>> statistics
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>> the paired objects, or can it only summarize the various object
>>>>>>>>> attributes?
>>>>>>>>>
>>>>>>>>>> Thanks for the help,
>>>>>>>>>> Jonathan
>>>>>>>>>>
>>>>>>>>>> ***********************************************************
>>>>>>>>>> Jonathan Case, ENSCO, Inc.
>>>>>>>>>> Aerospace Sciences & Engineering Division Short-term
>> Prediction
>>>>>>>>>> Research and Transition Center 320 Sparkman Drive, Room 3062
>>>>>>>>>> Huntsville, AL 35805-1912
>>>>>>>>>> Voice: (256) 961-7504 Fax: (256) 961-7788
>>>>>>>>>> Emails: Jonathan.Case-1 at nasa.gov
>>>>>>>>>> case.jonathan at ensco.com
>>>>>>>>>> ***********************************************************
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------
>> ---
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Met_help mailing list
>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>
>>>>> _______________________________________________
>>>>> Met_help mailing list
>>>>> Met_help at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>
>>>> --
>>>> *************************************************************
>>>> Barbara G. Brown Phone: (303) 497-8468
>>>> NCAR/P.O. Box 3000 FAX: (303) 497-8386
>>>> Boulder CO 80307-3000 U.S.A. e-mail: bgb at ucar.edu
More information about the Met_help
mailing list