John,<br>Thanks for help and info.<br><br>Ruifang<br><br><div class="gmail_quote">On Fri, Aug 21, 2009 at 2:13 PM, John Halley Gotway <span dir="ltr"><<a href="mailto:johnhg@rap.ucar.edu">johnhg@rap.ucar.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Ruifang,<br>
<br>
Thresholds are not applied when computing continuous (CNT) statistics.<br>
That's why those columns (FCST_THRESH and OBS_THRESH) contain NA's in the<br>
CNT lines of the MET output.<br>
<br>
Let me refer you to two places for more information about the verification<br>
statistics that MET generates. The first is Appendix C of the MET User's<br>
Guide. In there we give a description of how the stats are computed and<br>
what each means. And here's a good link about the definition of<br>
verification scores:<br>
<a href="http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html" target="_blank">http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html</a><br>
<br>
Hopefully these resources will answer your questions.<br>
<br>
But if you have a specific question about how to interpret one of the<br>
stats, go ahead and write MET-Help, and a statistician will get back to<br>
you.<br>
<br>
Thanks,<br>
<font color="#888888">John<br>
</font><div><div></div><div class="h5"><br>
> John,<br>
> I looked at CTC table and thresholds are applied. However I could not see<br>
> thresholds in CNT table. I guess thresholds should be also applied in CNT<br>
> to<br>
> get user desired statistic values (for example bias, rmse), right?<br>
><br>
> Thanks,<br>
> Ruifang<br>
><br>
> On Fri, Aug 21, 2009 at 8:06 AM, John Halley Gotway<br>
> <<a href="mailto:johnhg@rap.ucar.edu">johnhg@rap.ucar.edu</a>>wrote:<br>
><br>
>> Ruifang,<br>
>><br>
>> The behavior you're seeing is correct. The MPR output would be the<br>
>> same.<br>
>> However, the values in the contingency tables (CTC line type, for<br>
>> example)<br>
>> should be different when choosing different<br>
>> thresholds.<br>
>><br>
>> Here's what's going on... When you run Point-Stat, you can think of it<br>
>> as<br>
>> setting up a bucket for each combination of forecast field/masking<br>
>> region/interpolation method. So for example, suppose you<br>
>> configure Point-Stat to verify 2-meter temperature and 500mb temperature<br>
>> over 3 masking regions, your FULL domain and then two subdomains. And<br>
>> you're using the nearest neighbor interpolation method.<br>
>> In this case, we'd be verifying 2 fields over 3 regions using 1<br>
>> interpolation method. So Point-Stat will set up 6 "buckets", 3 for<br>
>> 2-meter<br>
>> temp and 3 for 500mb temp.<br>
>><br>
>> Next, Point-Stat looks at each one of the point observation values<br>
>> you've<br>
>> passed to it. If the observation type (i.e. 2-m temp) matches the<br>
>> forecast<br>
>> type, and it's in the correct masking region,<br>
>> it'll interpolate the forecast values to the observation location to<br>
>> create<br>
>> a matched pair. Then it throws that matched fcst-obs pair into the<br>
>> appropriate bucket. Point-Stat processes through all of<br>
>> the point observations in this way, computing matched pairs, and<br>
>> throwing<br>
>> them in the right buckets.<br>
>><br>
>> After that's finished, it'll compute whatever statistics you've<br>
>> requested<br>
>> using whatever threshold values you've set. For example, if you set<br>
>> threshold values of ">273.0" and ">283.0" for 2-m temp,<br>
>> it'll apply those to all of the matched pairs in the "bucket" and<br>
>> compute<br>
>> contingency tables and stats. However, the choice of threshold has no<br>
>> effect on which matched pairs make it into the bucket<br>
>> to begin with.<br>
>><br>
>> The matched pair (MPR) output consists of all the matched pairs that are<br>
>> in<br>
>> the buckets. So the MPR output does NOT depend on any choice of<br>
>> threshold<br>
>> value.<br>
>><br>
>> Does that make sense?<br>
>><br>
>> Thanks,<br>
>> John<br>
>><br>
>> Ruifang Li wrote:<br>
>> > Hi John,<br>
>> > I run point_stat tool and did two test with different fcst_thresh[] in<br>
>> > config file. I expect the different MPR output, but they create same<br>
>> MPR.<br>
>> I<br>
>> > could not figure out why. In my understanding, if fcst_thresh eq 0,<br>
>> fcst<br>
>> > value in MPR should be 0. if fcst_thresh le 100 fcst value in MPR<br>
>> should<br>
>> be<br>
>> > le 100, right?<br>
>> ><br>
>> > Here are two fcst_thresh:<br>
>> ><br>
>> > fcst_field[] =<br>
>> ><br>
>> ["UGRD/Z10","UGRD/P1000","UGRD/P925","UGRD/P850","UGRD/P700","UGRD/P500","UGRD/P400","UGRD/P300","UGRD/P250","UGRD/P200","UGRD/P150","UGRD/P100","UGRD/P70","UGRD/P50","UGRD/P30","UGRD/P20","UGRD/P10"<br>
>> > ];<br>
>> ><br>
>> > //fcst_thresh[] = [ "le100",<br>
>> ><br>
>> "le10","le10","le100","le100","le100","le100","le100","le100","le100","le100","le100",<br>
>> > "le100","le100", "le100","le100","le100" ];<br>
>> > fcst_thresh[] = [ "eq0",<br>
>> > "eq0","eq0","eq0","eq0","eq0","eq0","le10","eq0","eq0","eq0","eq0",<br>
>> > "eq0","eq0", "eq0","eq0","eq0" ];<br>
>> ><br>
>> > Here is MPR file:<br>
>> ><br>
>> > /ptmp/lir/test/2009_t8_15km/verify_met2.0/test% ls -l<br>
>> > /ptmp/lir/data/2009_t8_15km/verify_met/gsi/point_stat/UGRD/2007081512/<br>
>> > total 14208<br>
>> > -rw-r--r-- 1 lir ncar 786984 Aug 20 16:05<br>
>> > point_stat_000000L_20070815_120000V.stat<br>
>> > -rw-r--r-- 1 lir ncar 652568 Aug 20 16:05<br>
>> > point_stat_000000L_20070815_120000V_mpr.txt<br>
>> > -rw-r--r-- 1 lir ncar 698412 Aug 20 16:05<br>
>> > point_stat_120000L_20070816_000000V.stat<br>
>> > -rw-r--r-- 1 lir ncar 579124 Aug 20 16:05<br>
>> > point_stat_120000L_20070816_000000V_mpr.txt<br>
>> > -rw-r--r-- 1 lir ncar 761574 Aug 20 16:05<br>
>> > point_stat_240000L_20070816_120000V.stat<br>
>> > -rw-r--r-- 1 lir ncar 631498 Aug 20 16:05<br>
>> > point_stat_240000L_20070816_120000V_mpr.txt<br>
>> > -rw-r--r-- 1 lir ncar 740520 Aug 20 16:05<br>
>> > point_stat_360000L_20070817_000000V.stat<br>
>> > -rw-r--r-- 1 lir ncar 614040 Aug 20 16:05<br>
>> > point_stat_360000L_20070817_000000V_mpr.txt<br>
>> > -rw-r--r-- 1 lir ncar 791340 Aug 20 16:05<br>
>> > point_stat_480000L_20070817_120000V.stat<br>
>> > -rw-r--r-- 1 lir ncar 656180 Aug 20 16:05<br>
>> > point_stat_480000L_20070817_120000V_mpr.txt<br>
>> ><br>
>> > Script:<br>
>> > /ptmp/lir/test/2009_t8_15km/verify_met2.0/test% point_stat_gsi.ksh<br>
>> > *** Running POINT_STAT on WRF OUTPUT ***<br>
>> > *** UGRD ***<br>
>> > GSL_RNG_TYPE=mt19937<br>
>> > GSL_RNG_SEED=2412285258<br>
>> > Forecast File:<br>
>> ><br>
>> /ptmp/lir/data/2009_t8_15km/verify_met2.0/gsi/wpp/2007081512/postprd/wrfprs_d01.000<br>
>> > Climatology File: none<br>
>> > Configuration File: ./config/PointStatConfig_GSI.UGRD<br>
>> > Observation File:<br>
>> > /ptmp/lir/data/2009_t8_15km/verify_met2.0/ob_nc/2007081512/ob.nc.qc2<br>
>> ><br>
>> > ----------------------------------------<br>
>> ><br>
>> > Reading records for UGRD/Z10.<br>
>> > For UGRD/Z10 found 1 forecast levels and 0 climatology levels.<br>
>> ><br>
>> ><br>
>> > Thanks for your support,<br>
>> ><br>
>> ><br>
>> > ------------------------------------------------------------------------<br>
>> ><br>
>> > _______________________________________________<br>
>> > Met_help mailing list<br>
>> > <a href="mailto:Met_help@mailman.ucar.edu">Met_help@mailman.ucar.edu</a><br>
>> > <a href="http://mailman.ucar.edu/mailman/listinfo/met_help" target="_blank">http://mailman.ucar.edu/mailman/listinfo/met_help</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Ruifang<br>
><br>
> Mesoscale & Microscale Meteorology Division<br>
> Phone: 303-497-8938<br>
> Office: FL3-3085<br>
> Email: <a href="mailto:lir@ucar.edu">lir@ucar.edu</a><br>
><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Ruifang <br><br>Mesoscale & Microscale Meteorology Division<br>Phone: 303-497-8938<br>Office: FL3-3085<br>Email: <a href="mailto:lir@ucar.edu">lir@ucar.edu</a><br>