[Met_help] point_stat config question
John Halley Gotway
johnhg at rap.ucar.edu
Fri Aug 21 14:13:55 MDT 2009
Ruifang,
Thresholds are not applied when computing continuous (CNT) statistics.
That's why those columns (FCST_THRESH and OBS_THRESH) contain NA's in the
CNT lines of the MET output.
Let me refer you to two places for more information about the verification
statistics that MET generates. The first is Appendix C of the MET User's
Guide. In there we give a description of how the stats are computed and
what each means. And here's a good link about the definition of
verification scores:
http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html
Hopefully these resources will answer your questions.
But if you have a specific question about how to interpret one of the
stats, go ahead and write MET-Help, and a statistician will get back to
you.
Thanks,
John
> John,
> I looked at CTC table and thresholds are applied. However I could not see
> thresholds in CNT table. I guess thresholds should be also applied in CNT
> to
> get user desired statistic values (for example bias, rmse), right?
>
> Thanks,
> Ruifang
>
> On Fri, Aug 21, 2009 at 8:06 AM, John Halley Gotway
> <johnhg at rap.ucar.edu>wrote:
>
>> Ruifang,
>>
>> The behavior you're seeing is correct. The MPR output would be the
>> same.
>> However, the values in the contingency tables (CTC line type, for
>> example)
>> should be different when choosing different
>> thresholds.
>>
>> Here's what's going on... When you run Point-Stat, you can think of it
>> as
>> setting up a bucket for each combination of forecast field/masking
>> region/interpolation method. So for example, suppose you
>> configure Point-Stat to verify 2-meter temperature and 500mb temperature
>> over 3 masking regions, your FULL domain and then two subdomains. And
>> you're using the nearest neighbor interpolation method.
>> In this case, we'd be verifying 2 fields over 3 regions using 1
>> interpolation method. So Point-Stat will set up 6 "buckets", 3 for
>> 2-meter
>> temp and 3 for 500mb temp.
>>
>> Next, Point-Stat looks at each one of the point observation values
>> you've
>> passed to it. If the observation type (i.e. 2-m temp) matches the
>> forecast
>> type, and it's in the correct masking region,
>> it'll interpolate the forecast values to the observation location to
>> create
>> a matched pair. Then it throws that matched fcst-obs pair into the
>> appropriate bucket. Point-Stat processes through all of
>> the point observations in this way, computing matched pairs, and
>> throwing
>> them in the right buckets.
>>
>> After that's finished, it'll compute whatever statistics you've
>> requested
>> using whatever threshold values you've set. For example, if you set
>> threshold values of ">273.0" and ">283.0" for 2-m temp,
>> it'll apply those to all of the matched pairs in the "bucket" and
>> compute
>> contingency tables and stats. However, the choice of threshold has no
>> effect on which matched pairs make it into the bucket
>> to begin with.
>>
>> The matched pair (MPR) output consists of all the matched pairs that are
>> in
>> the buckets. So the MPR output does NOT depend on any choice of
>> threshold
>> value.
>>
>> Does that make sense?
>>
>> Thanks,
>> John
>>
>> Ruifang Li wrote:
>> > Hi John,
>> > I run point_stat tool and did two test with different fcst_thresh[] in
>> > config file. I expect the different MPR output, but they create same
>> MPR.
>> I
>> > could not figure out why. In my understanding, if fcst_thresh eq 0,
>> fcst
>> > value in MPR should be 0. if fcst_thresh le 100 fcst value in MPR
>> should
>> be
>> > le 100, right?
>> >
>> > Here are two fcst_thresh:
>> >
>> > fcst_field[] =
>> >
>> ["UGRD/Z10","UGRD/P1000","UGRD/P925","UGRD/P850","UGRD/P700","UGRD/P500","UGRD/P400","UGRD/P300","UGRD/P250","UGRD/P200","UGRD/P150","UGRD/P100","UGRD/P70","UGRD/P50","UGRD/P30","UGRD/P20","UGRD/P10"
>> > ];
>> >
>> > //fcst_thresh[] = [ "le100",
>> >
>> "le10","le10","le100","le100","le100","le100","le100","le100","le100","le100","le100",
>> > "le100","le100", "le100","le100","le100" ];
>> > fcst_thresh[] = [ "eq0",
>> > "eq0","eq0","eq0","eq0","eq0","eq0","le10","eq0","eq0","eq0","eq0",
>> > "eq0","eq0", "eq0","eq0","eq0" ];
>> >
>> > Here is MPR file:
>> >
>> > /ptmp/lir/test/2009_t8_15km/verify_met2.0/test% ls -l
>> > /ptmp/lir/data/2009_t8_15km/verify_met/gsi/point_stat/UGRD/2007081512/
>> > total 14208
>> > -rw-r--r-- 1 lir ncar 786984 Aug 20 16:05
>> > point_stat_000000L_20070815_120000V.stat
>> > -rw-r--r-- 1 lir ncar 652568 Aug 20 16:05
>> > point_stat_000000L_20070815_120000V_mpr.txt
>> > -rw-r--r-- 1 lir ncar 698412 Aug 20 16:05
>> > point_stat_120000L_20070816_000000V.stat
>> > -rw-r--r-- 1 lir ncar 579124 Aug 20 16:05
>> > point_stat_120000L_20070816_000000V_mpr.txt
>> > -rw-r--r-- 1 lir ncar 761574 Aug 20 16:05
>> > point_stat_240000L_20070816_120000V.stat
>> > -rw-r--r-- 1 lir ncar 631498 Aug 20 16:05
>> > point_stat_240000L_20070816_120000V_mpr.txt
>> > -rw-r--r-- 1 lir ncar 740520 Aug 20 16:05
>> > point_stat_360000L_20070817_000000V.stat
>> > -rw-r--r-- 1 lir ncar 614040 Aug 20 16:05
>> > point_stat_360000L_20070817_000000V_mpr.txt
>> > -rw-r--r-- 1 lir ncar 791340 Aug 20 16:05
>> > point_stat_480000L_20070817_120000V.stat
>> > -rw-r--r-- 1 lir ncar 656180 Aug 20 16:05
>> > point_stat_480000L_20070817_120000V_mpr.txt
>> >
>> > Script:
>> > /ptmp/lir/test/2009_t8_15km/verify_met2.0/test% point_stat_gsi.ksh
>> > *** Running POINT_STAT on WRF OUTPUT ***
>> > *** UGRD ***
>> > GSL_RNG_TYPE=mt19937
>> > GSL_RNG_SEED=2412285258
>> > Forecast File:
>> >
>> /ptmp/lir/data/2009_t8_15km/verify_met2.0/gsi/wpp/2007081512/postprd/wrfprs_d01.000
>> > Climatology File: none
>> > Configuration File: ./config/PointStatConfig_GSI.UGRD
>> > Observation File:
>> > /ptmp/lir/data/2009_t8_15km/verify_met2.0/ob_nc/2007081512/ob.nc.qc2
>> >
>> > ----------------------------------------
>> >
>> > Reading records for UGRD/Z10.
>> > For UGRD/Z10 found 1 forecast levels and 0 climatology levels.
>> >
>> >
>> > Thanks for your support,
>> >
>> >
>> > ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Met_help mailing list
>> > Met_help at mailman.ucar.edu
>> > http://mailman.ucar.edu/mailman/listinfo/met_help
>>
>
>
>
> --
> Ruifang
>
> Mesoscale & Microscale Meteorology Division
> Phone: 303-497-8938
> Office: FL3-3085
> Email: lir at ucar.edu
>
More information about the Met_help
mailing list