[Met_help] [rt.rap.ucar.edu #42518] History for Interpretation of the Point_Stat results
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Wed Feb 23 14:49:36 MST 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi again,
I'm try to understand the output from Point_Stat and I have couple of
questions (with small subquestions) about how to deal with the results. The
context is the study case of the tutorial:
http://www.dtcenter.org/met/users/support/online_tutorial/METv2.0/index.php
1. FHO.txt
I was playing a little with the PointStatConfig file. I wanted to know the
influence of the interpolation method and also the width of the box. I did 3
tests (by all of them
mask_poly[] = ["/home/wrfuser/METv3.0/data/poly/EAST.poly",
"/home/wrfuser/METv3.0/data/poly/WEST.poly"]):
Test 1. Any change in the PointStatConfig file
Test 2. interp_method[] = [ "DW_MEAN" ]
Test 3. interp_method[] = [ "DW_MEAN" ] and interp_width = [ 1, 5, 10 ]
>From the results of the tests arose several questions:
- If the varibles to consider are: fcst_field[] = [ "TMP/Z2",
"TMP/P750-850", "UGRD/Z10", "VGRD/Z10" ], why appears at the FHO file only
the second one? I suppose it has something to do with the "fcst_thresh[]"
right?
- The results F_RATE, H_RATE, and O_RATE corresponding to Test 1, Test 2,
and Test 3 are always exactly the same. Does it mean the interpolation
method does nothing?
- For each test the unique difference between F_RATE, H_RATE, and
O_RATE deals with the VX_MASK. By instance, the output of test 1 gives:
VX_MASK F_RATE H_RATE O_RATE
G212 0.49377 0.46633 0.53865
G212 0.49875 0.47132 0.53865
EAST 0.69182 0.68553 0.81132
EAST 0.70440 0.69811 0.81132
WEST 0.32000 0.24000 0.31200
WEST 0.32000 0.24000 0.31200
At this point I have several questions:
a. If my output from WRF is Lambert Conformal and has a resolution of 4
Km^2, which mask_grid should I use?
b. For the study case of the tutorial I have defined the west and the east
'mask_poly'. I understand that the best results concern to EAST and the
worst to WEST, but I don't know why.
c. Is there some relationship between the values of the mask_grid and the
mask_poly regions defined? From the previous list it looks like the
mask_grid values the average of the EAST and WEST values was. Is it false?
2. CNT
The same remarks about the F_RATE, H_RATE, and O_RATE values before done,
can be done here again.
3. Partial sums files (SL1L2, SAL1l2, VL1L2,VAL1L2)
Scalar Partial sums and Scalar Anomaly Partial Sums are new parameters. I
have search about them in the references of Wilks and Jolliffe-Stephenson
without success. Could you give please some brief description of the
relevance of these values and which substantian information from it can one
get?
I thank you in advance your attention in my case.
Best regards,
Ferdinand
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42518] Interpretation of the Point_Stat results
From: John Halley Gotway
Time: Mon Dec 06 09:33:21 2010
Ferdinand,
I read through your questions and would be happy to help explain more
about how Point-Stat works. However, I'm not exactly sure where you
are in the tutorial, so instead, let's rerun the test script
for Point-Stat that's distributed with the code. And rather than
looking at the FHO.txt file (forecast, hit, observation rates), let's
look at the CTC.txt file. The information is equivalent but CTC
contains the exact counts of the 4 elements in a 2x2 contingency
table. So it's easier to discern a small change when looking at the
counts than looking at the "rates", where the change might be out
in the 3rd or 4th decimal place.
(1) Start by running the Point-Stat test script:
> cd METv2.0/scripts
> ./test_point_stat.sh
(2) Open up the output CTC file:
../out/point_stat/point_stat_360000L_20070331_120000V_ctc.txt
It contains 1 header line followed by 48 data lines.
Look at the counts in the 1st and 3rd data lines:
... TMP P900-750 ADPUPA DTC165 UW_MEAN 1 <=273.000
<=273.000 NA NA CTC 155 29 8 29 89
... TMP P900-750 ADPUPA DTC165 MEDIAN 9 <=273.000
<=273.000 NA NA CTC 155 29 8 29 89
In this case, the nearest neighbor (UW_MEAN, 1) produced the same
table as the median (MEADIAN, 9). The two different interpolation
methods produced the same result.
Now look at the counts in the 10th and 13th data lines:
... TMP P900-750 ADPUPA DTC166 MEDIAN 9 >273.000
>273.000 NA NA CTC 334 252 20 17 45
... TMP P900-750 ADPUPA DTC166 DW_MEAN 9 >273.000
>273.000 NA NA CTC 334 253 20 16 45
In this case, the 2 different interpolation methods produced only a
very small difference in the contingency table counts. Generally
speaking, the interpolation method you choose will not have a
large impact on your results, unless you begin smoothing over a very
large area.
(3) Next, let's edit the interpolation methods we're using to only use
the "nearest neighbor" approach:
> vi config/PointStatConfig
Set:
interp_method[] = [ "UW_MEAN" ];
interp_width[] = [ 1 ];
And rerun Point-Stat:
./test_point_stat.sh
Now look again at the CTC file. It now only contains 1 header row
followed by 16 data lines. If you edit the config file and add or
remove thresholds in the "fcst_thresh" and "obs_thresh" entries,
you'll get more or less output lines. Look at the first 21 header
columns to see what each output CTC line contains.
(4) Regarding masking, I suggest that you start by setting
"mask_grid[] = [ "FULL" ];". That will compute statistics over your
entire domain. From there, you can choose to define masking polylines
however you'd like to subdivide your domain. Generally, it's a good
idea to compute verification scores over a region that's small enough
for you to interpret the results. For example, if you're
only looking over the south-western desert of the United States and
see that the temperatures from your model are too warm in the
mornings, perhaps you'd be able to interpret that error in a
meteorological way. Ultimately, it's up to you to design the
verification to investigate the performance of your model in the way
you want.
(5) Partial sums like SL1L2 and VL1L2 can be used to recompute several
continuous statistics. For example, from an SL1L2 line you can
compute RMSE and MSE. They serve as a nice concise way of
aggregating together results from multiple runs. For example, suppose
you verify 2-m temperature over your entire domain every day for 30
days, and you'd like to see how your model performed over
those 30 days. One thing you could do is look at a time series of
RMSE values - one for each day. Another thing you could do is
aggregate together the SL1L2 partial sums for those 30 days, and then
recompute an aggregate RMSE value for that time period. You'd use the
STAT-Analysis tool to perform this job. Something like the following
would do it:
stat_analysis -lookin /stat/directory -job aggregate_stat
-line_type SL1L2 -out_line_type CNT -fcst_var TMP -fcst_lev Z2
-vx_mask FULL -dump_row dump.stat
This will look in the "/stat/directory" directory for file ending in
".stat". It'll filter out only those lines matching the search
criteria (SL1L2, TMP, Z2, FULL). And it'll aggregate together
those lines, and recompute continuous statistics for them. It'll also
dump the stat lines it used in it's analysis to the file "dump.stat".
It's always a good idea to take a look at that file to
make sure the job really did operate over the lines you intended.
You know another way to do this sort of analysis is using the raw
matched pair (MPR) lines themselves. You have the most flexibility
using the MPR lines, but it just takes up a lot more space than
using the SL1L2 lines. FYI, the same sort of aggregate can be done to
recompute categorical statistics from the contingency table counts
(CTC -> CTS).
Not sure if I answered all your questions or not, but hopefully it
helps.
Thanks,
John Halley Gotway
On 12/03/2010 09:58 AM, RAL HelpDesk {for Ferdinand Rennen} wrote:
>
> Fri Dec 03 09:58:45 2010: Request 42518 was acted upon.
> Transaction: Ticket created by ferdinand.rennen at gmail.com
> Queue: met_help
> Subject: Interpretation of the Point_Stat results
> Owner: Nobody
> Requestors: ferdinand.rennen at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42518 >
>
>
> Hi again,
>
> I'm try to understand the output from Point_Stat and I have couple
of
> questions (with small subquestions) about how to deal with the
results. The
> context is the study case of the tutorial:
>
http://www.dtcenter.org/met/users/support/online_tutorial/METv2.0/index.php
>
> 1. FHO.txt
>
> I was playing a little with the PointStatConfig file. I wanted to
know the
> influence of the interpolation method and also the width of the box.
I did 3
> tests (by all of them
> mask_poly[] = ["/home/wrfuser/METv3.0/data/poly/EAST.poly",
> "/home/wrfuser/METv3.0/data/poly/WEST.poly"]):
>
> Test 1. Any change in the PointStatConfig file
> Test 2. interp_method[] = [ "DW_MEAN" ]
> Test 3. interp_method[] = [ "DW_MEAN" ] and interp_width = [ 1, 5,
10 ]
>
>>From the results of the tests arose several questions:
>
> - If the varibles to consider are: fcst_field[] = [ "TMP/Z2",
> "TMP/P750-850", "UGRD/Z10", "VGRD/Z10" ], why appears at the FHO
file only
> the second one? I suppose it has something to do with the
"fcst_thresh[]"
> right?
>
> - The results F_RATE, H_RATE, and O_RATE corresponding to Test 1,
Test 2,
> and Test 3 are always exactly the same. Does it mean the
interpolation
> method does nothing?
>
> - For each test the unique difference between F_RATE, H_RATE, and
> O_RATE deals with the VX_MASK. By instance, the output of test 1
gives:
>
> VX_MASK F_RATE H_RATE O_RATE
> G212 0.49377 0.46633 0.53865
> G212 0.49875 0.47132 0.53865
> EAST 0.69182 0.68553 0.81132
> EAST 0.70440 0.69811 0.81132
> WEST 0.32000 0.24000 0.31200
> WEST 0.32000 0.24000 0.31200
>
> At this point I have several questions:
>
> a. If my output from WRF is Lambert Conformal and has a resolution
of 4
> Km^2, which mask_grid should I use?
> b. For the study case of the tutorial I have defined the west and
the east
> 'mask_poly'. I understand that the best results concern to EAST and
the
> worst to WEST, but I don't know why.
> c. Is there some relationship between the values of the mask_grid
and the
> mask_poly regions defined? From the previous list it looks like the
> mask_grid values the average of the EAST and WEST values was. Is it
false?
>
>
> 2. CNT
>
> The same remarks about the F_RATE, H_RATE, and O_RATE values
before done,
> can be done here again.
>
>
> 3. Partial sums files (SL1L2, SAL1l2, VL1L2,VAL1L2)
>
> Scalar Partial sums and Scalar Anomaly Partial Sums are new
parameters. I
> have search about them in the references of Wilks and Jolliffe-
Stephenson
> without success. Could you give please some brief description of the
> relevance of these values and which substantian information from it
can one
> get?
>
>
> I thank you in advance your attention in my case.
>
> Best regards,
>
> Ferdinand
------------------------------------------------
More information about the Met_help
mailing list