[Met_help] [rt.rap.ucar.edu #50694] History for Question about MET Point-Stat (UNCLASSIFIED)

[Tressa Fowler via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I'm trying to use MET Point-Stat to verify the performance of a high 
resolution WRF model we run here at ARL at 3km and 1km horizontal resolution. 
I am also comparing the error statistics I get with those generated by the 
model we use to initialize the WRF which is the North American Mesoscale (NAM) 
model run at 12 km resolution. I want to see if the error statistics for 
surface met variables for the 3 models are similar or widely differing.

I presented a seminar last week on this work and a colleague asked a question 
regarding the use of the distance weighted mean interpolation method. Our 
current configuration file specifies DW_MEAN with interp_width of 2 and 
interp_thresh of 1.0. I don't specify interp_flag.

The question was:

... when you compare the statistics using the 2x2 forecast to ob mapping is
there a bias due to the fact that for the coarse model (NAM) you are
allowing grid points that are significantly farther away to have influence
{i.e. grid point could potentially be almost 24 km away and the terrain
could be significantly different, albeit smoothed out quite a bit.  Whereas
at 3 km and/or 1km the influence of these points might be more appropriate?
What is any change is there if you use coincident/collocated points when
mapping forecasts to observations?  Is that even relevant?  I don't know for
sure. (don't worry about some of the numbers used in the wording as they might 
have been looking at a mistaken graphic)

This question has to do with the size of the atmospheric volume which will 
influence the value of the forecast interpolated to the location of the 
observation. In the case of the lower resolution model (NAM), using our 
current configuration file setting, forecast values in the area of 12X12 km or 
144 sq km surrounding the observation are given weight in the interpolation 
scheme.

In the case of the higher resolution WRF models, the volume of influence, 
using the same config file, is much smaller. For 3km resolution, forecast 
values in the area of 3X3 km or 9 sq km surrounding the observation are given 
weight in the interpolation scheme.

In the case of the higher resolution WRF models, the volume of influence, 
using the same config file, is much smaller. For 1km resolution, forecast 
values in the area of 1X1 km or 1 sq km surrounding the observation are given 
weight in the interpolation scheme.

For purposes of comparing error stats for models of varying resolutions should 
I be altering the config file (interp_width) based on the resolution of the 
model so the volume of influence is the same (i.e use interp_width of 2 for 
12km NAM and interp_width of 5 for 3km WRF and interp_width of 13 for 1km 
WRF)?

Thanks.

R/
John

Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil


Classification: UNCLASSIFIED
Caveats: NONE




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Question about MET Point-Stat (UNCLASSIFIED)
From: Tressa Fowler
Time: Wed Oct 19 11:03:23 2011

Hi John,

This is a tough question, one which is discussed in verification
circles quite often. There are no answers, really. You cannot compare
models on different scales using traditional verification metrics.
Using the spatial verification methods such as mode, wavelets, or
neighborhood methods can provide more comparable results between
models on different scales. Unfortunately, these don't work with point
observations. Sometimes, users will upscale the higher resolution
model. This makes the verification results more comparable, but of
course you lose something in the upscaling.

If you come up with any ideas about how to do this better, please let
us know.

Thanks,

Tressa

------------------------------------------------
Subject: Question about MET Point-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 19 11:46:41 2011

Classification: UNCLASSIFIED
Caveats: NONE

Tressa -

Thanks for your response on this and my other question about the ME
statistic
with winds. Good discussion and a very concise answer to a complex
subject.

We met here last week with a post doc meteorologist who I believe
voiced your
latter concern about "losing something in the upscaling" if we were to
increase the interp_width  to assure that the area of influence of the
higher
resolution model was the same area as that of the lower resolution
model with
a small interp_width. He opined that when you do this, you lose the
value of
going to the higher resolution which is to focus in on the local
effects and
not pay attention to what's happening far away.

Regards your comments on the spatial methods, we are currently
grappling with
the issue of acquiring a high resolution gridded observation dataset
to match
the resolution of our WRF-FDDA Nowcast model which is capable of 1-3
km
resolution now and we hope to increase this to 500 meters soon. We are
trying
to come up with an independent gridded observation dataset such as
RTMA or
perhaps a product from LAPS, RUC or STMAS so we avoid using the same
models
analysis to verify the forecasts.

If I use RTMA at 2.5 km res, Can I regrid that down to a 1km grid to
verify
the 1km WRF or is it better to regrid it up to 3km and use it to
verify the
3km WRF? My opinion is that it's better to regrid it up to a lower
resolution
than its original resolution so you don't spread out the 2.5 km data
beyond
its applicablility.

R/
John


-----Original Message-----
From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 19, 2011 11:03 AM
To: Raby, John W USA CIV (US)
Cc: Vaucher, Gail T USA CIV (US); Jameson, Terry C USA CIV (US)
Subject: [rt.rap.ucar.edu #50694] Question about MET Point-Stat
(UNCLASSIFIED)

Hi John,

This is a tough question, one which is discussed in verification
circles quite
often. There are no answers, really. You cannot compare models on
different
scales using traditional verification metrics. Using the spatial
verification
methods such as mode, wavelets, or neighborhood methods can provide
more
comparable results between models on different scales. Unfortunately,
these
don't work with point observations. Sometimes, users will upscale the
higher
resolution model. This makes the verification results more comparable,
but of
course you lose something in the upscaling.

If you come up with any ideas about how to do this better, please let
us know.

Thanks,

Tressa

Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Question about MET Point-Stat (UNCLASSIFIED)
From: Tressa Fowler
Time: Wed Oct 19 13:19:02 2011

Hi John,

In my opinion, you should take the RTMA to the 3 km rather than down
to 1. I have never actually done a test of this, so you might do that
if you have time. Please let us know how it turns out if you do.

Tressa

------------------------------------------------