[Met_help] [rt.rap.ucar.edu #50850] History for Interpolation Methods (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 21 14:35:40 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Classification: UNCLASSIFIED
Caveats: NONE
Regarding the interpolation methods available for use in Point-Stat, do you
have any information or references you can send me which describe the
relative merits, advantages/disadvantages, shortcomings, pitfalls, etc which
I could use to compare and contrast the various methods in order to select
the best one for my use?
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
Classification: UNCLASSIFIED
Caveats: NONE
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Oct 21 10:59:23 2011
John,
I'm not aware of any references that would guide you as to which
interpolation method you should choose. I can tell you that two very
common ones people choose are nearest-neighbor and bilinear
interpolation of the 4 closets points. The following settings in the
Point-Stat config file would give you those two methods:
interp_method[] = [ "BILIN" ];
interp_width[] = [ 1, 2 ];
The intention of providing very configurable interpolation options was
to facilitate people in studying what impact the interpolation method
choice has on the verification scores. But if you're
looking for a single choice, I'd guess that bilinear interpolation of
the 4 closest points is the most commonly used one.
I've CC'ed Tressa on the met_help ticket in case she has any more
information to offer.
Thanks,
John Halley Gotway
On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>
> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Interpolation Methods (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Regarding the interpolation methods available for use in Point-Stat,
do you
> have any information or references you can send me which describe
the
> relative merits, advantages/disadvantages, shortcomings, pitfalls,
etc which
> I could use to compare and contrast the various methods in order to
select
> the best one for my use?
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Interpolation Methods (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Oct 21 11:35:08 2011
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for that info. I was not aware that these were the most common
ones in
use. They are different from the one I've been using which is distance
weighted mean and the reason for this was simply that my predecessor
(Barb
Sauter) used that one. Now, people are asking questions as to why we
are using
that method and I don't have a reason or justification. Barb retired a
couple
of years ago, so I can't get her reasons.
This gets me back to the another part of the original question which
is to try
and find out if there is any guidance out there which advises of the
inherent
strengths and weaknesses of each method.
I suppose that I could arbitrarily change the methods and compare the
error
stats I get with each, but it gets down to the question of which set
of stats
do I consider the most valid and I don't know how I would be able to
answer
that question. Because I don't know what the "perfect" error stat is,
I can't
judge which of the interpolation methods produces error stats closet
to
"perfect".
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, October 21, 2011 10:59 AM
To: Raby, John W USA CIV (US)
Cc: tressa at ucar.edu
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
John,
I'm not aware of any references that would guide you as to which
interpolation
method you should choose. I can tell you that two very common ones
people
choose are nearest-neighbor and bilinear interpolation of the 4
closets
points. The following settings in the Point-Stat config file would
give you
those two methods:
interp_method[] = [ "BILIN" ];
interp_width[] = [ 1, 2 ];
The intention of providing very configurable interpolation options was
to
facilitate people in studying what impact the interpolation method
choice has
on the verification scores. But if you're looking for a single
choice, I'd
guess that bilinear interpolation of the 4 closest points is the most
commonly
used one.
I've CC'ed Tressa on the met_help ticket in case she has any more
information
to offer.
Thanks,
John Halley Gotway
On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>
> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Interpolation Methods (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Regarding the interpolation methods available for use in Point-Stat,
> do you have any information or references you can send me which
> describe the relative merits, advantages/disadvantages,
shortcomings,
> pitfalls, etc which I could use to compare and contrast the various
> methods in order to select the best one for my use?
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Oct 21 13:02:09 2011
John,
Prior to METv3.0.1, the "BILIN" interpolation option was not available
in Point-Stat. We had incorrectly instructed users that DW_MEAN with
a width of 2 was exactly the same as bilinear
interpolation, when in fact it is not. We realized the discrepancy
and added direct support for bilinear interpolation in METv3.0.1. I
suspect that this is the reason Barb was using DW_MEAN -
intending it to be bilinear.
To larger question of which interpolation method is best,
unfortunately, I think the answer is that it all depends on the data
...
If your observing site is on the side of a steep mountain, it may be
best to use the nearest neighbor method - so as to avoid using
forecast values from grid points with very different elevations.
Whereas if your the area around your observation site is relatively
uniform, averaging over a larger area is more justifiable.
The question of which interpolation method to use is frequently asked,
and my general take on it is that it usually doesn't matter.
In Point-Stat and Grid-Stat, it's very easy to use multiple
interpolation methods on the same run. So investigating the impact of
interpolation method on your statistics is pretty easy. I'd suggest
choosing a handful of interpolation methods, running them for a sample
of your data, and look to see how the statistics differ. I'm guessing
you'll find that the choice of interpolation method has
little impact on your results - especially when you consider the
potential errors of the observing equipment itself.
One thing to note though is that, generally speaking, smoother
forecasts yield higher traditional verification scores (e.g. RMSE)
than hi-resolution ones. And when you interpolate over larger and
larger numbers of points, it has the effect of smoothing the forecast.
So I would guess you'd find that RMSE values improve overall as you
use larger and larger interpolation sizes.
Sorry there isn't an easy answer.
John
On 10/21/2011 11:35 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for that info. I was not aware that these were the most
common ones in
> use. They are different from the one I've been using which is
distance
> weighted mean and the reason for this was simply that my predecessor
(Barb
> Sauter) used that one. Now, people are asking questions as to why we
are using
> that method and I don't have a reason or justification. Barb retired
a couple
> of years ago, so I can't get her reasons.
>
> This gets me back to the another part of the original question which
is to try
> and find out if there is any guidance out there which advises of the
inherent
> strengths and weaknesses of each method.
>
> I suppose that I could arbitrarily change the methods and compare
the error
> stats I get with each, but it gets down to the question of which set
of stats
> do I consider the most valid and I don't know how I would be able to
answer
> that question. Because I don't know what the "perfect" error stat
is, I can't
> judge which of the interpolation methods produces error stats closet
to
> "perfect".
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, October 21, 2011 10:59 AM
> To: Raby, John W USA CIV (US)
> Cc: tressa at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
>
> John,
>
> I'm not aware of any references that would guide you as to which
interpolation
> method you should choose. I can tell you that two very common ones
people
> choose are nearest-neighbor and bilinear interpolation of the 4
closets
> points. The following settings in the Point-Stat config file would
give you
> those two methods:
> interp_method[] = [ "BILIN" ];
> interp_width[] = [ 1, 2 ];
>
> The intention of providing very configurable interpolation options
was to
> facilitate people in studying what impact the interpolation method
choice has
> on the verification scores. But if you're looking for a single
choice, I'd
> guess that bilinear interpolation of the 4 closest points is the
most commonly
> used one.
>
> I've CC'ed Tressa on the met_help ticket in case she has any more
information
> to offer.
>
> Thanks,
> John Halley Gotway
>
>
> On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>>
>> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Interpolation Methods (UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> Regarding the interpolation methods available for use in Point-
Stat,
>> do you have any information or references you can send me which
>> describe the relative merits, advantages/disadvantages,
shortcomings,
>> pitfalls, etc which I could use to compare and contrast the various
>> methods in order to select the best one for my use?
>>
>> Thanks.
>>
>> R/
>> John
>>
>> Mr John W. Raby, Meteorologist
>> U.S. Army Research Laboratory
>> White Sands Missile Range, NM 88002
>> (575) 678-2004 DSN 258-2004
>> FAX (575) 678-1230 DSN 258-1230
>> Email: john.w.raby2.civ at mail.mil
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Interpolation Methods (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Oct 21 13:25:53 2011
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for your discussion on that subject. It will help me as I
attempt to
provide some answers to the folks here.
Good point about the observation location. I validate over 2 domains
centered
over UT which are 546km x 546km (500+ obs) and 102km x 102km (~25 obs)
using
PrepBUFR plus mesonet observations from MADIS.
I suspect that my observing sites fall into both categories you
mention in
that area.
Can you clarify your last statement to be sure I'm understanding you?
You said: " generally speaking, smoother forecasts yield higher
traditional
verification scores (e.g. RMSE) than hi-resolution ones. And when you
interpolate over larger and larger numbers of points, it has the
effect of
smoothing the forecast. So I would guess you'd find that RMSE values
improve
overall as you use larger and larger interpolation sizes."
My question is: when you say "smoother forecasts" do you mean lower
resolution
(large grid spacing)?
When you say "higher traditional verification scores" you mean larger
RMSE
values (worse scores)?
The reason I ask is that I thought that the lower resolution models
typically
had lower RMSE values than high resolution models when you verify with
the
traditional methods.
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, October 21, 2011 1:02 PM
To: Raby, John W USA CIV (US)
Cc: tressa at ucar.edu
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
John,
Prior to METv3.0.1, the "BILIN" interpolation option was not available
in
Point-Stat. We had incorrectly instructed users that DW_MEAN with a
width of
2 was exactly the same as bilinear interpolation, when in fact it is
not. We
realized the discrepancy and added direct support for bilinear
interpolation
in METv3.0.1. I suspect that this is the reason Barb was using
DW_MEAN -
intending it to be bilinear.
To larger question of which interpolation method is best,
unfortunately, I
think the answer is that it all depends on the data ...
If your observing site is on the side of a steep mountain, it may be
best to
use the nearest neighbor method - so as to avoid using forecast values
from
grid points with very different elevations.
Whereas if your the area around your observation site is relatively
uniform,
averaging over a larger area is more justifiable.
The question of which interpolation method to use is frequently asked,
and my
general take on it is that it usually doesn't matter.
In Point-Stat and Grid-Stat, it's very easy to use multiple
interpolation
methods on the same run. So investigating the impact of interpolation
method
on your statistics is pretty easy. I'd suggest choosing a handful of
interpolation methods, running them for a sample of your data, and
look to see
how the statistics differ. I'm guessing you'll find that the choice
of
interpolation method has little impact on your results - especially
when you
consider the potential errors of the observing equipment itself.
One thing to note though is that, generally speaking, smoother
forecasts yield
higher traditional verification scores (e.g. RMSE) than hi-resolution
ones.
And when you interpolate over larger and larger numbers of points, it
has the
effect of smoothing the forecast. So I would guess you'd find that
RMSE
values improve overall as you use larger and larger interpolation
sizes.
Sorry there isn't an easy answer.
John
On 10/21/2011 11:35 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for that info. I was not aware that these were the most
common
> ones in use. They are different from the one I've been using which
is
> distance weighted mean and the reason for this was simply that my
> predecessor (Barb
> Sauter) used that one. Now, people are asking questions as to why we
> are using that method and I don't have a reason or justification.
Barb
> retired a couple of years ago, so I can't get her reasons.
>
> This gets me back to the another part of the original question which
> is to try and find out if there is any guidance out there which
> advises of the inherent strengths and weaknesses of each method.
>
> I suppose that I could arbitrarily change the methods and compare
the
> error stats I get with each, but it gets down to the question of
which
> set of stats do I consider the most valid and I don't know how I
would
> be able to answer that question. Because I don't know what the
> "perfect" error stat is, I can't judge which of the interpolation
> methods produces error stats closet to "perfect".
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, October 21, 2011 10:59 AM
> To: Raby, John W USA CIV (US)
> Cc: tressa at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
> (UNCLASSIFIED)
>
> John,
>
> I'm not aware of any references that would guide you as to which
> interpolation method you should choose. I can tell you that two
very
> common ones people choose are nearest-neighbor and bilinear
> interpolation of the 4 closets points. The following settings in
the
> Point-Stat config file would give you those two methods:
> interp_method[] = [ "BILIN" ];
> interp_width[] = [ 1, 2 ];
>
> The intention of providing very configurable interpolation options
was
> to facilitate people in studying what impact the interpolation
method
> choice has on the verification scores. But if you're looking for a
> single choice, I'd guess that bilinear interpolation of the 4
closest
> points is the most commonly used one.
>
> I've CC'ed Tressa on the met_help ticket in case she has any more
> information to offer.
>
> Thanks,
> John Halley Gotway
>
>
> On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>>
>> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Interpolation Methods (UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> Regarding the interpolation methods available for use in Point-
Stat,
>> do you have any information or references you can send me which
>> describe the relative merits, advantages/disadvantages,
shortcomings,
>> pitfalls, etc which I could use to compare and contrast the various
>> methods in order to select the best one for my use?
>>
>> Thanks.
>>
>> R/
>> John
>>
>> Mr John W. Raby, Meteorologist
>> U.S. Army Research Laboratory
>> White Sands Missile Range, NM 88002
>> (575) 678-2004 DSN 258-2004
>> FAX (575) 678-1230 DSN 258-1230
>> Email: john.w.raby2.civ at mail.mil
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Oct 21 14:01:48 2011
John,
Yes, the point I was making was that lower resolution models typically
have lower RMSE values than higher resolution models. Instead of
saying "higher" vx scores, I should have said "better" - since
for some scores higher is better and for others, lower is better.
But I decided to run a quick test here to see if the numbers bear out
my claim. I used the test case for Point-Stat that's distributed with
MET. I chose the unweighted-mean interpolation method
using widths of 1, 3, 5,..., 31. I ran it for two variables "TMP
P900-750" and "WIND Z10". For TMP, what I stated is true - the larger
the interpolation size, the lower the RMSE. But for WIND, the
opposite was true - the larger the size, the higher the RMSE. The
actual numbers are included below.
So I guess it really depends on the data!
John
[johnhg at rambler]% cat point_stat_360000L_20070331_120000V_cnt.txt |
egrep "WIND|MODEL" | egrep "ADPSFC|MODEL" | sed -r 's/ +/ /g' | cut
-d' ' -f9-10,13-16,75
FCST_VAR FCST_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS RMSE
WIND Z10 ADPSFC FULL UW_MEAN 1 2.49200
WIND Z10 ADPSFC FULL UW_MEAN 9 2.49328
WIND Z10 ADPSFC FULL UW_MEAN 25 2.50707
WIND Z10 ADPSFC FULL UW_MEAN 49 2.52199
WIND Z10 ADPSFC FULL UW_MEAN 81 2.53132
WIND Z10 ADPSFC FULL UW_MEAN 121 2.53513
WIND Z10 ADPSFC FULL UW_MEAN 169 2.53895
WIND Z10 ADPSFC FULL UW_MEAN 225 2.54130
WIND Z10 ADPSFC FULL UW_MEAN 289 2.54373
WIND Z10 ADPSFC FULL UW_MEAN 441 2.55230
WIND Z10 ADPSFC FULL UW_MEAN 529 2.55848
WIND Z10 ADPSFC FULL UW_MEAN 625 2.56552
WIND Z10 ADPSFC FULL UW_MEAN 729 2.57279
WIND Z10 ADPSFC FULL UW_MEAN 841 2.58002
WIND Z10 ADPSFC FULL UW_MEAN 961 2.58759
[johnhg at rambler]% cat point_stat_360000L_20070331_120000V_cnt.txt |
egrep "TMP|MODEL" | egrep "ADPUPA|MODEL" | sed -r 's/ +/ /g' | cut -d'
' -f9-10,13-16,75
FCST_VAR FCST_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS RMSE
TMP P900-750 ADPUPA FULL UW_MEAN 1 2.52597
TMP P900-750 ADPUPA FULL UW_MEAN 9 2.50120
TMP P900-750 ADPUPA FULL UW_MEAN 25 2.50412
TMP P900-750 ADPUPA FULL UW_MEAN 49 2.49679
TMP P900-750 ADPUPA FULL UW_MEAN 81 2.48384
TMP P900-750 ADPUPA FULL UW_MEAN 121 2.46361
TMP P900-750 ADPUPA FULL UW_MEAN 169 2.44203
TMP P900-750 ADPUPA FULL UW_MEAN 225 2.42300
TMP P900-750 ADPUPA FULL UW_MEAN 289 2.40548
TMP P900-750 ADPUPA FULL UW_MEAN 441 2.38303
TMP P900-750 ADPUPA FULL UW_MEAN 529 2.37634
TMP P900-750 ADPUPA FULL UW_MEAN 625 2.37169
TMP P900-750 ADPUPA FULL UW_MEAN 729 2.36807
TMP P900-750 ADPUPA FULL UW_MEAN 841 2.36550
TMP P900-750 ADPUPA FULL UW_MEAN 961 2.36424
On 10/21/2011 01:25 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your discussion on that subject. It will help me as I
attempt to
> provide some answers to the folks here.
>
> Good point about the observation location. I validate over 2 domains
centered
> over UT which are 546km x 546km (500+ obs) and 102km x 102km (~25
obs) using
> PrepBUFR plus mesonet observations from MADIS.
>
> I suspect that my observing sites fall into both categories you
mention in
> that area.
>
> Can you clarify your last statement to be sure I'm understanding
you?
>
> You said: " generally speaking, smoother forecasts yield higher
traditional
> verification scores (e.g. RMSE) than hi-resolution ones. And when
you
> interpolate over larger and larger numbers of points, it has the
effect of
> smoothing the forecast. So I would guess you'd find that RMSE
values improve
> overall as you use larger and larger interpolation sizes."
>
> My question is: when you say "smoother forecasts" do you mean lower
resolution
> (large grid spacing)?
>
> When you say "higher traditional verification scores" you mean
larger RMSE
> values (worse scores)?
>
>
> The reason I ask is that I thought that the lower resolution models
typically
> had lower RMSE values than high resolution models when you verify
with the
> traditional methods.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, October 21, 2011 1:02 PM
> To: Raby, John W USA CIV (US)
> Cc: tressa at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
>
> John,
>
> Prior to METv3.0.1, the "BILIN" interpolation option was not
available in
> Point-Stat. We had incorrectly instructed users that DW_MEAN with a
width of
> 2 was exactly the same as bilinear interpolation, when in fact it is
not. We
> realized the discrepancy and added direct support for bilinear
interpolation
> in METv3.0.1. I suspect that this is the reason Barb was using
DW_MEAN -
> intending it to be bilinear.
>
> To larger question of which interpolation method is best,
unfortunately, I
> think the answer is that it all depends on the data ...
>
> If your observing site is on the side of a steep mountain, it may be
best to
> use the nearest neighbor method - so as to avoid using forecast
values from
> grid points with very different elevations.
> Whereas if your the area around your observation site is relatively
uniform,
> averaging over a larger area is more justifiable.
>
> The question of which interpolation method to use is frequently
asked, and my
> general take on it is that it usually doesn't matter.
>
> In Point-Stat and Grid-Stat, it's very easy to use multiple
interpolation
> methods on the same run. So investigating the impact of
interpolation method
> on your statistics is pretty easy. I'd suggest choosing a handful
of
> interpolation methods, running them for a sample of your data, and
look to see
> how the statistics differ. I'm guessing you'll find that the choice
of
> interpolation method has little impact on your results - especially
when you
> consider the potential errors of the observing equipment itself.
>
> One thing to note though is that, generally speaking, smoother
forecasts yield
> higher traditional verification scores (e.g. RMSE) than hi-
resolution ones.
> And when you interpolate over larger and larger numbers of points,
it has the
> effect of smoothing the forecast. So I would guess you'd find that
RMSE
> values improve overall as you use larger and larger interpolation
sizes.
>
> Sorry there isn't an easy answer.
>
> John
>
> On 10/21/2011 11:35 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> John -
>>
>> Thanks for that info. I was not aware that these were the most
common
>> ones in use. They are different from the one I've been using which
is
>> distance weighted mean and the reason for this was simply that my
>> predecessor (Barb
>> Sauter) used that one. Now, people are asking questions as to why
we
>> are using that method and I don't have a reason or justification.
Barb
>> retired a couple of years ago, so I can't get her reasons.
>>
>> This gets me back to the another part of the original question
which
>> is to try and find out if there is any guidance out there which
>> advises of the inherent strengths and weaknesses of each method.
>>
>> I suppose that I could arbitrarily change the methods and compare
the
>> error stats I get with each, but it gets down to the question of
which
>> set of stats do I consider the most valid and I don't know how I
would
>> be able to answer that question. Because I don't know what the
>> "perfect" error stat is, I can't judge which of the interpolation
>> methods produces error stats closet to "perfect".
>>
>> R/
>> John
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Friday, October 21, 2011 10:59 AM
>> To: Raby, John W USA CIV (US)
>> Cc: tressa at ucar.edu
>> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
>> (UNCLASSIFIED)
>>
>> John,
>>
>> I'm not aware of any references that would guide you as to which
>> interpolation method you should choose. I can tell you that two
very
>> common ones people choose are nearest-neighbor and bilinear
>> interpolation of the 4 closets points. The following settings in
the
>> Point-Stat config file would give you those two methods:
>> interp_method[] = [ "BILIN" ];
>> interp_width[] = [ 1, 2 ];
>>
>> The intention of providing very configurable interpolation options
was
>> to facilitate people in studying what impact the interpolation
method
>> choice has on the verification scores. But if you're looking for a
>> single choice, I'd guess that bilinear interpolation of the 4
closest
>> points is the most commonly used one.
>>
>> I've CC'ed Tressa on the met_help ticket in case she has any more
>> information to offer.
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Interpolation Methods (UNCLASSIFIED)
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850
>>>>
>>>
>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>> Regarding the interpolation methods available for use in Point-
Stat,
>>> do you have any information or references you can send me which
>>> describe the relative merits, advantages/disadvantages,
shortcomings,
>>> pitfalls, etc which I could use to compare and contrast the
various
>>> methods in order to select the best one for my use?
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>>
>>> Mr John W. Raby, Meteorologist
>>> U.S. Army Research Laboratory
>>> White Sands Missile Range, NM 88002
>>> (575) 678-2004 DSN 258-2004
>>> FAX (575) 678-1230 DSN 258-1230
>>> Email: john.w.raby2.civ at mail.mil
>>>
>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>>
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #50850] Interpolation Methods (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Oct 21 14:17:18 2011
John -
Thanks for that clarification. I wanted to be sure we were both saying
the same thing and we were.
Interesting comparison and really an eye-opener about the fact that it
really does depend on which met variable you are working with.
Thanks for taking the time to answer those questions. Now I have some
info I can respond with here.
The bottom line is that maybe I should try testing with different
methods on my datasets to see if it bears out that it doesn't really
make much difference. If it doesn't, then stick with what I've been
using and if it does, then gain a consensus (management decision?) on
what method I should use (based on your info from this thread) and
move on (hopefully).
Its really great having you all there to bring these issues up with!
Have a great weekend!
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, October 21, 2011 2:01 PM
To: Raby, John W USA CIV (US)
Cc: tressa at ucar.edu
Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
John,
Yes, the point I was making was that lower resolution models typically
have lower RMSE values than higher resolution models. Instead of
saying "higher" vx scores, I should have said "better" - since
for some scores higher is better and for others, lower is better.
But I decided to run a quick test here to see if the numbers bear out
my claim. I used the test case for Point-Stat that's distributed with
MET. I chose the unweighted-mean interpolation method
using widths of 1, 3, 5,..., 31. I ran it for two variables "TMP
P900-750" and "WIND Z10". For TMP, what I stated is true - the larger
the interpolation size, the lower the RMSE. But for WIND, the
opposite was true - the larger the size, the higher the RMSE. The
actual numbers are included below.
So I guess it really depends on the data!
John
[johnhg at rambler]% cat point_stat_360000L_20070331_120000V_cnt.txt |
egrep "WIND|MODEL" | egrep "ADPSFC|MODEL" | sed -r 's/ +/ /g' | cut
-d' ' -f9-10,13-16,75
FCST_VAR FCST_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS RMSE
WIND Z10 ADPSFC FULL UW_MEAN 1 2.49200
WIND Z10 ADPSFC FULL UW_MEAN 9 2.49328
WIND Z10 ADPSFC FULL UW_MEAN 25 2.50707
WIND Z10 ADPSFC FULL UW_MEAN 49 2.52199
WIND Z10 ADPSFC FULL UW_MEAN 81 2.53132
WIND Z10 ADPSFC FULL UW_MEAN 121 2.53513
WIND Z10 ADPSFC FULL UW_MEAN 169 2.53895
WIND Z10 ADPSFC FULL UW_MEAN 225 2.54130
WIND Z10 ADPSFC FULL UW_MEAN 289 2.54373
WIND Z10 ADPSFC FULL UW_MEAN 441 2.55230
WIND Z10 ADPSFC FULL UW_MEAN 529 2.55848
WIND Z10 ADPSFC FULL UW_MEAN 625 2.56552
WIND Z10 ADPSFC FULL UW_MEAN 729 2.57279
WIND Z10 ADPSFC FULL UW_MEAN 841 2.58002
WIND Z10 ADPSFC FULL UW_MEAN 961 2.58759
[johnhg at rambler]% cat point_stat_360000L_20070331_120000V_cnt.txt |
egrep "TMP|MODEL" | egrep "ADPUPA|MODEL" | sed -r 's/ +/ /g' | cut -d'
' -f9-10,13-16,75
FCST_VAR FCST_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS RMSE
TMP P900-750 ADPUPA FULL UW_MEAN 1 2.52597
TMP P900-750 ADPUPA FULL UW_MEAN 9 2.50120
TMP P900-750 ADPUPA FULL UW_MEAN 25 2.50412
TMP P900-750 ADPUPA FULL UW_MEAN 49 2.49679
TMP P900-750 ADPUPA FULL UW_MEAN 81 2.48384
TMP P900-750 ADPUPA FULL UW_MEAN 121 2.46361
TMP P900-750 ADPUPA FULL UW_MEAN 169 2.44203
TMP P900-750 ADPUPA FULL UW_MEAN 225 2.42300
TMP P900-750 ADPUPA FULL UW_MEAN 289 2.40548
TMP P900-750 ADPUPA FULL UW_MEAN 441 2.38303
TMP P900-750 ADPUPA FULL UW_MEAN 529 2.37634
TMP P900-750 ADPUPA FULL UW_MEAN 625 2.37169
TMP P900-750 ADPUPA FULL UW_MEAN 729 2.36807
TMP P900-750 ADPUPA FULL UW_MEAN 841 2.36550
TMP P900-750 ADPUPA FULL UW_MEAN 961 2.36424
On 10/21/2011 01:25 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your discussion on that subject. It will help me as I
attempt to
> provide some answers to the folks here.
>
> Good point about the observation location. I validate over 2 domains
centered
> over UT which are 546km x 546km (500+ obs) and 102km x 102km (~25
obs) using
> PrepBUFR plus mesonet observations from MADIS.
>
> I suspect that my observing sites fall into both categories you
mention in
> that area.
>
> Can you clarify your last statement to be sure I'm understanding
you?
>
> You said: " generally speaking, smoother forecasts yield higher
traditional
> verification scores (e.g. RMSE) than hi-resolution ones. And when
you
> interpolate over larger and larger numbers of points, it has the
effect of
> smoothing the forecast. So I would guess you'd find that RMSE
values improve
> overall as you use larger and larger interpolation sizes."
>
> My question is: when you say "smoother forecasts" do you mean lower
resolution
> (large grid spacing)?
>
> When you say "higher traditional verification scores" you mean
larger RMSE
> values (worse scores)?
>
>
> The reason I ask is that I thought that the lower resolution models
typically
> had lower RMSE values than high resolution models when you verify
with the
> traditional methods.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, October 21, 2011 1:02 PM
> To: Raby, John W USA CIV (US)
> Cc: tressa at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
(UNCLASSIFIED)
>
> John,
>
> Prior to METv3.0.1, the "BILIN" interpolation option was not
available in
> Point-Stat. We had incorrectly instructed users that DW_MEAN with a
width of
> 2 was exactly the same as bilinear interpolation, when in fact it is
not. We
> realized the discrepancy and added direct support for bilinear
interpolation
> in METv3.0.1. I suspect that this is the reason Barb was using
DW_MEAN -
> intending it to be bilinear.
>
> To larger question of which interpolation method is best,
unfortunately, I
> think the answer is that it all depends on the data ...
>
> If your observing site is on the side of a steep mountain, it may be
best to
> use the nearest neighbor method - so as to avoid using forecast
values from
> grid points with very different elevations.
> Whereas if your the area around your observation site is relatively
uniform,
> averaging over a larger area is more justifiable.
>
> The question of which interpolation method to use is frequently
asked, and my
> general take on it is that it usually doesn't matter.
>
> In Point-Stat and Grid-Stat, it's very easy to use multiple
interpolation
> methods on the same run. So investigating the impact of
interpolation method
> on your statistics is pretty easy. I'd suggest choosing a handful
of
> interpolation methods, running them for a sample of your data, and
look to see
> how the statistics differ. I'm guessing you'll find that the choice
of
> interpolation method has little impact on your results - especially
when you
> consider the potential errors of the observing equipment itself.
>
> One thing to note though is that, generally speaking, smoother
forecasts yield
> higher traditional verification scores (e.g. RMSE) than hi-
resolution ones.
> And when you interpolate over larger and larger numbers of points,
it has the
> effect of smoothing the forecast. So I would guess you'd find that
RMSE
> values improve overall as you use larger and larger interpolation
sizes.
>
> Sorry there isn't an easy answer.
>
> John
>
> On 10/21/2011 11:35 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850 >
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> John -
>>
>> Thanks for that info. I was not aware that these were the most
common
>> ones in use. They are different from the one I've been using which
is
>> distance weighted mean and the reason for this was simply that my
>> predecessor (Barb
>> Sauter) used that one. Now, people are asking questions as to why
we
>> are using that method and I don't have a reason or justification.
Barb
>> retired a couple of years ago, so I can't get her reasons.
>>
>> This gets me back to the another part of the original question
which
>> is to try and find out if there is any guidance out there which
>> advises of the inherent strengths and weaknesses of each method.
>>
>> I suppose that I could arbitrarily change the methods and compare
the
>> error stats I get with each, but it gets down to the question of
which
>> set of stats do I consider the most valid and I don't know how I
would
>> be able to answer that question. Because I don't know what the
>> "perfect" error stat is, I can't judge which of the interpolation
>> methods produces error stats closet to "perfect".
>>
>> R/
>> John
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Friday, October 21, 2011 10:59 AM
>> To: Raby, John W USA CIV (US)
>> Cc: tressa at ucar.edu
>> Subject: Re: [rt.rap.ucar.edu #50850] Interpolation Methods
>> (UNCLASSIFIED)
>>
>> John,
>>
>> I'm not aware of any references that would guide you as to which
>> interpolation method you should choose. I can tell you that two
very
>> common ones people choose are nearest-neighbor and bilinear
>> interpolation of the 4 closets points. The following settings in
the
>> Point-Stat config file would give you those two methods:
>> interp_method[] = [ "BILIN" ];
>> interp_width[] = [ 1, 2 ];
>>
>> The intention of providing very configurable interpolation options
was
>> to facilitate people in studying what impact the interpolation
method
>> choice has on the verification scores. But if you're looking for a
>> single choice, I'd guess that bilinear interpolation of the 4
closest
>> points is the most commonly used one.
>>
>> I've CC'ed Tressa on the met_help ticket in case she has any more
>> information to offer.
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On 10/21/2011 07:49 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> Fri Oct 21 07:49:23 2011: Request 50850 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Interpolation Methods (UNCLASSIFIED)
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=50850
>>>>
>>>
>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>> Regarding the interpolation methods available for use in Point-
Stat,
>>> do you have any information or references you can send me which
>>> describe the relative merits, advantages/disadvantages,
shortcomings,
>>> pitfalls, etc which I could use to compare and contrast the
various
>>> methods in order to select the best one for my use?
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>>
>>> Mr John W. Raby, Meteorologist
>>> U.S. Army Research Laboratory
>>> White Sands Missile Range, NM 88002
>>> (575) 678-2004 DSN 258-2004
>>> FAX (575) 678-1230 DSN 258-1230
>>> Email: john.w.raby2.civ at mail.mil
>>>
>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>>
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
More information about the Met_help
mailing list