[Met_help] [rt.rap.ucar.edu #60133] History for MET Question
John Halley Gotway via RT
met_help at ucar.edu
Tue Mar 26 09:58:32 MDT 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
My name is Andrew, and I am a meteorologist doing some statistical analysis with the MET system. I would like to examine not only the statistical errors associated with model output (we'll see WRF for example), but also the results of a downscaling algorithm that I have applied to the WRF model output. The problem (for MET purposes at least), is that my downscaling and interpolation algorithm makes forecasts exactly at the confirmation station points. In other words, my confirmation points are the exact same as my forecast points, and the forecast points after downscaling and interpolation are no longer on a grid (they are dispersed throughout the country at the same location as the station models). Is there any way for MET to compare point forecasts (not grid forecasts) with point observations? If not, would you have any suggestions of programs that might be able to do this for me? Thank you in advance...
Andrew
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #60133] MET Question
From: John Halley Gotway
Time: Fri Feb 01 08:54:17 2013
Andrew,
Good question. The basic answer to your question is no. MET was
designed to verify gridded forecasts against gridded or point
observations.
But there are a couple of workarounds you could consider.
Here's one approach... If you have a bunch of point forecasts for
stations around the country, it shouldn't be too hard to "grid" them
by creating a NetCDF file who's values are bad data everywhere
except for the handful of grid points where you have computed a
downscaled value. Then you could verify that gridded forecast against
point observations using the Point-Stat tool.
Here's a second approach... Your downscaling method is basically
producing a forecast value at stations for which you already know the
observation value. So really, you already have forecast and
observation matched pair data. (Often, that's the most difficult part
of verification!) You could simply reformat that matched pair data to
look like the matched pair (MPR) output lines from the
Point-Stat tool. Then just save them all in a file that ends with a
".stat" extension. Then, run the STAT-Analysis tool too read in those
matched pairs and compute whatever verification statistics
you'd like.
In this case, you STAT-Analysis job might look something like this:
stat_analysis -lookin my_data.stat -job aggregate_stat -line_type
MPR -out_line_type CNT
That'd read in all the matched pair lines and compute the
corresponding continuous statistics (like RMSE, for example). STAT-
Analysis has the ability to filter your data down however you'd like
and
compute all the traditional types of continuous, categorical, and
probabilistic statistics.
Either route will require some work on your part - either creating a
gridded NetCDF file or reformatting your ASCII data.
There is a third alternative outside of MET. If you happen to be
familiar with R, you could read your forecast and observation matched
pair values into R and use the "verification" package to compute
stats on them.
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 02/01/2013 04:58 AM, Andrew J. via RT wrote:
>
> Fri Feb 01 04:58:28 2013: Request 60133 was acted upon.
> Transaction: Ticket created by andrewwx at yahoo.com
> Queue: met_help
> Subject: MET Question
> Owner: Nobody
> Requestors: andrewwx at yahoo.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>
>
> Hello,
>
> My name is Andrew, and I am a meteorologist doing some statistical
analysis with the MET system. I would like to examine not only the
statistical errors associated with model output (we'll see WRF for
example), but also the results of a downscaling algorithm that I have
applied to the WRF model output. The problem (for MET purposes at
least), is that my downscaling and interpolation algorithm makes
forecasts exactly at the confirmation station points. In other words,
my confirmation points are the exact same as my forecast points, and
the forecast points after downscaling and interpolation are no longer
on a grid (they are dispersed throughout the country at the same
location as the station models). Is there any way for MET to compare
point forecasts (not grid forecasts) with point observations? If not,
would you have any suggestions of programs that might be able to do
this for me? Thank you in advance...
>
> Andrew
>
------------------------------------------------
Subject: MET Question
From: Andrew J.
Time: Fri Mar 08 11:44:53 2013
Hello,
I have follow up question to this e-mail...
You have suggested verifying my point observations by computing our
statistics over a set of matched points.
Great idea, I have a structure now set up to do that.
However, in the process, I decided to proof this method and verify
model fields using two different methods to see if they were the same.
The methods were as follows:
1) I output CNT statistics from point-stat data and then examined a "-
job summary" of these statistics over ~20 parameters (RMSE, MAE, etc.)
2) I did not initally compute CNT statistics, but rather I output
matched pairs, and then used '-job aggregate_stat' to create the same
CNT parameters (RMSE, MAE, etc.)
I expected these two methods to produce an exactly identical outcome.
However, while the values are similar, they are different in almost
all cases, sometimes by as much
as 0.5 (in the RMSE column). These results challenged my way of
thinking about the MET system, because I was under the impression that
the CNT statistics originally calculated in the point-stat step were
taken from the matched pair data. Therefore, these two methods should
have produced the same result. Is there something that I am
overlooking? Or perhaps a weighting technique that is internally
varied between the two methods? As always...thank you for your time.
________________________________
Von: John Halley Gotway via RT <met_help at ucar.edu>
An: andrewwx at yahoo.com
Gesendet: 16:54 Freitag, 1.Februar 2013
Betreff: Re: [rt.rap.ucar.edu #60133] MET Question
Andrew,
Good question. The basic answer to your question is no. MET was
designed to verify gridded forecasts against gridded or point
observations.
But there are a couple of workarounds you could consider.
Here's one approach... If you have a bunch of point forecasts for
stations around the country, it shouldn't be too hard to "grid" them
by creating a NetCDF file who's values are bad data everywhere
except for the handful of grid points where you have computed a
downscaled value. Then you could verify that gridded forecast against
point observations using the Point-Stat tool.
Here's a second approach... Your downscaling method is basically
producing a forecast value at stations for which you already know the
observation value. So really, you already have forecast and
observation matched pair data. (Often, that's the most difficult part
of verification!) You could simply reformat that matched pair data to
look like the matched pair (MPR) output lines from the
Point-Stat tool. Then just save them all in a file that ends with a
".stat" extension. Then, run the STAT-Analysis tool too read in those
matched pairs and compute whatever verification statistics
you'd like.
In this case, you STAT-Analysis job might look something like this:
stat_analysis -lookin my_data.stat -job aggregate_stat -line_type
MPR -out_line_type CNT
That'd read in all the matched pair lines and compute the
corresponding continuous statistics (like RMSE, for example). STAT-
Analysis has the ability to filter your data down however you'd like
and
compute all the traditional types of continuous, categorical, and
probabilistic statistics.
Either route will require some work on your part - either creating a
gridded NetCDF file or reformatting your ASCII data.
There is a third alternative outside of MET. If you happen to be
familiar with R, you could read your forecast and observation matched
pair values into R and use the "verification" package to compute
stats on them.
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 02/01/2013 04:58 AM, Andrew J. via RT wrote:
>
> Fri Feb 01 04:58:28 2013: Request 60133 was acted upon.
> Transaction: Ticket created by andrewwx at yahoo.com
> Queue: met_help
> Subject: MET Question
> Owner: Nobody
> Requestors: andrewwx at yahoo.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>
>
> Hello,
>
> My name is Andrew, and I am a meteorologist doing some statistical
analysis with the MET system. I would like to examine not only the
statistical errors associated with model output (we'll see WRF for
example), but also the results of a downscaling algorithm that I have
applied to the WRF model output. The problem (for MET purposes at
least), is that my downscaling and interpolation algorithm makes
forecasts exactly at the confirmation station points. In other words,
my confirmation points are the exact same as my forecast points, and
the forecast points after downscaling and interpolation are no longer
on a grid (they are dispersed throughout the country at the same
location as the station models). Is there any way for MET to compare
point forecasts (not grid forecasts) with point observations? If not,
would you have any suggestions of programs that might be able to do
this for me? Thank you in advance...
>
> Andrew
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #60133] MET Question
From: John Halley Gotway
Time: Fri Mar 08 17:02:20 2013
Andrew,
Hopefully, I'm interpreting your question correctly here. Here's what
I'm assuming:
- I assume that you ran Point-Stat multiple times to compute about
20 CNT lines and also dump out a bunch of MPR lines for each case.
- Then I assume you ran 2 jobs:
- Job 1 is something like: -job summary -line_type CNT
-column RMSE
- Job 2 is something like: -job aggregate_stat -line_type MPR
-out_line_type CNT
- Then I assume you're looking at the *MEAN* of the 20 RMSE values
from job 1 and comparing it to the RMSE value computed in job 2.
- You see that those numbers differ, and you're not sure why.
Please let me know if my assumptions are not correct.
Those two you computed numbers will not, in general, be the same.
They'll only match if exactly the same number of matched pairs went
into each of the 20 computations of the CNT line.
The summary job is just taking a mean of the 20 RMSE values,
regardless of the number of matched pairs that went into them. The
aggregate_stat job is operating on the individual matched pair values
-
and each matched pair is weighted equally.
That, I suspect, is the source of the difference. The two jobs you
ran really are looking at separate quantities. The summary job
describes how the RMSE values change from case to case. The
aggregate_stat job describes performance aggregated across all cases
and all points. So it really is a subtle difference in weighting.
Generally speaking, I think statisticians prefer the
aggregate_stat approach to the summary approach.
To be clear though, if you look at just one of the RMSE values that
Point-Stat generated, and then use STAT-Analysis to recompute the CNT
stats from the corresponding MPR lines, you should get exactly
identical results. If you find that's not the case, please send me
some sample data to illustrate it.
Hope that helps clarify.
Thanks,
John
On 03/08/2013 11:44 AM, Andrew J. via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>
> Hello,
>
> I have follow up question to this e-mail...
>
> You have suggested verifying my point observations by computing our
statistics over a set of matched points.
>
> Great idea, I have a structure now set up to do that.
>
> However, in the process, I decided to proof this method and verify
model fields using two different methods to see if they were the same.
The methods were as follows:
>
> 1) I output CNT statistics from point-stat data and then examined a
"-job summary" of these statistics over ~20 parameters (RMSE, MAE,
etc.)
> 2) I did not initally compute CNT statistics, but rather I output
matched pairs, and then used '-job aggregate_stat' to create the same
CNT parameters (RMSE, MAE, etc.)
>
> I expected these two methods to produce an exactly identical
outcome. However, while the values are similar, they are different in
almost all cases, sometimes by as much
> as 0.5 (in the RMSE column). These results challenged my way of
thinking about the MET system, because I was under the impression that
the CNT statistics originally calculated in the point-stat step were
taken from the matched pair data. Therefore, these two methods should
have produced the same result. Is there something that I am
overlooking? Or perhaps a weighting technique that is internally
varied between the two methods? As always...thank you for your time.
>
>
>
>
> ________________________________
> Von: John Halley Gotway via RT <met_help at ucar.edu>
> An: andrewwx at yahoo.com
> Gesendet: 16:54 Freitag, 1.Februar 2013
> Betreff: Re: [rt.rap.ucar.edu #60133] MET Question
>
> Andrew,
>
> Good question. The basic answer to your question is no. MET was
designed to verify gridded forecasts against gridded or point
observations.
>
> But there are a couple of workarounds you could consider.
>
> Here's one approach... If you have a bunch of point forecasts for
stations around the country, it shouldn't be too hard to "grid" them
by creating a NetCDF file who's values are bad data everywhere
> except for the handful of grid points where you have computed a
downscaled value. Then you could verify that gridded forecast against
point observations using the Point-Stat tool.
>
> Here's a second approach... Your downscaling method is basically
producing a forecast value at stations for which you already know the
observation value. So really, you already have forecast and
> observation matched pair data. (Often, that's the most difficult
part of verification!) You could simply reformat that matched pair
data to look like the matched pair (MPR) output lines from the
> Point-Stat tool. Then just save them all in a file that ends with a
".stat" extension. Then, run the STAT-Analysis tool too read in those
matched pairs and compute whatever verification statistics
> you'd like.
>
> In this case, you STAT-Analysis job might look something like this:
> stat_analysis -lookin my_data.stat -job aggregate_stat
-line_type MPR -out_line_type CNT
>
> That'd read in all the matched pair lines and compute the
corresponding continuous statistics (like RMSE, for example). STAT-
Analysis has the ability to filter your data down however you'd like
and
> compute all the traditional types of continuous, categorical, and
probabilistic statistics.
>
> Either route will require some work on your part - either creating a
gridded NetCDF file or reformatting your ASCII data.
>
> There is a third alternative outside of MET. If you happen to be
familiar with R, you could read your forecast and observation matched
pair values into R and use the "verification" package to compute
> stats on them.
>
> Hope that helps.
>
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 02/01/2013 04:58 AM, Andrew J. via RT wrote:
>>
>> Fri Feb 01 04:58:28 2013: Request 60133 was acted upon.
>> Transaction: Ticket created by andrewwx at yahoo.com
>> Queue: met_help
>> Subject: MET Question
>> Owner: Nobody
>> Requestors: andrewwx at yahoo.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>>
>>
>> Hello,
>>
>> My name is Andrew, and I am a meteorologist doing some statistical
analysis with the MET system. I would like to examine not only the
statistical errors associated with model output (we'll see WRF for
example), but also the results of a downscaling algorithm that I have
applied to the WRF model output. The problem (for MET purposes at
least), is that my downscaling and interpolation algorithm makes
forecasts exactly at the confirmation station points. In other words,
my confirmation points are the exact same as my forecast points, and
the forecast points after downscaling and interpolation are no longer
on a grid (they are dispersed throughout the country at the same
location as the station models). Is there any way for MET to compare
point forecasts (not grid forecasts) with point observations? If not,
would you have any suggestions of programs that might be able to do
this for me? Thank you in advance...
>>
>> Andrew
>>
------------------------------------------------
Subject: MET Question
From: Andrew J.
Time: Sat Mar 09 06:51:11 2013
Oh right, of course. Sorry, I dont know why I didnt think of that.
The weighting would only be identical if every run had the exact same
number of matched pairs. Dumb math moment by me. Thank you for your
answer...
________________________________
Von: John Halley Gotway via RT <met_help at ucar.edu>
An: andrewwx at yahoo.com
Gesendet: 18:02 Freitag, 8.März 2013
Betreff: Re: [rt.rap.ucar.edu #60133] MET Question
Andrew,
Hopefully, I'm interpreting your question correctly here. Here's what
I'm assuming:
- I assume that you ran Point-Stat multiple times to compute about
20 CNT lines and also dump out a bunch of MPR lines for each case.
- Then I assume you ran 2 jobs:
- Job 1 is something like: -job summary -line_type CNT
-column RMSE
- Job 2 is something like: -job aggregate_stat -line_type MPR
-out_line_type CNT
- Then I assume you're looking at the *MEAN* of the 20 RMSE values
from job 1 and comparing it to the RMSE value computed in job 2.
- You see that those numbers differ, and you're not sure why.
Please let me know if my assumptions are not correct.
Those two you computed numbers will not, in general, be the same.
They'll only match if exactly the same number of matched pairs went
into each of the 20 computations of the CNT line.
The summary job is just taking a mean of the 20 RMSE values,
regardless of the number of matched pairs that went into them. The
aggregate_stat job is operating on the individual matched pair values
-
and each matched pair is weighted equally.
That, I suspect, is the source of the difference. The two jobs you
ran really are looking at separate quantities. The summary job
describes how the RMSE values change from case to case. The
aggregate_stat job describes performance aggregated across all cases
and all points. So it really is a subtle difference in weighting.
Generally speaking, I think statisticians prefer the
aggregate_stat approach to the summary approach.
To be clear though, if you look at just one of the RMSE values that
Point-Stat generated, and then use STAT-Analysis to recompute the CNT
stats from the corresponding MPR lines, you should get exactly
identical results. If you find that's not the case, please send me
some sample data to illustrate it.
Hope that helps clarify.
Thanks,
John
On 03/08/2013 11:44 AM, Andrew J. via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>
> Hello,
>
> I have follow up question to this e-mail...
>
> You have suggested verifying my point observations by computing our
statistics over a set of matched points.
>
> Great idea, I have a structure now set up to do that.
>
> However, in the process, I decided to proof this method and verify
model fields using two different methods to see if they were the same.
The methods were as follows:
>
> 1) I output CNT statistics from point-stat data and then examined a
"-job summary" of these statistics over ~20 parameters (RMSE, MAE,
etc.)
> 2) I did not initally compute CNT statistics, but rather I output
matched pairs, and then used '-job aggregate_stat' to create the same
CNT parameters (RMSE, MAE, etc.)
>
> I expected these two methods to produce an exactly identical
outcome. However, while the values are similar, they are different in
almost all cases, sometimes by as much
> as 0.5 (in the RMSE column). These results challenged my way of
thinking about the MET system, because I was under the impression that
the CNT statistics originally calculated in the point-stat step were
taken from the matched pair data. Therefore, these two methods should
have produced the same result. Is there something that I am
overlooking? Or perhaps a weighting technique that is internally
varied between the two methods? As always...thank you for your time.
>
>
>
>
> ________________________________
> Von: John Halley Gotway via RT <met_help at ucar.edu>
> An: andrewwx at yahoo.com
> Gesendet: 16:54 Freitag, 1.Februar 2013
> Betreff: Re: [rt.rap.ucar.edu #60133] MET Question
>
> Andrew,
>
> Good question. The basic answer to your question is no. MET was
designed to verify gridded forecasts against gridded or point
observations.
>
> But there are a couple of workarounds you could consider.
>
> Here's one approach... If you have a bunch of point forecasts for
stations around the country, it shouldn't be too hard to "grid" them
by creating a NetCDF file who's values are bad data everywhere
> except for the handful of grid points where you have computed a
downscaled value. Then you could verify that gridded forecast against
point observations using the Point-Stat tool.
>
> Here's a second approach... Your downscaling method is basically
producing a forecast value at stations for which you already know the
observation value. So really, you already have forecast and
> observation matched pair data. (Often, that's the most difficult
part of verification!) You could simply reformat that matched pair
data to look like the matched pair (MPR) output lines from the
> Point-Stat tool. Then just save them all in a file that ends with a
".stat" extension. Then, run the STAT-Analysis tool too read in those
matched pairs and compute whatever verification statistics
> you'd like.
>
> In this case, you STAT-Analysis job might look something like this:
> stat_analysis -lookin my_data.stat -job aggregate_stat
-line_type MPR -out_line_type CNT
>
> That'd read in all the matched pair lines and compute the
corresponding continuous statistics (like RMSE, for example). STAT-
Analysis has the ability to filter your data down however you'd like
and
> compute all the traditional types of continuous, categorical, and
probabilistic statistics.
>
> Either route will require some work on your part - either creating a
gridded NetCDF file or reformatting your ASCII data.
>
> There is a third alternative outside of MET. If you happen to be
familiar with R, you could read your forecast and observation matched
pair values into R and use the "verification" package to compute
> stats on them.
>
> Hope that helps.
>
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 02/01/2013 04:58 AM, Andrew J. via RT wrote:
>>
>> Fri Feb 01 04:58:28 2013: Request 60133 was acted upon.
>> Transaction: Ticket created by andrewwx at yahoo.com
>> Queue: met_help
>> Subject: MET Question
>> Owner: Nobody
>> Requestors: andrewwx at yahoo.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60133 >
>>
>>
>> Hello,
>>
>> My name is Andrew, and I am a meteorologist doing some statistical
analysis with the MET system. I would like to examine not only the
statistical errors associated with model output (we'll see WRF for
example), but also the results of a downscaling algorithm that I have
applied to the WRF model output. The problem (for MET purposes at
least), is that my downscaling and interpolation algorithm makes
forecasts exactly at the confirmation station points. In other words,
my confirmation points are the exact same as my forecast points, and
the forecast points after downscaling and interpolation are no longer
on a grid (they are dispersed throughout the country at the same
location as the station models). Is there any way for MET to compare
point forecasts (not grid forecasts) with point observations? If not,
would you have any suggestions of programs that might be able to do
this for me? Thank you in advance...
>>
>> Andrew
>>
------------------------------------------------
More information about the Met_help
mailing list