[Met_help] [rt.rap.ucar.edu #41328] History for 5-Minute Winds in WRF
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Wed Oct 13 13:07:12 MDT 2010
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
As I mentioned in an earlier met_help thread, I've added code to WRF to compute 5-min average u and v winds at both 10 m and 100 m, and would like to use the Texas Tech Reese Tower obs to verify the winds. Based on John's earlier comments I've created NetCDF files for the obs each day, and I've modified WPP/WRFPOST to create GRIB files of the WRF 5-min average u and v winds (along with speed and direction). In the GRIB files I coded the valid time for the wind data in minutes from the forecast start time so that each field is valid at 5 minutes, 10 minutes, 15 minutes, etc., throughout the length of the forecast. This should allow MET to match the obs and the forecast perfectly in time.
For some reason I can't get Point-Stat to give me any results (the output files contain headers only, no data), and I'm not sure if the problem is with my input data or the way I've configured Point-Stat. I wonder if I could pass along some sample data to see if you can help diagnose my problem?
Thanks,
Scott
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
From: John Halley Gotway
Time: Wed Oct 06 13:57:27 2010
Scott,
Sure, we'd be happy to help.
First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
used. That may provide you with a clue of the problem.
Here's an example of what the output from Point-Stat will look like:
Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
Number of matched pairs = 334
Observations processed = 91899
Rejected: GRIB code = 81017
Rejected: valid time = 0
Rejected: bad obs value = 0
Rejected: off the grid = 6
Rejected: level mismatch = 9925
Rejected: message type = 374
Rejected: masking region = 243
Rejected: bad fcst value = 0
If you are still having problems, feel free to post data to our ftp
site by following these instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
Thanks,
John
RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
> Transaction: Ticket created by scott.r.dembek at nasa.gov
> Queue: met_help
> Subject: 5-Minute Winds in WRF
> Owner: Nobody
> Requestors: scott.r.dembek at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
>
> As I mentioned in an earlier met_help thread, I've added code to WRF
to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>
> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>
> Thanks,
> Scott
>
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: Dembek, Scott Robert[Universities Space Research Association]
Time: Thu Oct 07 09:45:08 2010
Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
Regarding the obs, MET says it is "using 0 pairs" for both UGRD and
VGRD. For UGRD it processed 2224 obs which is correct based on the
number of obs in the NetCDF file I created from the Reese Tower data
on this particular day. It rejected 1668 because of the GRIB code --
I understand this because I have u, v, wspd and wdir for both 10 m and
100 m so that number makes sense to me. Beyond that, even though MET
reported only a single time from the GRIB file I would have thought it
could find the matching time in the NetCDF file because an ob does
exist at that exact time. Instead it says it rejected 510 obs based
on valid time and 46 based on level mismatch. I don't understand that
breakdown. And I get the exact same feedback for VGRD/Z010.
Any guidance you can offer short of me sending over sample data?
Thanks again,
Scott
________________________________
From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
Sent: Wednesday, October 06, 2010 3:57 PM
To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
Scott,
Sure, we'd be happy to help.
First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
used. That may provide you with a clue of the problem.
Here's an example of what the output from Point-Stat will look like:
Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
Number of matched pairs = 334
Observations processed = 91899
Rejected: GRIB code = 81017
Rejected: valid time = 0
Rejected: bad obs value = 0
Rejected: off the grid = 6
Rejected: level mismatch = 9925
Rejected: message type = 374
Rejected: masking region = 243
Rejected: bad fcst value = 0
If you are still having problems, feel free to post data to our ftp
site by following these instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
Thanks,
John
RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
> Transaction: Ticket created by scott.r.dembek at nasa.gov
> Queue: met_help
> Subject: 5-Minute Winds in WRF
> Owner: Nobody
> Requestors: scott.r.dembek at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
>
> As I mentioned in an earlier met_help thread, I've added code to WRF
to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>
> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>
> Thanks,
> Scott
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
From: John Halley Gotway
Time: Thu Oct 07 10:06:07 2010
Scott,
The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
the record for the 25-minute forecast in that file.
Since you're verifying every 5 minutes, in the Point-Stat config file,
I'd set "beg_ds" and "end_ds" very small, probably set them to -150
and 150, respectively. That means, only use observations
that are valid +/- 2.5 minutes around the forecast valid time.
Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
on only the MPR line type in the "output_flag" parameter. Then, once
you've run Point-Stat for all of the output times you'd like, use the
STAT-Analysis tool to compute a bunch of statistics. With
STAT-Analysis, you can run jobs like this:
# Compute continuous statistics (I'm assuming the station name is
REESE)
stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
# Compute categorical statistics
stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
The STAT-Analysis tool can be used to read in matched pair lines and
recompute any of the stats that Point-Stat can compute.
Lastly, why are you still getting 0 matched pairs. I really don't
know.
Go ahead and send me some sample data:
- forecast GRIB file
- ASCII and/or NetCDF point observation file
- Point-Stat config file
- command line you use to run Point-Stat
I'll take a look and see if I can figure out what's going on.
Just follow these directions for posting data to our anonymous ftp
site:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>
> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>
> Regarding the obs, MET says it is "using 0 pairs" for both UGRD and
VGRD. For UGRD it processed 2224 obs which is correct based on the
number of obs in the NetCDF file I created from the Reese Tower data
on this particular day. It rejected 1668 because of the GRIB code --
I understand this because I have u, v, wspd and wdir for both 10 m and
100 m so that number makes sense to me. Beyond that, even though MET
reported only a single time from the GRIB file I would have thought it
could find the matching time in the NetCDF file because an ob does
exist at that exact time. Instead it says it rejected 510 obs based
on valid time and 46 based on level mismatch. I don't understand that
breakdown. And I get the exact same feedback for VGRD/Z010.
>
> Any guidance you can offer short of me sending over sample data?
>
> Thanks again,
> Scott
>
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Wednesday, October 06, 2010 3:57 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> Sure, we'd be happy to help.
>
> First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
> used. That may provide you with a clue of the problem.
>
> Here's an example of what the output from Point-Stat will look like:
>
> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
> Number of matched pairs = 334
> Observations processed = 91899
> Rejected: GRIB code = 81017
> Rejected: valid time = 0
> Rejected: bad obs value = 0
> Rejected: off the grid = 6
> Rejected: level mismatch = 9925
> Rejected: message type = 374
> Rejected: masking region = 243
> Rejected: bad fcst value = 0
>
> If you are still having problems, feel free to post data to our ftp
site by following these instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>
> Thanks,
> John
>
> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>> Queue: met_help
>> Subject: 5-Minute Winds in WRF
>> Owner: Nobody
>> Requestors: scott.r.dembek at nasa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>>
>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>
>> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>>
>> Thanks,
>> Scott
>>
>
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: Dembek, Scott Robert[Universities Space Research Association]
Time: Thu Oct 07 12:42:36 2010
Thanks for the help, John! I put some sample data on the server after
making the changes you suggested to the Point-Stat config file. I'm
going to be away from my desk until Monday. If you need anything I
haven't already provided I'll get it to you first thing Monday.
Regards,
Scott
________________________________
From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
Sent: Thursday, October 07, 2010 12:06 PM
To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
Scott,
The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
the record for the 25-minute forecast in that file.
Since you're verifying every 5 minutes, in the Point-Stat config file,
I'd set "beg_ds" and "end_ds" very small, probably set them to -150
and 150, respectively. That means, only use observations
that are valid +/- 2.5 minutes around the forecast valid time.
Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
on only the MPR line type in the "output_flag" parameter. Then, once
you've run Point-Stat for all of the output times you'd like, use the
STAT-Analysis tool to compute a bunch of statistics. With
STAT-Analysis, you can run jobs like this:
# Compute continuous statistics (I'm assuming the station name is
REESE)
stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
# Compute categorical statistics
stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
The STAT-Analysis tool can be used to read in matched pair lines and
recompute any of the stats that Point-Stat can compute.
Lastly, why are you still getting 0 matched pairs. I really don't
know.
Go ahead and send me some sample data:
- forecast GRIB file
- ASCII and/or NetCDF point observation file
- Point-Stat config file
- command line you use to run Point-Stat
I'll take a look and see if I can figure out what's going on.
Just follow these directions for posting data to our anonymous ftp
site:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>
> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>
> Regarding the obs, MET says it is "using 0 pairs" for both UGRD and
VGRD. For UGRD it processed 2224 obs which is correct based on the
number of obs in the NetCDF file I created from the Reese Tower data
on this particular day. It rejected 1668 because of the GRIB code --
I understand this because I have u, v, wspd and wdir for both 10 m and
100 m so that number makes sense to me. Beyond that, even though MET
reported only a single time from the GRIB file I would have thought it
could find the matching time in the NetCDF file because an ob does
exist at that exact time. Instead it says it rejected 510 obs based
on valid time and 46 based on level mismatch. I don't understand that
breakdown. And I get the exact same feedback for VGRD/Z010.
>
> Any guidance you can offer short of me sending over sample data?
>
> Thanks again,
> Scott
>
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Wednesday, October 06, 2010 3:57 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> Sure, we'd be happy to help.
>
> First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
> used. That may provide you with a clue of the problem.
>
> Here's an example of what the output from Point-Stat will look like:
>
> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
> Number of matched pairs = 334
> Observations processed = 91899
> Rejected: GRIB code = 81017
> Rejected: valid time = 0
> Rejected: bad obs value = 0
> Rejected: off the grid = 6
> Rejected: level mismatch = 9925
> Rejected: message type = 374
> Rejected: masking region = 243
> Rejected: bad fcst value = 0
>
> If you are still having problems, feel free to post data to our ftp
site by following these instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>
> Thanks,
> John
>
> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>> Queue: met_help
>> Subject: 5-Minute Winds in WRF
>> Owner: Nobody
>> Requestors: scott.r.dembek at nasa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>>
>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>
>> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>>
>> Thanks,
>> Scott
>>
>
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: John Halley Gotway
Time: Tue Oct 12 10:48:42 2010
Scott,
OK, I ran you're data through Point-Stat, and I see what's going on
here.
Let me say up front, I'd suggest changing the output_flag parameter:
FROM: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2
];
TO: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1
];
There's no real need to dump out both ".txt" and ".stat" versions of
the same data. I'd suggest just using the ".stat" data to save space.
Now, here's an excerpt from the Point-Stat output when I the Point-
Stat command you sent me:
--------------------------------------------------------------------------------
Processing UGRD/Z10 versus UGRD/Z10, for observation type MSONET, over
region FULL, for interpolation method UW_MEAN(1), using 0 pairs.
Number of matched pairs = 0
Observations processed = 2224
Rejected: GRIB code = 1668
Rejected: valid time = 554
Rejected: bad obs value = 0
Rejected: off the grid = 0
Rejected: level mismatch = 2
Rejected: message type = 0
Rejected: masking region = 0
Rejected: bad fcst value = 0
--------------------------------------------------------------------------------
So the question is, why are we getting those 2 level mismatches and 0
matched pairs?
I looked in your config file and saw that you were verifying 10-meter
winds:
fcst_field[] = [ "33/Z10", "34/Z10" ];
Against mesonet observations:
message_type[] = [ "MSONET" ];
Here's the logic in Point-Stat that's causing the confusion:
- MET assumes 2-meter temperature and 10-meter winds to be at the
"surface".
- Point-Stat verifies these surface forecast against "surface
observations".
- Observations are defined to be at the surface only based on message
type. And only the "ADPSFC" and "SFCSHP" message types are assumed to
be at the surface.
- Since your observations are "MSONET", Point-Stat does not consider
them to be at the surface.
- So if you were to change all of the message types for these
observations from MSONET to ADPSFC, you'd get matches.
- Alternatively, you could modify the MET source code to tell it to
consider MSONET observations to be at the surface:
In the file "METv3.0/lib/vx_met_util/constants.h", change:
FROM: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP";
TO: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP
MSONET";
Recompile MET.
I tried the latter, and was able to get 2 matches. However, there
still seems to be problems here. I've attached an output file from
Point-Stat showing the MPR output.
You'll see that you have 2 matches for UGRD and 2 matched for VGRD.
Looking at the "OBS_ELV" column, you'll see these two observations
have values of 1026.5 and 1116.5 in that column. My questions are:
(1) Are these "OBS_ELV" values really supposed to the "OBS_LVL" - i.e.
pressure level values for the obs? The values seem more like pressure
values than elevations. I wonder if you have elevation
and pressure level values switched in your ASCII input to ASCII2NC.
(2) What do you really want to be doing here? Which of these
observations do you intend to be comparing to the 10-meter wind
forecast?
Ultimately, it's up to you to decide how to match up your observations
to your forecasts. Whatever you want to consider to be "surface"
observations, I'd suggest assigning the "ADPSFC" message type,
and they'll be matched to the 10-meter forecasts.
Clear as mud?
Thanks,
John
On 10/07/2010 12:42 PM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> Thanks for the help, John! I put some sample data on the server
after making the changes you suggested to the Point-Stat config file.
I'm going to be away from my desk until Monday. If you need anything
I haven't already provided I'll get it to you first thing Monday.
>
> Regards,
> Scott
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Thursday, October 07, 2010 12:06 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
> depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
>
> And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
> request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
> there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
> the record for the 25-minute forecast in that file.
>
> Since you're verifying every 5 minutes, in the Point-Stat config
file, I'd set "beg_ds" and "end_ds" very small, probably set them to
-150 and 150, respectively. That means, only use observations
> that are valid +/- 2.5 minutes around the forecast valid time.
>
> Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
> it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
> on only the MPR line type in the "output_flag" parameter. Then,
once you've run Point-Stat for all of the output times you'd like, use
the STAT-Analysis tool to compute a bunch of statistics. With
> STAT-Analysis, you can run jobs like this:
>
> # Compute continuous statistics (I'm assuming the station name is
REESE)
> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
>
> # Compute categorical statistics
> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
>
> The STAT-Analysis tool can be used to read in matched pair lines and
recompute any of the stats that Point-Stat can compute.
>
> Lastly, why are you still getting 0 matched pairs. I really don't
know.
>
> Go ahead and send me some sample data:
> - forecast GRIB file
> - ASCII and/or NetCDF point observation file
> - Point-Stat config file
> - command line you use to run Point-Stat
>
> I'll take a look and see if I can figure out what's going on.
>
> Just follow these directions for posting data to our anonymous ftp
site:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
>
> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>>
>> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>>
>> Regarding the obs, MET says it is "using 0 pairs" for both UGRD and
VGRD. For UGRD it processed 2224 obs which is correct based on the
number of obs in the NetCDF file I created from the Reese Tower data
on this particular day. It rejected 1668 because of the GRIB code --
I understand this because I have u, v, wspd and wdir for both 10 m and
100 m so that number makes sense to me. Beyond that, even though MET
reported only a single time from the GRIB file I would have thought it
could find the matching time in the NetCDF file because an ob does
exist at that exact time. Instead it says it rejected 510 obs based
on valid time and 46 based on level mismatch. I don't understand that
breakdown. And I get the exact same feedback for VGRD/Z010.
>>
>> Any guidance you can offer short of me sending over sample data?
>>
>> Thanks again,
>> Scott
>>
>>
>>
>> ________________________________
>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>> Sent: Wednesday, October 06, 2010 3:57 PM
>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>
>> Scott,
>>
>> Sure, we'd be happy to help.
>>
>> First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
>> used. That may provide you with a clue of the problem.
>>
>> Here's an example of what the output from Point-Stat will look
like:
>>
>> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
>> Number of matched pairs = 334
>> Observations processed = 91899
>> Rejected: GRIB code = 81017
>> Rejected: valid time = 0
>> Rejected: bad obs value = 0
>> Rejected: off the grid = 6
>> Rejected: level mismatch = 9925
>> Rejected: message type = 374
>> Rejected: masking region = 243
>> Rejected: bad fcst value = 0
>>
>> If you are still having problems, feel free to post data to our ftp
site by following these instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>>
>> Thanks,
>> John
>>
>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>>> Queue: met_help
>>> Subject: 5-Minute Winds in WRF
>>> Owner: Nobody
>>> Requestors: scott.r.dembek at nasa.gov
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>
>>>
>>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>>
>>> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>>>
>>> Thanks,
>>> Scott
>>>
>>
>
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: John Halley Gotway
Time: Tue Oct 12 10:48:42 2010
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL INDEX OBS_SID OBS_LAT OBS_LON
OBS_LVL OBS_ELV FCST OBS CLIMO
V3.0 WRF 113000 20100504_113000 20100504_113000 000000
20100504_113000 20100504_113000 UGRD Z10 UGRD Z10
MSONET FULL UW_MEAN 1 NA NA NA
NA MPR 2 1 Reese 33.61055 -102.05055 NA
1026.50000 3.72033 4.47320 NA
V3.0 WRF 113000 20100504_113000 20100504_113000 000000
20100504_113000 20100504_113000 UGRD Z10 UGRD Z10
MSONET FULL UW_MEAN 1 NA NA NA
NA MPR 2 2 Reese 33.61055 -102.05055 NA
1116.50000 3.72033 14.48440 NA
V3.0 WRF 113000 20100504_113000 20100504_113000 000000
20100504_113000 20100504_113000 VGRD Z10 VGRD Z10
MSONET FULL UW_MEAN 1 NA NA NA
NA MPR 2 1 Reese 33.61055 -102.05055 NA
1026.50000 1.24765 0.07540 NA
V3.0 WRF 113000 20100504_113000 20100504_113000 000000
20100504_113000 20100504_113000 VGRD Z10 VGRD Z10
MSONET FULL UW_MEAN 1 NA NA NA
NA MPR 2 2 Reese 33.61055 -102.05055 NA
1116.50000 1.24765 1.54300 NA
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: Dembek, Scott Robert[Universities Space Research Association]
Time: Tue Oct 12 11:18:49 2010
John,
Thanks for the information! I will follow your lead on the obs
message type, changing it from MSONET to ADPSFC. Regarding the
OBS_ELV column, that value is the station elevation plus the
observation height. The station elevation of the Reese Tower is
1016.5m so the 10m winds are set to 1026.5 and the 100m winds are set
to 1116.5. That's what I thought the MET code would be expecting. I
suppose I can continue to include the station elevation (is that
information critical to MET?), but the OBS_ELV should be changed to
10m and 100m, respectively. Is that right?
Regards,
Scott
________________________________
From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
Sent: Tuesday, October 12, 2010 12:48 PM
To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
Scott,
OK, I ran you're data through Point-Stat, and I see what's going on
here.
Let me say up front, I'd suggest changing the output_flag parameter:
FROM: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2
];
TO: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1
];
There's no real need to dump out both ".txt" and ".stat" versions of
the same data. I'd suggest just using the ".stat" data to save space.
Now, here's an excerpt from the Point-Stat output when I the Point-
Stat command you sent me:
--------------------------------------------------------------------------------
Processing UGRD/Z10 versus UGRD/Z10, for observation type MSONET, over
region FULL, for interpolation method UW_MEAN(1), using 0 pairs.
Number of matched pairs = 0
Observations processed = 2224
Rejected: GRIB code = 1668
Rejected: valid time = 554
Rejected: bad obs value = 0
Rejected: off the grid = 0
Rejected: level mismatch = 2
Rejected: message type = 0
Rejected: masking region = 0
Rejected: bad fcst value = 0
--------------------------------------------------------------------------------
So the question is, why are we getting those 2 level mismatches and 0
matched pairs?
I looked in your config file and saw that you were verifying 10-meter
winds:
fcst_field[] = [ "33/Z10", "34/Z10" ];
Against mesonet observations:
message_type[] = [ "MSONET" ];
Here's the logic in Point-Stat that's causing the confusion:
- MET assumes 2-meter temperature and 10-meter winds to be at the
"surface".
- Point-Stat verifies these surface forecast against "surface
observations".
- Observations are defined to be at the surface only based on message
type. And only the "ADPSFC" and "SFCSHP" message types are assumed to
be at the surface.
- Since your observations are "MSONET", Point-Stat does not consider
them to be at the surface.
- So if you were to change all of the message types for these
observations from MSONET to ADPSFC, you'd get matches.
- Alternatively, you could modify the MET source code to tell it to
consider MSONET observations to be at the surface:
In the file "METv3.0/lib/vx_met_util/constants.h", change:
FROM: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP";
TO: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP
MSONET";
Recompile MET.
I tried the latter, and was able to get 2 matches. However, there
still seems to be problems here. I've attached an output file from
Point-Stat showing the MPR output.
You'll see that you have 2 matches for UGRD and 2 matched for VGRD.
Looking at the "OBS_ELV" column, you'll see these two observations
have values of 1026.5 and 1116.5 in that column. My questions are:
(1) Are these "OBS_ELV" values really supposed to the "OBS_LVL" - i.e.
pressure level values for the obs? The values seem more like pressure
values than elevations. I wonder if you have elevation
and pressure level values switched in your ASCII input to ASCII2NC.
(2) What do you really want to be doing here? Which of these
observations do you intend to be comparing to the 10-meter wind
forecast?
Ultimately, it's up to you to decide how to match up your observations
to your forecasts. Whatever you want to consider to be "surface"
observations, I'd suggest assigning the "ADPSFC" message type,
and they'll be matched to the 10-meter forecasts.
Clear as mud?
Thanks,
John
On 10/07/2010 12:42 PM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> Thanks for the help, John! I put some sample data on the server
after making the changes you suggested to the Point-Stat config file.
I'm going to be away from my desk until Monday. If you need anything
I haven't already provided I'll get it to you first thing Monday.
>
> Regards,
> Scott
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Thursday, October 07, 2010 12:06 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
> depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
>
> And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
> request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
> there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
> the record for the 25-minute forecast in that file.
>
> Since you're verifying every 5 minutes, in the Point-Stat config
file, I'd set "beg_ds" and "end_ds" very small, probably set them to
-150 and 150, respectively. That means, only use observations
> that are valid +/- 2.5 minutes around the forecast valid time.
>
> Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
> it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
> on only the MPR line type in the "output_flag" parameter. Then,
once you've run Point-Stat for all of the output times you'd like, use
the STAT-Analysis tool to compute a bunch of statistics. With
> STAT-Analysis, you can run jobs like this:
>
> # Compute continuous statistics (I'm assuming the station name is
REESE)
> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
>
> # Compute categorical statistics
> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
>
> The STAT-Analysis tool can be used to read in matched pair lines and
recompute any of the stats that Point-Stat can compute.
>
> Lastly, why are you still getting 0 matched pairs. I really don't
know.
>
> Go ahead and send me some sample data:
> - forecast GRIB file
> - ASCII and/or NetCDF point observation file
> - Point-Stat config file
> - command line you use to run Point-Stat
>
> I'll take a look and see if I can figure out what's going on.
>
> Just follow these directions for posting data to our anonymous ftp
site:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
>
> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>>
>> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>>
>> Regarding the obs, MET says it is "using 0 pairs" for both UGRD and
VGRD. For UGRD it processed 2224 obs which is correct based on the
number of obs in the NetCDF file I created from the Reese Tower data
on this particular day. It rejected 1668 because of the GRIB code --
I understand this because I have u, v, wspd and wdir for both 10 m and
100 m so that number makes sense to me. Beyond that, even though MET
reported only a single time from the GRIB file I would have thought it
could find the matching time in the NetCDF file because an ob does
exist at that exact time. Instead it says it rejected 510 obs based
on valid time and 46 based on level mismatch. I don't understand that
breakdown. And I get the exact same feedback for VGRD/Z010.
>>
>> Any guidance you can offer short of me sending over sample data?
>>
>> Thanks again,
>> Scott
>>
>>
>>
>> ________________________________
>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>> Sent: Wednesday, October 06, 2010 3:57 PM
>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>
>> Scott,
>>
>> Sure, we'd be happy to help.
>>
>> First though, please try running Point-Stat with the "-v 3" option.
This is new in METv3.0 and for each verification task, it'll dump out
counts of reason codes for why observations were/were not
>> used. That may provide you with a clue of the problem.
>>
>> Here's an example of what the output from Point-Stat will look
like:
>>
>> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
>> Number of matched pairs = 334
>> Observations processed = 91899
>> Rejected: GRIB code = 81017
>> Rejected: valid time = 0
>> Rejected: bad obs value = 0
>> Rejected: off the grid = 6
>> Rejected: level mismatch = 9925
>> Rejected: message type = 374
>> Rejected: masking region = 243
>> Rejected: bad fcst value = 0
>>
>> If you are still having problems, feel free to post data to our ftp
site by following these instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>>
>> Thanks,
>> John
>>
>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>>> Queue: met_help
>>> Subject: 5-Minute Winds in WRF
>>> Owner: Nobody
>>> Requestors: scott.r.dembek at nasa.gov
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>
>>>
>>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>>
>>> For some reason I can't get Point-Stat to give me any results (the
output files contain headers only, no data), and I'm not sure if the
problem is with my input data or the way I've configured Point-Stat.
I wonder if I could pass along some sample data to see if you can help
diagnose my problem?
>>>
>>> Thanks,
>>> Scott
>>>
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
From: John Halley Gotway
Time: Tue Oct 12 11:33:22 2010
Scott,
Ah OK, I understand now. Since you have the elevation values
themselves, I'd actually suggest keeping the observation message type
as MSONET.
Then in your Point-Stat config file, you could request something like
this:
fcst_field[] = [ "33/Z10", "34/Z10" ];
obs_field[] = [ "33/Z1025-1027", "34/Z1025-1027" ];
This will compare the 10-meter forecast of wind to observations with
elevation values falling between 1025 and 1027m.
Alternatively, you could change the level values for your observations
from MSL to AGL (meters above sea level to meters above ground level).
So that would mean changing 1026.5m to 10m, and 1116.5m
to 100m, as you suggested. If you did that, then the following would
work:
fcst_field[] = [ "33/Z10", "34/Z10" ];
obs_field[] = [];
Either way works. It's up to you to decide how you'd like to store
you observation levels.
If I had to pick one, I suppose I'd suggest the latter. If you even
include more than just the REESE station, the additional stations will
likely have different elevations so that setting of the
observation height range wouldn't be a easy as it is with just one
station.
Hope that helps.
John
On 10/12/2010 11:18 AM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> John,
>
> Thanks for the information! I will follow your lead on the obs
message type, changing it from MSONET to ADPSFC. Regarding the
OBS_ELV column, that value is the station elevation plus the
observation height. The station elevation of the Reese Tower is
1016.5m so the 10m winds are set to 1026.5 and the 100m winds are set
to 1116.5. That's what I thought the MET code would be expecting. I
suppose I can continue to include the station elevation (is that
information critical to MET?), but the OBS_ELV should be changed to
10m and 100m, respectively. Is that right?
>
> Regards,
> Scott
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Tuesday, October 12, 2010 12:48 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> OK, I ran you're data through Point-Stat, and I see what's going on
here.
>
> Let me say up front, I'd suggest changing the output_flag parameter:
> FROM: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
2 ];
> TO: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1 ];
> There's no real need to dump out both ".txt" and ".stat" versions of
the same data. I'd suggest just using the ".stat" data to save space.
>
> Now, here's an excerpt from the Point-Stat output when I the Point-
Stat command you sent me:
>
--------------------------------------------------------------------------------
>
> Processing UGRD/Z10 versus UGRD/Z10, for observation type MSONET,
over region FULL, for interpolation method UW_MEAN(1), using 0 pairs.
> Number of matched pairs = 0
> Observations processed = 2224
> Rejected: GRIB code = 1668
> Rejected: valid time = 554
> Rejected: bad obs value = 0
> Rejected: off the grid = 0
> Rejected: level mismatch = 2
> Rejected: message type = 0
> Rejected: masking region = 0
> Rejected: bad fcst value = 0
>
>
--------------------------------------------------------------------------------
>
> So the question is, why are we getting those 2 level mismatches and
0 matched pairs?
>
> I looked in your config file and saw that you were verifying 10-
meter winds:
> fcst_field[] = [ "33/Z10", "34/Z10" ];
> Against mesonet observations:
> message_type[] = [ "MSONET" ];
>
> Here's the logic in Point-Stat that's causing the confusion:
> - MET assumes 2-meter temperature and 10-meter winds to be at the
"surface".
> - Point-Stat verifies these surface forecast against "surface
observations".
> - Observations are defined to be at the surface only based on
message type. And only the "ADPSFC" and "SFCSHP" message types are
assumed to be at the surface.
> - Since your observations are "MSONET", Point-Stat does not consider
them to be at the surface.
> - So if you were to change all of the message types for these
observations from MSONET to ADPSFC, you'd get matches.
> - Alternatively, you could modify the MET source code to tell it to
consider MSONET observations to be at the surface:
> In the file "METv3.0/lib/vx_met_util/constants.h", change:
> FROM: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP";
> TO: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP
MSONET";
> Recompile MET.
>
> I tried the latter, and was able to get 2 matches. However, there
still seems to be problems here. I've attached an output file from
Point-Stat showing the MPR output.
>
> You'll see that you have 2 matches for UGRD and 2 matched for VGRD.
Looking at the "OBS_ELV" column, you'll see these two observations
have values of 1026.5 and 1116.5 in that column. My questions are:
> (1) Are these "OBS_ELV" values really supposed to the "OBS_LVL" -
i.e. pressure level values for the obs? The values seem more like
pressure values than elevations. I wonder if you have elevation
> and pressure level values switched in your ASCII input to ASCII2NC.
> (2) What do you really want to be doing here? Which of these
observations do you intend to be comparing to the 10-meter wind
forecast?
>
> Ultimately, it's up to you to decide how to match up your
observations to your forecasts. Whatever you want to consider to be
"surface" observations, I'd suggest assigning the "ADPSFC" message
type,
> and they'll be matched to the 10-meter forecasts.
>
> Clear as mud?
>
> Thanks,
> John
>
> On 10/07/2010 12:42 PM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>> Thanks for the help, John! I put some sample data on the server
after making the changes you suggested to the Point-Stat config file.
I'm going to be away from my desk until Monday. If you need anything
I haven't already provided I'll get it to you first thing Monday.
>>
>> Regards,
>> Scott
>>
>>
>> ________________________________
>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>> Sent: Thursday, October 07, 2010 12:06 PM
>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>
>> Scott,
>>
>> The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
>> depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
>>
>> And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
>> request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
>> there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
>> the record for the 25-minute forecast in that file.
>>
>> Since you're verifying every 5 minutes, in the Point-Stat config
file, I'd set "beg_ds" and "end_ds" very small, probably set them to
-150 and 150, respectively. That means, only use observations
>> that are valid +/- 2.5 minutes around the forecast valid time.
>>
>> Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
>> it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
>> on only the MPR line type in the "output_flag" parameter. Then,
once you've run Point-Stat for all of the output times you'd like, use
the STAT-Analysis tool to compute a bunch of statistics. With
>> STAT-Analysis, you can run jobs like this:
>>
>> # Compute continuous statistics (I'm assuming the station name is
REESE)
>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
>>
>> # Compute categorical statistics
>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
>>
>> The STAT-Analysis tool can be used to read in matched pair lines
and recompute any of the stats that Point-Stat can compute.
>>
>> Lastly, why are you still getting 0 matched pairs. I really don't
know.
>>
>> Go ahead and send me some sample data:
>> - forecast GRIB file
>> - ASCII and/or NetCDF point observation file
>> - Point-Stat config file
>> - command line you use to run Point-Stat
>>
>> I'll take a look and see if I can figure out what's going on.
>>
>> Just follow these directions for posting data to our anonymous ftp
site:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>>
>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>
>>> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>>>
>>> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>>>
>>> Regarding the obs, MET says it is "using 0 pairs" for both UGRD
and VGRD. For UGRD it processed 2224 obs which is correct based on
the number of obs in the NetCDF file I created from the Reese Tower
data on this particular day. It rejected 1668 because of the GRIB
code -- I understand this because I have u, v, wspd and wdir for both
10 m and 100 m so that number makes sense to me. Beyond that, even
though MET reported only a single time from the GRIB file I would have
thought it could find the matching time in the NetCDF file because an
ob does exist at that exact time. Instead it says it rejected 510 obs
based on valid time and 46 based on level mismatch. I don't
understand that breakdown. And I get the exact same feedback for
VGRD/Z010.
>>>
>>> Any guidance you can offer short of me sending over sample data?
>>>
>>> Thanks again,
>>> Scott
>>>
>>>
>>>
>>> ________________________________
>>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>>> Sent: Wednesday, October 06, 2010 3:57 PM
>>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>>
>>> Scott,
>>>
>>> Sure, we'd be happy to help.
>>>
>>> First though, please try running Point-Stat with the "-v 3"
option. This is new in METv3.0 and for each verification task, it'll
dump out counts of reason codes for why observations were/were not
>>> used. That may provide you with a clue of the problem.
>>>
>>> Here's an example of what the output from Point-Stat will look
like:
>>>
>>> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
>>> Number of matched pairs = 334
>>> Observations processed = 91899
>>> Rejected: GRIB code = 81017
>>> Rejected: valid time = 0
>>> Rejected: bad obs value = 0
>>> Rejected: off the grid = 6
>>> Rejected: level mismatch = 9925
>>> Rejected: message type = 374
>>> Rejected: masking region = 243
>>> Rejected: bad fcst value = 0
>>>
>>> If you are still having problems, feel free to post data to our
ftp site by following these instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>>>
>>> Thanks,
>>> John
>>>
>>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>>>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>>>> Queue: met_help
>>>> Subject: 5-Minute Winds in WRF
>>>> Owner: Nobody
>>>> Requestors: scott.r.dembek at nasa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>>
>>>>
>>>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>>>
>>>> For some reason I can't get Point-Stat to give me any results
(the output files contain headers only, no data), and I'm not sure if
the problem is with my input data or the way I've configured Point-
Stat. I wonder if I could pass along some sample data to see if you
can help diagnose my problem?
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>
>>
>
------------------------------------------------
Subject: 5-Minute Winds in WRF
From: Dembek, Scott Robert[Universities Space Research Association]
Time: Wed Oct 13 09:52:12 2010
Thanks John. I will continue to use MSONET but change the obs level
to AGL with values of 10m and 100m. Regarding interp_method and
interp_width in the config file ... if I want to use nearest neighbor,
I understand I should set the interp_width to 1. Does that mean the
value of interp_method is ignored in this case? And should I expect
Point-Stat to find exactly one matching pair when using nearest
neighbor given that there is exactly one Reese Tower ob for u, v,
wdir, wspd at each 5-min time?
Regards,
Scott
________________________________
From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
Sent: Tuesday, October 12, 2010 1:33 PM
To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
Scott,
Ah OK, I understand now. Since you have the elevation values
themselves, I'd actually suggest keeping the observation message type
as MSONET.
Then in your Point-Stat config file, you could request something like
this:
fcst_field[] = [ "33/Z10", "34/Z10" ];
obs_field[] = [ "33/Z1025-1027", "34/Z1025-1027" ];
This will compare the 10-meter forecast of wind to observations with
elevation values falling between 1025 and 1027m.
Alternatively, you could change the level values for your observations
from MSL to AGL (meters above sea level to meters above ground level).
So that would mean changing 1026.5m to 10m, and 1116.5m
to 100m, as you suggested. If you did that, then the following would
work:
fcst_field[] = [ "33/Z10", "34/Z10" ];
obs_field[] = [];
Either way works. It's up to you to decide how you'd like to store
you observation levels.
If I had to pick one, I suppose I'd suggest the latter. If you even
include more than just the REESE station, the additional stations will
likely have different elevations so that setting of the
observation height range wouldn't be a easy as it is with just one
station.
Hope that helps.
John
On 10/12/2010 11:18 AM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> John,
>
> Thanks for the information! I will follow your lead on the obs
message type, changing it from MSONET to ADPSFC. Regarding the
OBS_ELV column, that value is the station elevation plus the
observation height. The station elevation of the Reese Tower is
1016.5m so the 10m winds are set to 1026.5 and the 100m winds are set
to 1116.5. That's what I thought the MET code would be expecting. I
suppose I can continue to include the station elevation (is that
information critical to MET?), but the OBS_ELV should be changed to
10m and 100m, respectively. Is that right?
>
> Regards,
> Scott
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Tuesday, October 12, 2010 12:48 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> OK, I ran you're data through Point-Stat, and I see what's going on
here.
>
> Let me say up front, I'd suggest changing the output_flag parameter:
> FROM: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
2 ];
> TO: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1 ];
> There's no real need to dump out both ".txt" and ".stat" versions of
the same data. I'd suggest just using the ".stat" data to save space.
>
> Now, here's an excerpt from the Point-Stat output when I the Point-
Stat command you sent me:
>
--------------------------------------------------------------------------------
>
> Processing UGRD/Z10 versus UGRD/Z10, for observation type MSONET,
over region FULL, for interpolation method UW_MEAN(1), using 0 pairs.
> Number of matched pairs = 0
> Observations processed = 2224
> Rejected: GRIB code = 1668
> Rejected: valid time = 554
> Rejected: bad obs value = 0
> Rejected: off the grid = 0
> Rejected: level mismatch = 2
> Rejected: message type = 0
> Rejected: masking region = 0
> Rejected: bad fcst value = 0
>
>
--------------------------------------------------------------------------------
>
> So the question is, why are we getting those 2 level mismatches and
0 matched pairs?
>
> I looked in your config file and saw that you were verifying 10-
meter winds:
> fcst_field[] = [ "33/Z10", "34/Z10" ];
> Against mesonet observations:
> message_type[] = [ "MSONET" ];
>
> Here's the logic in Point-Stat that's causing the confusion:
> - MET assumes 2-meter temperature and 10-meter winds to be at the
"surface".
> - Point-Stat verifies these surface forecast against "surface
observations".
> - Observations are defined to be at the surface only based on
message type. And only the "ADPSFC" and "SFCSHP" message types are
assumed to be at the surface.
> - Since your observations are "MSONET", Point-Stat does not consider
them to be at the surface.
> - So if you were to change all of the message types for these
observations from MSONET to ADPSFC, you'd get matches.
> - Alternatively, you could modify the MET source code to tell it to
consider MSONET observations to be at the surface:
> In the file "METv3.0/lib/vx_met_util/constants.h", change:
> FROM: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP";
> TO: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP
MSONET";
> Recompile MET.
>
> I tried the latter, and was able to get 2 matches. However, there
still seems to be problems here. I've attached an output file from
Point-Stat showing the MPR output.
>
> You'll see that you have 2 matches for UGRD and 2 matched for VGRD.
Looking at the "OBS_ELV" column, you'll see these two observations
have values of 1026.5 and 1116.5 in that column. My questions are:
> (1) Are these "OBS_ELV" values really supposed to the "OBS_LVL" -
i.e. pressure level values for the obs? The values seem more like
pressure values than elevations. I wonder if you have elevation
> and pressure level values switched in your ASCII input to ASCII2NC.
> (2) What do you really want to be doing here? Which of these
observations do you intend to be comparing to the 10-meter wind
forecast?
>
> Ultimately, it's up to you to decide how to match up your
observations to your forecasts. Whatever you want to consider to be
"surface" observations, I'd suggest assigning the "ADPSFC" message
type,
> and they'll be matched to the 10-meter forecasts.
>
> Clear as mud?
>
> Thanks,
> John
>
> On 10/07/2010 12:42 PM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>> Thanks for the help, John! I put some sample data on the server
after making the changes you suggested to the Point-Stat config file.
I'm going to be away from my desk until Monday. If you need anything
I haven't already provided I'll get it to you first thing Monday.
>>
>> Regards,
>> Scott
>>
>>
>> ________________________________
>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>> Sent: Thursday, October 07, 2010 12:06 PM
>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>
>> Scott,
>>
>> The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
>> depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
>>
>> And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
>> request "33/Z10" from that GRIB file, Point-Stat will use the FIRST
matching record it finds in that file. Since you have 12 output times
in that file, there are really 12 records of "33/Z10" in
>> there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
>> the record for the 25-minute forecast in that file.
>>
>> Since you're verifying every 5 minutes, in the Point-Stat config
file, I'd set "beg_ds" and "end_ds" very small, probably set them to
-150 and 150, respectively. That means, only use observations
>> that are valid +/- 2.5 minutes around the forecast valid time.
>>
>> Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
>> it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
>> on only the MPR line type in the "output_flag" parameter. Then,
once you've run Point-Stat for all of the output times you'd like, use
the STAT-Analysis tool to compute a bunch of statistics. With
>> STAT-Analysis, you can run jobs like this:
>>
>> # Compute continuous statistics (I'm assuming the station name is
REESE)
>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
>>
>> # Compute categorical statistics
>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
>>
>> The STAT-Analysis tool can be used to read in matched pair lines
and recompute any of the stats that Point-Stat can compute.
>>
>> Lastly, why are you still getting 0 matched pairs. I really don't
know.
>>
>> Go ahead and send me some sample data:
>> - forecast GRIB file
>> - ASCII and/or NetCDF point observation file
>> - Point-Stat config file
>> - command line you use to run Point-Stat
>>
>> I'll take a look and see if I can figure out what's going on.
>>
>> Just follow these directions for posting data to our anonymous ftp
site:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>>
>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>
>>> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>>>
>>> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>>>
>>> Regarding the obs, MET says it is "using 0 pairs" for both UGRD
and VGRD. For UGRD it processed 2224 obs which is correct based on
the number of obs in the NetCDF file I created from the Reese Tower
data on this particular day. It rejected 1668 because of the GRIB
code -- I understand this because I have u, v, wspd and wdir for both
10 m and 100 m so that number makes sense to me. Beyond that, even
though MET reported only a single time from the GRIB file I would have
thought it could find the matching time in the NetCDF file because an
ob does exist at that exact time. Instead it says it rejected 510 obs
based on valid time and 46 based on level mismatch. I don't
understand that breakdown. And I get the exact same feedback for
VGRD/Z010.
>>>
>>> Any guidance you can offer short of me sending over sample data?
>>>
>>> Thanks again,
>>> Scott
>>>
>>>
>>>
>>> ________________________________
>>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>>> Sent: Wednesday, October 06, 2010 3:57 PM
>>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>>
>>> Scott,
>>>
>>> Sure, we'd be happy to help.
>>>
>>> First though, please try running Point-Stat with the "-v 3"
option. This is new in METv3.0 and for each verification task, it'll
dump out counts of reason codes for why observations were/were not
>>> used. That may provide you with a clue of the problem.
>>>
>>> Here's an example of what the output from Point-Stat will look
like:
>>>
>>> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
>>> Number of matched pairs = 334
>>> Observations processed = 91899
>>> Rejected: GRIB code = 81017
>>> Rejected: valid time = 0
>>> Rejected: bad obs value = 0
>>> Rejected: off the grid = 6
>>> Rejected: level mismatch = 9925
>>> Rejected: message type = 374
>>> Rejected: masking region = 243
>>> Rejected: bad fcst value = 0
>>>
>>> If you are still having problems, feel free to post data to our
ftp site by following these instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Please let me know if you are able to resolve this yourself, or if
you've posted data to our ftp site.
>>>
>>> Thanks,
>>> John
>>>
>>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>>>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>>>> Queue: met_help
>>>> Subject: 5-Minute Winds in WRF
>>>> Owner: Nobody
>>>> Requestors: scott.r.dembek at nasa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>>
>>>>
>>>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>>>
>>>> For some reason I can't get Point-Stat to give me any results
(the output files contain headers only, no data), and I'm not sure if
the problem is with my input data or the way I've configured Point-
Stat. I wonder if I could pass along some sample data to see if you
can help diagnose my problem?
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
From: John Halley Gotway
Time: Wed Oct 13 10:02:07 2010
Scott,
When you set the interp_width = 1, then, yes, you are getting the
nearest neighbor. And the interp_method will be written in the output
as unweighted-mean (UW_MEAN). That's just a convention we
chose that we always write out UW_MEAN when interp_width = 1.
You may want to consider applying additional interpolation methods in
your Point-Stat runs. For example, you could also do bi-linear
interpolation. Then when you run STAT-Analysis, you could see if
the different interpolation methods give you different results. To be
honest, I doubt it'll have much of an impact, but it's easy to run.
For example, for these settings:
interp_method[] = [ "DW_MEAN" ];
interp_width[] = [ 1, 2, 3 ];
You would be performing 3 types of interpolation options:
(1) Nearest neighbor (it'll show up as 1 and UW_MEAN in your output)
(2) Bilinear interpolation (distance weighted mean of the 4 closest
points)
(3) Distance-weighted mean of the 9 closest points
Then you'd run STAT-Analysis jobs that look something like this:
# Continuous statistics for nearest neighbor
stat_analysis -job aggregate_stat -line_type MPR -out_line_type CNT
-interp_pnts 1
# Continuous statistics for bilinear interpolation
stat_analysis -job aggregate_stat -line_type MPR -out_line_type CNT
-interp_pnts 4
And yes, for each verification task, I'd expect you'd get just 1
matched pair that uses the 10-meter observation value at the Reese
Tower.
John
On 10/13/2010 09:52 AM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>
> Thanks John. I will continue to use MSONET but change the obs level
to AGL with values of 10m and 100m. Regarding interp_method and
interp_width in the config file ... if I want to use nearest neighbor,
I understand I should set the interp_width to 1. Does that mean the
value of interp_method is ignored in this case? And should I expect
Point-Stat to find exactly one matching pair when using nearest
neighbor given that there is exactly one Reese Tower ob for u, v,
wdir, wspd at each 5-min time?
>
> Regards,
> Scott
>
>
> ________________________________
> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
> Sent: Tuesday, October 12, 2010 1:33 PM
> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>
> Scott,
>
> Ah OK, I understand now. Since you have the elevation values
themselves, I'd actually suggest keeping the observation message type
as MSONET.
>
> Then in your Point-Stat config file, you could request something
like this:
> fcst_field[] = [ "33/Z10", "34/Z10" ];
> obs_field[] = [ "33/Z1025-1027", "34/Z1025-1027" ];
>
> This will compare the 10-meter forecast of wind to observations with
elevation values falling between 1025 and 1027m.
>
> Alternatively, you could change the level values for your
observations from MSL to AGL (meters above sea level to meters above
ground level). So that would mean changing 1026.5m to 10m, and
1116.5m
> to 100m, as you suggested. If you did that, then the following
would work:
> fcst_field[] = [ "33/Z10", "34/Z10" ];
> obs_field[] = [];
>
> Either way works. It's up to you to decide how you'd like to store
you observation levels.
>
> If I had to pick one, I suppose I'd suggest the latter. If you even
include more than just the REESE station, the additional stations will
likely have different elevations so that setting of the
> observation height range wouldn't be a easy as it is with just one
station.
>
> Hope that helps.
>
> John
>
> On 10/12/2010 11:18 AM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>
>> John,
>>
>> Thanks for the information! I will follow your lead on the obs
message type, changing it from MSONET to ADPSFC. Regarding the
OBS_ELV column, that value is the station elevation plus the
observation height. The station elevation of the Reese Tower is
1016.5m so the 10m winds are set to 1026.5 and the 100m winds are set
to 1116.5. That's what I thought the MET code would be expecting. I
suppose I can continue to include the station elevation (is that
information critical to MET?), but the OBS_ELV should be changed to
10m and 100m, respectively. Is that right?
>>
>> Regards,
>> Scott
>>
>>
>> ________________________________
>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>> Sent: Tuesday, October 12, 2010 12:48 PM
>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>
>> Scott,
>>
>> OK, I ran you're data through Point-Stat, and I see what's going on
here.
>>
>> Let me say up front, I'd suggest changing the output_flag
parameter:
>> FROM: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 2 ];
>> TO: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1 ];
>> There's no real need to dump out both ".txt" and ".stat" versions
of the same data. I'd suggest just using the ".stat" data to save
space.
>>
>> Now, here's an excerpt from the Point-Stat output when I the Point-
Stat command you sent me:
>>
--------------------------------------------------------------------------------
>>
>> Processing UGRD/Z10 versus UGRD/Z10, for observation type MSONET,
over region FULL, for interpolation method UW_MEAN(1), using 0 pairs.
>> Number of matched pairs = 0
>> Observations processed = 2224
>> Rejected: GRIB code = 1668
>> Rejected: valid time = 554
>> Rejected: bad obs value = 0
>> Rejected: off the grid = 0
>> Rejected: level mismatch = 2
>> Rejected: message type = 0
>> Rejected: masking region = 0
>> Rejected: bad fcst value = 0
>>
>>
--------------------------------------------------------------------------------
>>
>> So the question is, why are we getting those 2 level mismatches and
0 matched pairs?
>>
>> I looked in your config file and saw that you were verifying 10-
meter winds:
>> fcst_field[] = [ "33/Z10", "34/Z10" ];
>> Against mesonet observations:
>> message_type[] = [ "MSONET" ];
>>
>> Here's the logic in Point-Stat that's causing the confusion:
>> - MET assumes 2-meter temperature and 10-meter winds to be at the
"surface".
>> - Point-Stat verifies these surface forecast against "surface
observations".
>> - Observations are defined to be at the surface only based on
message type. And only the "ADPSFC" and "SFCSHP" message types are
assumed to be at the surface.
>> - Since your observations are "MSONET", Point-Stat does not
consider them to be at the surface.
>> - So if you were to change all of the message types for these
observations from MSONET to ADPSFC, you'd get matches.
>> - Alternatively, you could modify the MET source code to tell it to
consider MSONET observations to be at the surface:
>> In the file "METv3.0/lib/vx_met_util/constants.h", change:
>> FROM: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP";
>> TO: static const char onlysf_msg_typ_str[] = "ADPSFC SFCSHP
MSONET";
>> Recompile MET.
>>
>> I tried the latter, and was able to get 2 matches. However, there
still seems to be problems here. I've attached an output file from
Point-Stat showing the MPR output.
>>
>> You'll see that you have 2 matches for UGRD and 2 matched for VGRD.
Looking at the "OBS_ELV" column, you'll see these two observations
have values of 1026.5 and 1116.5 in that column. My questions are:
>> (1) Are these "OBS_ELV" values really supposed to the "OBS_LVL" -
i.e. pressure level values for the obs? The values seem more like
pressure values than elevations. I wonder if you have elevation
>> and pressure level values switched in your ASCII input to ASCII2NC.
>> (2) What do you really want to be doing here? Which of these
observations do you intend to be comparing to the 10-meter wind
forecast?
>>
>> Ultimately, it's up to you to decide how to match up your
observations to your forecasts. Whatever you want to consider to be
"surface" observations, I'd suggest assigning the "ADPSFC" message
type,
>> and they'll be matched to the 10-meter forecasts.
>>
>> Clear as mud?
>>
>> Thanks,
>> John
>>
>> On 10/07/2010 12:42 PM, RAL HelpDesk {for Dembek, Scott
Robert[Universities Space Research Association]} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>
>>> Thanks for the help, John! I put some sample data on the server
after making the changes you suggested to the Point-Stat config file.
I'm going to be away from my desk until Monday. If you need anything
I haven't already provided I'll get it to you first thing Monday.
>>>
>>> Regards,
>>> Scott
>>>
>>>
>>> ________________________________
>>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>>> Sent: Thursday, October 07, 2010 12:06 PM
>>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>>
>>> Scott,
>>>
>>> The MET statistics tools (Grid-Stat, Point-Stat, Wavelet-Stat, and
MODE) are designed to be run once per valid time. For many users,
that means running them once every 1, 3, 6, 12, or 24 hours -
>>> depending on the frequency of the verification you're doing. But
since you're trying to verify 5-minutes winds, it unfortunately means
that you need to run them once every 5 minutes!
>>>
>>> And it sounds like you have multiple output times in a single GRIB
forecast file. That's fine, Point-Stat in METv3.0 can handle that,
but you need to use an additional command line argument. If you
>>> request "33/Z10" from that GRIB file, Point-Stat will use the
FIRST matching record it finds in that file. Since you have 12 output
times in that file, there are really 12 records of "33/Z10" in
>>> there. You can tell Point-Stat which of those records to use by
setting the "-fcst_valid YYYYMMDD[_HH[MMSS]]" or "-fcst_lead time
HH[MMSS]" argument. For example, "-fcst_lead 002500" will only match
>>> the record for the 25-minute forecast in that file.
>>>
>>> Since you're verifying every 5 minutes, in the Point-Stat config
file, I'd set "beg_ds" and "end_ds" very small, probably set them to
-150 and 150, respectively. That means, only use observations
>>> that are valid +/- 2.5 minutes around the forecast valid time.
>>>
>>> Next, I assume that you're not really interested in the model
performance every 5 minutes. You probably really want to know how it
performs over time. You can do this type of analysis with MET, but
>>> it's a little cumbersome. I'd suggest running Point-Stat once for
each 5-minute output time. But for each run, only write out the
matched pair (MPR) line type. In the Point-Stat config file, turn
>>> on only the MPR line type in the "output_flag" parameter. Then,
once you've run Point-Stat for all of the output times you'd like, use
the STAT-Analysis tool to compute a bunch of statistics. With
>>> STAT-Analysis, you can run jobs like this:
>>>
>>> # Compute continuous statistics (I'm assuming the station name is
REESE)
>>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var UGRD -out_line_type CNT
-column_str SID REESE
>>>
>>> # Compute categorical statistics
>>> stat_analysis -lookin /directory/containing/stat/output/file -job
aggregate_stat -line_type MPR -fcst_var WIND -out_line_type CTS
-column_str SID REESE -out_fcst_thresh "ge5.0" -out_obs_thresh "ge5.0"
>>>
>>> The STAT-Analysis tool can be used to read in matched pair lines
and recompute any of the stats that Point-Stat can compute.
>>>
>>> Lastly, why are you still getting 0 matched pairs. I really don't
know.
>>>
>>> Go ahead and send me some sample data:
>>> - forecast GRIB file
>>> - ASCII and/or NetCDF point observation file
>>> - Point-Stat config file
>>> - command line you use to run Point-Stat
>>>
>>> I'll take a look and see if I can figure out what's going on.
>>>
>>> Just follow these directions for posting data to our anonymous ftp
site:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space Research
Association]} wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>>
>>>> Thanks John. I ran with "-v 3" to generate the feedback you
suggested. At this point I'm limiting the verification to U and V at
10 m to keep things as simple as possible. I coded "33/Z010" and
"34/Z010" in the Point-Stat config file. Is that correct?
>>>>
>>>> I'm handing MET a single GRIB file from forecast hour 12 which
contains u, v, wdir and wspd at 5-minute intervals during that hour at
both 10 m and 100 m, so there are 96 GRIB records of wind data (4
fields x 2 levels x 12 5-minute intervals in the hour = 96). MET only
reports one match, which happens to be the first 5-minute time period
in the file. How can I get MET to accept all 12 UGRD/Z010 and
VGRD/Z010 record pairs in the hourly GRIB file? Or will I have to
generate unique GRIB files for each 5-minute time period?
>>>>
>>>> Regarding the obs, MET says it is "using 0 pairs" for both UGRD
and VGRD. For UGRD it processed 2224 obs which is correct based on
the number of obs in the NetCDF file I created from the Reese Tower
data on this particular day. It rejected 1668 because of the GRIB
code -- I understand this because I have u, v, wspd and wdir for both
10 m and 100 m so that number makes sense to me. Beyond that, even
though MET reported only a single time from the GRIB file I would have
thought it could find the matching time in the NetCDF file because an
ob does exist at that exact time. Instead it says it rejected 510 obs
based on valid time and 46 based on level mismatch. I don't
understand that breakdown. And I get the exact same feedback for
VGRD/Z010.
>>>>
>>>> Any guidance you can offer short of me sending over sample data?
>>>>
>>>> Thanks again,
>>>> Scott
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: RAL HelpDesk {for John Halley Gotway} [met_help at ucar.edu]
>>>> Sent: Wednesday, October 06, 2010 3:57 PM
>>>> To: Dembek, Scott Robert (MSFC-VP61)[Universities Space Research
Association (USRA)]
>>>> Subject: Re: [rt.rap.ucar.edu #41328] 5-Minute Winds in WRF
>>>>
>>>> Scott,
>>>>
>>>> Sure, we'd be happy to help.
>>>>
>>>> First though, please try running Point-Stat with the "-v 3"
option. This is new in METv3.0 and for each verification task, it'll
dump out counts of reason codes for why observations were/were not
>>>> used. That may provide you with a clue of the problem.
>>>>
>>>> Here's an example of what the output from Point-Stat will look
like:
>>>>
>>>> Processing TMP/P900-750 versus TMP/P900-750, for observation type
ADPUPA, over region DTC166, for interpolation method MEDIAN(9), using
334 pairs.
>>>> Number of matched pairs = 334
>>>> Observations processed = 91899
>>>> Rejected: GRIB code = 81017
>>>> Rejected: valid time = 0
>>>> Rejected: bad obs value = 0
>>>> Rejected: off the grid = 6
>>>> Rejected: level mismatch = 9925
>>>> Rejected: message type = 374
>>>> Rejected: masking region = 243
>>>> Rejected: bad fcst value = 0
>>>>
>>>> If you are still having problems, feel free to post data to our
ftp site by following these instructions:
>>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>
>>>> Please let me know if you are able to resolve this yourself, or
if you've posted data to our ftp site.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> RAL HelpDesk {for Dembek, Scott Robert[Universities Space
Research Association]} wrote:
>>>>> Wed Oct 06 12:31:10 2010: Request 41328 was acted upon.
>>>>> Transaction: Ticket created by scott.r.dembek at nasa.gov
>>>>> Queue: met_help
>>>>> Subject: 5-Minute Winds in WRF
>>>>> Owner: Nobody
>>>>> Requestors: scott.r.dembek at nasa.gov
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41328 >
>>>>>
>>>>>
>>>>> As I mentioned in an earlier met_help thread, I've added code to
WRF to compute 5-min average u and v winds at both 10 m and 100 m, and
would like to use the Texas Tech Reese Tower obs to verify the winds.
Based on John's earlier comments I've created NetCDF files for the obs
each day, and I've modified WPP/WRFPOST to create GRIB files of the
WRF 5-min average u and v winds (along with speed and direction). In
the GRIB files I coded the valid time for the wind data in minutes
from the forecast start time so that each field is valid at 5 minutes,
10 minutes, 15 minutes, etc., throughout the length of the forecast.
This should allow MET to match the obs and the forecast perfectly in
time.
>>>>>
>>>>> For some reason I can't get Point-Stat to give me any results
(the output files contain headers only, no data), and I'm not sure if
the problem is with my input data or the way I've configured Point-
Stat. I wonder if I could pass along some sample data to see if you
can help diagnose my problem?
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>
>>>
>>
>
------------------------------------------------
More information about the Met_help
mailing list