[Met_help] [rt.rap.ucar.edu #59665] History for MET v2.0 PointStat validation test
John Halley Gotway via RT
met_help at ucar.edu
Mon Feb 4 09:07:05 MST 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I am trying to validate a simple test case of MET PointStat against
in-house software. The verification statistics computed by each are
different. In an attempt to make the test case as simple as possible, I am
trying to verify surface data only since the MET User's Guide indicates no
vertical interpolation is done in this case. (I am only verifying
temperature for a single time frame.) However, I have good reason to
believe vertical interpolation is being done by PointStat anyway, and I
really need to be able to turn this feature off, or at least be able to
replicate the vertical interpolation exactly on my end. In light of this,
I have the following three questions that I could not find answers for
anywhere:
1. How exactly does MET define "surface" data? Is there a specific
pressure level or vertical height that I need to indicate in the PointStat
config file?
2. I inherited the verification project I'm currently working on from
someone who has since left my company. In their original PointStat config
file, "message_type" was set to "ONLYSF", which I assume means "Only
Surface". I could not find a definition for this message type anywhere.
How is it defined/interpreted by PointStat?
3. Can you provide more information about how PointStat does its vertical
interpolation? All I could find about it in the User's Guide was: "If any
discrepancy exists between the vertical levels, then the forecasts are
interpolated to the level of the observation. The vertical interpolation
is done in natural log of pressure coordinates.". What is the actual
interpolation method being used?
Thanks so much for any insight you have.
Kiana
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #59665] MET v2.0 PointStat validation test
From: John Halley Gotway
Time: Wed Dec 19 12:52:58 2012
Kiana,
I'll try to cover all your questions, but just let me know if I missed
anything...
Regarding "surface" data, MET determines what is/is not surface data
strictly by the message type. This behavior is inherited from the
NOAA-NCEP verification system, upon which MET was originally
based. Basically, the following message types means that observation
exists at the surface: ADPSFC (groud-based surface obs) and SFCSHP
(ship-based surface obs) (see
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm).
"ONLYSF" is a short-cut way of saying "ADPSFC" and "SFCSHP", as
mentioned in the METv4.0/config/README file:
//
// The "message_type" entry is an array of message types to be
retianed. An
// empty list indicates that all should be retained.
// List of valid message types:
// ADPUPA AIRCAR AIRCFT ADPSFC ERS1DA GOESND GPSIPW
// MSONET PROFLR QKSWND RASSDA SATEMP SATWND SFCBOG
// SFCSHP SPSSMI SYNDAT VADWND
// ANYAIR (= AIRCAR, AIRCFT)
// ANYSFC (= ADPSFC, SFCSHP, ADPUPA, PROFLR)
// ONLYSF (= ADPSFC, SFCSHP)
//
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
//
So if you're verifying a forecast field on a vertical level (e.g.
TMP/Z2, WIND/Z10 for 2-meter temperature and 10-meter wind-speed), and
you request that "ONLYSF" message types will be used, it'll
match any observation with a message type of "SFCSHP" or "ADFPSFC".
You may also want to try cranking up the verbosity level of point_stat
(-v 3 or -v 4) to see more information about what the tool
is doing. There should be no vertical interpolation done in this
case.
If you're not already doing so, I'd suggest writing the MPR output
line from Point-Stat which will list the explicit forecast and
observation values for each matched-pair.
Regarding the vertical interpolation, I can't describe it much more
than "linear in the natural log of pressure coordinates". Again, this
functionality is inherited from NCEP.
Suppose you're verifying temperature over a range of pressure levels:
TMP/P500-700. And you forecast file contains output at 500mb, 600mb,
and 700mb. Next suppose, you have an observation of
temperature at 550mb. Point-Stat will do the following:
(1) Identify that 550mb falls between the 500mb and 600mb levels.
(2) For the 500mb field, it'll interpolate to the lat,lon of the
observation using whatever interpolation method you've specified in
the config file.
(3) For the 600mb field, it'll do the same.
(4) Next, it'll interpolate the 500mb and 600mb values to the 550mb
level, doing that interpolation "linear in the natural log of
pressure".
This is done by the "compute_vert_pinterp" function in this source
file:
METv4.0/src/basic/vx_util/interp_util.cc
For vertical levels (as opposed to pressure levels), the
"compute_vert_zinterp" function is called instead.
And for point_stat, these functions are called from this file:
METv4.0/src/libcode/vx_statistics/pair_data_point.cc
Hopefully that helps.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/18/2012 03:29 PM, Kiana L Ross via RT wrote:
>
> Tue Dec 18 15:29:02 2012: Request 59665 was acted upon.
> Transaction: Ticket created by Kiana.L.Ross at aero.org
> Queue: met_help
> Subject: MET v2.0 PointStat validation test
> Owner: Nobody
> Requestors: Kiana.L.Ross at aero.org
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=59665 >
>
>
> Hello,
>
> I am trying to validate a simple test case of MET PointStat against
> in-house software. The verification statistics computed by each are
> different. In an attempt to make the test case as simple as
possible, I am
> trying to verify surface data only since the MET User's Guide
indicates no
> vertical interpolation is done in this case. (I am only verifying
> temperature for a single time frame.) However, I have good reason to
> believe vertical interpolation is being done by PointStat anyway,
and I
> really need to be able to turn this feature off, or at least be able
to
> replicate the vertical interpolation exactly on my end. In light of
this,
> I have the following three questions that I could not find answers
for
> anywhere:
>
> 1. How exactly does MET define "surface" data? Is there a specific
> pressure level or vertical height that I need to indicate in the
PointStat
> config file?
>
> 2. I inherited the verification project I'm currently working on
from
> someone who has since left my company. In their original PointStat
config
> file, "message_type" was set to "ONLYSF", which I assume means "Only
> Surface". I could not find a definition for this message type
anywhere.
> How is it defined/interpreted by PointStat?
>
> 3. Can you provide more information about how PointStat does its
vertical
> interpolation? All I could find about it in the User's Guide was:
"If any
> discrepancy exists between the vertical levels, then the forecasts
are
> interpolated to the level of the observation. The vertical
interpolation
> is done in natural log of pressure coordinates.". What is the actual
> interpolation method being used?
>
> Thanks so much for any insight you have.
>
> Kiana
>
------------------------------------------------
Subject: MET v2.0 PointStat validation test
From: Kiana L Ross
Time: Sun Jan 13 14:40:42 2013
John,
Thank you so much for your detailed response before the holidays. The
mpr
files have been very helpful. Unfortunately I do not have access to
MET
source code, (although maybe I'll just download it at home already!),
otherwise I could probably figure out a lot more on my own. I hope you
don't mind, but I have one more confusion regarding how
observation/model
pairs are matched:
I am running with observation data that I know has been tagged
"ADPSFC",
(so no vertical interpolation should be done by MET), and I have
chosen
"nearest neighbor" as my horizontal matching scheme in my config file.
Let's say I have a temperature observation that falls between 975 mb
and
1000 mb at a given (lat,lon), say right in the middle at 987.5 mb.
I've
configured the WRF post-processor to interpolate native model
temperature
values onto both 975 mb and 1000 mb pressure levels. My current
understanding is that MET will choose the gridded model (lat, lon)
that is
the "nearest neighbor" to my observation point, but how does it choose
between the model temperature value at 975 mb and the model
temperature
value at 1000 mb?
Anyway, thanks again for all the help. It's such a relief to get some
of
these questions answered, I really appreciate it.
Kiana
From: "John Halley Gotway via RT" <met_help at ucar.edu>
To: Kiana.L.Ross at aero.org,
Date: 12/19/2012 11:53 AM
Subject: Re: [rt.rap.ucar.edu #59665] MET v2.0 PointStat
validation
test
Kiana,
I'll try to cover all your questions, but just let me know if I missed
anything...
Regarding "surface" data, MET determines what is/is not surface data
strictly by the message type. This behavior is inherited from the
NOAA-NCEP verification system, upon which MET was originally
based. Basically, the following message types means that observation
exists at the surface: ADPSFC (groud-based surface obs) and SFCSHP
(ship-based surface obs) (see
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
).
"ONLYSF" is a short-cut way of saying "ADPSFC" and "SFCSHP", as
mentioned
in the METv4.0/config/README file:
//
// The "message_type" entry is an array of message types to be
retianed.
An
// empty list indicates that all should be retained.
// List of valid message types:
// ADPUPA AIRCAR AIRCFT ADPSFC ERS1DA GOESND GPSIPW
// MSONET PROFLR QKSWND RASSDA SATEMP SATWND SFCBOG
// SFCSHP SPSSMI SYNDAT VADWND
// ANYAIR (= AIRCAR, AIRCFT)
// ANYSFC (= ADPSFC, SFCSHP, ADPUPA, PROFLR)
// ONLYSF (= ADPSFC, SFCSHP)
//
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
//
So if you're verifying a forecast field on a vertical level (e.g.
TMP/Z2,
WIND/Z10 for 2-meter temperature and 10-meter wind-speed), and you
request
that "ONLYSF" message types will be used, it'll
match any observation with a message type of "SFCSHP" or "ADFPSFC".
You
may also want to try cranking up the verbosity level of point_stat (-v
3
or -v 4) to see more information about what the tool
is doing. There should be no vertical interpolation done in this
case.
If you're not already doing so, I'd suggest writing the MPR output
line
from Point-Stat which will list the explicit forecast and observation
values for each matched-pair.
Regarding the vertical interpolation, I can't describe it much more
than
"linear in the natural log of pressure coordinates". Again, this
functionality is inherited from NCEP.
Suppose you're verifying temperature over a range of pressure levels:
TMP/P500-700. And you forecast file contains output at 500mb, 600mb,
and
700mb. Next suppose, you have an observation of
temperature at 550mb. Point-Stat will do the following:
(1) Identify that 550mb falls between the 500mb and 600mb levels.
(2) For the 500mb field, it'll interpolate to the lat,lon of the
observation using whatever interpolation method you've specified in
the
config file.
(3) For the 600mb field, it'll do the same.
(4) Next, it'll interpolate the 500mb and 600mb values to the 550mb
level,
doing that interpolation "linear in the natural log of pressure".
This is done by the "compute_vert_pinterp" function in this source
file:
METv4.0/src/basic/vx_util/interp_util.cc
For vertical levels (as opposed to pressure levels), the
"compute_vert_zinterp" function is called instead.
And for point_stat, these functions are called from this file:
METv4.0/src/libcode/vx_statistics/pair_data_point.cc
Hopefully that helps.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/18/2012 03:29 PM, Kiana L Ross via RT wrote:
>
> Tue Dec 18 15:29:02 2012: Request 59665 was acted upon.
> Transaction: Ticket created by Kiana.L.Ross at aero.org
> Queue: met_help
> Subject: MET v2.0 PointStat validation test
> Owner: Nobody
> Requestors: Kiana.L.Ross at aero.org
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=59665 >
>
>
> Hello,
>
> I am trying to validate a simple test case of MET PointStat against
> in-house software. The verification statistics computed by each are
> different. In an attempt to make the test case as simple as
possible, I
am
> trying to verify surface data only since the MET User's Guide
indicates
no
> vertical interpolation is done in this case. (I am only verifying
> temperature for a single time frame.) However, I have good reason to
> believe vertical interpolation is being done by PointStat anyway,
and I
> really need to be able to turn this feature off, or at least be able
to
> replicate the vertical interpolation exactly on my end. In light of
this,
> I have the following three questions that I could not find answers
for
> anywhere:
>
> 1. How exactly does MET define "surface" data? Is there a specific
> pressure level or vertical height that I need to indicate in the
PointStat
> config file?
>
> 2. I inherited the verification project I'm currently working on
from
> someone who has since left my company. In their original PointStat
config
> file, "message_type" was set to "ONLYSF", which I assume means "Only
> Surface". I could not find a definition for this message type
anywhere.
> How is it defined/interpreted by PointStat?
>
> 3. Can you provide more information about how PointStat does its
vertical
> interpolation? All I could find about it in the User's Guide was:
"If
any
> discrepancy exists between the vertical levels, then the forecasts
are
> interpolated to the level of the observation. The vertical
interpolation
> is done in natural log of pressure coordinates.". What is the actual
> interpolation method being used?
>
> Thanks so much for any insight you have.
>
> Kiana
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #59665] MET v2.0 PointStat validation test
From: John Halley Gotway
Time: Tue Jan 15 09:05:52 2013
Kiana,
Hmm, you ask an interesting question. I don't think I fully
understood it before.
It sounds like you really want to prevent vertical interpolation from
occurring. And you used the ADPSFC message type thinking it would
prevent vertical interpolation from occurring. However, that
is not the case in general. It is true that no vertical interpolation
is performed when verifying surface forecast variables, such as 2m TMP
and 10m winds. And those are typically verified against
the ADPSFC message type. But you're verifying upper-air temperature
on pressure levels, for which vertical interpolation is in general
performed, regardless of the message type you choose.
However, there is good news. As long as you define the verification
tasks in an explicit way, then we can prevent vertical interpolation
from occurring.
First, choose the pressure level from your model output you'd like to
verify, let's say 1000mb for example.
Second, choose the range of observation pressure values you'd like to
match to that forecast level, let's say +/-25mb for example.
Lastly, specify those levels in the PointStatConfig file like this:
fcst_field[] = [ "TMP/P1000" ];
obs_field[] = [ "TMP/P1025-975" ];
That will verify temperature at a single level, matching any
observations whose pressure level falls within that range. I know
that no vertical interpolation will be performed since you're
verifying
a single level from the forecast file instead of a range of levels.
If you're interested in output from multiple levels, but with no
vertical interpolation, you can string them together like this:
fcst_field[] = [ "TMP/P1000", "TMP/P900", "TMP/P800" ];
obs_field[] = [ "TMP/P1025-975", "TMP/P925-P875", "TMP/P825-775" ];
And so on. Hope that helps clarify. Just let me know if you have
more questions.
Also, we've been continuing to address bugs and issues in the
development of the MET software. So upgrading the METv4.0 whenever
you find it convenient would be a good idea.
Thanks,
John
On 01/13/2013 02:40 PM, Kiana L Ross via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=59665 >
>
> John,
>
> Thank you so much for your detailed response before the holidays.
The mpr
> files have been very helpful. Unfortunately I do not have access to
MET
> source code, (although maybe I'll just download it at home
already!),
> otherwise I could probably figure out a lot more on my own. I hope
you
> don't mind, but I have one more confusion regarding how
observation/model
> pairs are matched:
>
> I am running with observation data that I know has been tagged
"ADPSFC",
> (so no vertical interpolation should be done by MET), and I have
chosen
> "nearest neighbor" as my horizontal matching scheme in my config
file.
> Let's say I have a temperature observation that falls between 975 mb
and
> 1000 mb at a given (lat,lon), say right in the middle at 987.5 mb.
I've
> configured the WRF post-processor to interpolate native model
temperature
> values onto both 975 mb and 1000 mb pressure levels. My current
> understanding is that MET will choose the gridded model (lat, lon)
that is
> the "nearest neighbor" to my observation point, but how does it
choose
> between the model temperature value at 975 mb and the model
temperature
> value at 1000 mb?
>
> Anyway, thanks again for all the help. It's such a relief to get
some of
> these questions answered, I really appreciate it.
>
> Kiana
>
>
>
> From: "John Halley Gotway via RT" <met_help at ucar.edu>
> To: Kiana.L.Ross at aero.org,
> Date: 12/19/2012 11:53 AM
> Subject: Re: [rt.rap.ucar.edu #59665] MET v2.0 PointStat
validation
> test
>
>
>
> Kiana,
>
> I'll try to cover all your questions, but just let me know if I
missed
> anything...
>
> Regarding "surface" data, MET determines what is/is not surface data
> strictly by the message type. This behavior is inherited from the
> NOAA-NCEP verification system, upon which MET was originally
> based. Basically, the following message types means that
observation
> exists at the surface: ADPSFC (groud-based surface obs) and SFCSHP
> (ship-based surface obs) (see
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
> ).
>
> "ONLYSF" is a short-cut way of saying "ADPSFC" and "SFCSHP", as
mentioned
> in the METv4.0/config/README file:
>
> //
> // The "message_type" entry is an array of message types to be
retianed.
> An
> // empty list indicates that all should be retained.
> // List of valid message types:
> // ADPUPA AIRCAR AIRCFT ADPSFC ERS1DA GOESND GPSIPW
> // MSONET PROFLR QKSWND RASSDA SATEMP SATWND SFCBOG
> // SFCSHP SPSSMI SYNDAT VADWND
> // ANYAIR (= AIRCAR, AIRCFT)
> // ANYSFC (= ADPSFC, SFCSHP, ADPUPA, PROFLR)
> // ONLYSF (= ADPSFC, SFCSHP)
> //
> //
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
> //
>
> So if you're verifying a forecast field on a vertical level (e.g.
TMP/Z2,
> WIND/Z10 for 2-meter temperature and 10-meter wind-speed), and you
request
> that "ONLYSF" message types will be used, it'll
> match any observation with a message type of "SFCSHP" or "ADFPSFC".
You
> may also want to try cranking up the verbosity level of point_stat
(-v 3
> or -v 4) to see more information about what the tool
> is doing. There should be no vertical interpolation done in this
case.
>
> If you're not already doing so, I'd suggest writing the MPR output
line
> from Point-Stat which will list the explicit forecast and
observation
> values for each matched-pair.
>
> Regarding the vertical interpolation, I can't describe it much more
than
> "linear in the natural log of pressure coordinates". Again, this
> functionality is inherited from NCEP.
>
> Suppose you're verifying temperature over a range of pressure
levels:
> TMP/P500-700. And you forecast file contains output at 500mb,
600mb, and
> 700mb. Next suppose, you have an observation of
> temperature at 550mb. Point-Stat will do the following:
> (1) Identify that 550mb falls between the 500mb and 600mb levels.
> (2) For the 500mb field, it'll interpolate to the lat,lon of the
> observation using whatever interpolation method you've specified in
the
> config file.
> (3) For the 600mb field, it'll do the same.
> (4) Next, it'll interpolate the 500mb and 600mb values to the 550mb
level,
> doing that interpolation "linear in the natural log of pressure".
>
> This is done by the "compute_vert_pinterp" function in this source
file:
> METv4.0/src/basic/vx_util/interp_util.cc
>
> For vertical levels (as opposed to pressure levels), the
> "compute_vert_zinterp" function is called instead.
>
> And for point_stat, these functions are called from this file:
> METv4.0/src/libcode/vx_statistics/pair_data_point.cc
>
> Hopefully that helps.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
>
> On 12/18/2012 03:29 PM, Kiana L Ross via RT wrote:
>>
>> Tue Dec 18 15:29:02 2012: Request 59665 was acted upon.
>> Transaction: Ticket created by Kiana.L.Ross at aero.org
>> Queue: met_help
>> Subject: MET v2.0 PointStat validation test
>> Owner: Nobody
>> Requestors: Kiana.L.Ross at aero.org
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=59665 >
>>
>>
>> Hello,
>>
>> I am trying to validate a simple test case of MET PointStat against
>> in-house software. The verification statistics computed by each are
>> different. In an attempt to make the test case as simple as
possible, I
> am
>> trying to verify surface data only since the MET User's Guide
indicates
> no
>> vertical interpolation is done in this case. (I am only verifying
>> temperature for a single time frame.) However, I have good reason
to
>> believe vertical interpolation is being done by PointStat anyway,
and I
>> really need to be able to turn this feature off, or at least be
able to
>> replicate the vertical interpolation exactly on my end. In light of
> this,
>> I have the following three questions that I could not find answers
for
>> anywhere:
>>
>> 1. How exactly does MET define "surface" data? Is there a specific
>> pressure level or vertical height that I need to indicate in the
> PointStat
>> config file?
>>
>> 2. I inherited the verification project I'm currently working on
from
>> someone who has since left my company. In their original PointStat
> config
>> file, "message_type" was set to "ONLYSF", which I assume means
"Only
>> Surface". I could not find a definition for this message type
anywhere.
>> How is it defined/interpreted by PointStat?
>>
>> 3. Can you provide more information about how PointStat does its
> vertical
>> interpolation? All I could find about it in the User's Guide was:
"If
> any
>> discrepancy exists between the vertical levels, then the forecasts
are
>> interpolated to the level of the observation. The vertical
interpolation
>> is done in natural log of pressure coordinates.". What is the
actual
>> interpolation method being used?
>>
>> Thanks so much for any insight you have.
>>
>> Kiana
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list