[Met_help] [rt.rap.ucar.edu #63032] History for Possible MET bug
John Halley Gotway via RT
met_help at ucar.edu
Mon Oct 14 14:22:29 MDT 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello, I was running point-stat and comparing values with my colleague the other day, and we noticed an inconsistency n our data, seemingly using the exact same model and dataset for verification...the only difference being that he was running MET4.0 and I was running MET4.1. I dove a bit deeper into the problem, and wanted to inquire about my findings...
I have attached 4 documents. The first two are point_stat files, containing a list of 249 matched pairs, which were used for evaluation at forecast lead hour 1, and also CTS and CTC statistics for that time point. I quickly produced an stat file, containing both aggregate CTC and CTS statistics from the point_stat file. In one case, I used a threshold of all forecasts and observations > 0.10 mm/hr. In the second case, I used a threshold of >= 0.10 mm/hr. Both of these stat files are also attached. I noticed that the aggregate CTC numbers do not change depending on which threshold I am using, but the probability of detection (and other scores in the aggregate CTS output) have slightly changed. To me, this means that MET must have an error in computing either its CTS or CTC statistics. I am leaning towards the CTC statistics, since there are several data points that lie exactly on 0.10 mm/hr, which should cause the CTC statistics to be
different depending on whether > or >= is used for aggregation.
Furthermore...the aggregate CTS and CTC statistics in the case of the >0.10 threshold are NOT the same as the CTS and CTC scores in the point_stat file, although this was the only time point that I used, and so the scores should be exactly the same I would assume. I can provide more information/data if necessary, but this was the furthest that I could trace the problem. Has this error been documented or examined before? Am I overlooking something in my analysis?
________________________________
Von: John Halley Gotway via RT <met_help at ucar.edu>
An: andrewwx at yahoo.com
Gesendet: 18:32 Dienstag, 27.August 2013
Betreff: Re: [rt.rap.ucar.edu #62586] MET question
Andrew,
I suspect that we have some problems applying the lat/lon polylines to the global dataset. That's an open issue that we have yet to resolve. And I suspect that's the source of the different counts
that you're seeing.
Specifying an explicit list of station id's is a great solution. And METv4.1 actually does have that ability. This is mentioned in the METv4.1 release notes about halfway down in the section for
"Enhancements to Existing Tools":
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
Here's how you'd set that up in the Point-Stat config file:
mask = {
grid = [ "FULL" ];
poly = [];
sid = [ "sid_list1.txt", "sid_list2.txt", "sid_list3.txt" ];
};
Where "sid_listn.txt" is a file containing (1) a name for the region and (2) a list of station id's to be grouped together.
Here's a selection from the METv4.1/data/config/README file describing this:
// - The "sid" entry is an array of station ID groups. Each element is
// either the name of a single station ID or a station ID group file name.
// A station ID group file consists of a name for the group followed by a
// list of station ID's.
Please give that a shot and let me know if you run into any problems.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 08/27/2013 08:09 AM, Andrew J. via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=62586 >
>
> Hello, I have a few questions:
>
> First of all, I am using MET to verify a set of polyline masking regions for a variety of different models (ECMWF, WRF, GFS). Obviously, these models have different resolutions, but the region that I am evaluating is not near the boundaries of any of the models. The three models each find the same number of matched pairs in the FULL domains, but for each polygon regions, different matched pairs are found in GFS, ECMWF, and WRF. I do not know how this could be possible, or how I could go about fixing this issue so that the same matched pairs are evaluated for each model in each polygon....
>
> Secondly, as a potential solution to this problem, I though of perhaps specifying the station id's that lie within the polygons and compute matched pairs that way. However, MET currently only accepts one sid file, and not a comma separated list for evaluating different regions. I like the solution of using station ID's, but I'm not sure how I can specify separate lists of station ID's, which MET will sequentially evaluate. Would there be a way to accomplish this?
>
> Thank you in advance for your always helpful responses....
>
>
>
>
> ________________________________
> Von: John Halley Gotway via RT <met_help at ucar.edu>
> An: andrewwx at yahoo.com
> Gesendet: 15:42 Dienstag, 13.August 2013
> Betreff: Re: [rt.rap.ucar.edu #62586] MET question
>
>
> Andrew,
>
> I'll assume you're using METv4.1. Please let me know if that assumption is incorrect. For METv4.1, the GRIB1 tables are contained here:
> METv4.1/data/table_files/nceptab_flat.txt
>
> The 5 columns of information are as follows:
> (1) GRIB code number (integer)
> (2) parameter table version number (integer)
> (3) abbreviation (string)
> (4) long name (string)
> (5) units (string)
>
> You are correct, MET hasn't been well tested here using ECMWF data. Unfortunately, we don't have ready access to ECMWF data and therefore haven't exercised MET on it well. Looking at the example
> GRIB record info you sent, I see the following:
>
> (1) The GRIB code being used is 167 (kpds5=167).
> (2) I'm not sure of the parameter table version number. Try running "wgrib -PDS10" on that GRIB file. The 4th entry of the product description section will tell you the PTV.
> (3) Use "2T" for the name.
> (4) Specify a long name, perhaps "2-m Temperature".
> (5) Specify units, likely "K".
>
> Do the conventions in use match the information contained here?
> ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib/ectab_128
>
> Thanks,
> John
>
>
> On 08/12/2013 07:55 AM, Andrew J. via RT wrote:
>>
>> Mon Aug 12 07:55:31 2013: Request 62586 was acted upon.
>> Transaction: Ticket created by andrewwx at yahoo.com
>> Queue: met_help
>> Subject: MET question
>> Owner: Nobody
>> Requestors: andrewwx at yahoo.com
>> Status: new
>> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=62586 >
>>
>>
>> Hello hello,
>>
>> I am trying to use MET to evaluate some ECMWF data and I am running into a grib code problem. As you are probably already aware, ECMWF grib codes are different than those used by WRF, GFS, etc. Therefore, when a parameter is specified in the MET settings file, it does not correctly match up with the grib code names and levels of ECMWF. Lets take TMP/Z2 for example. In ECMWF, the raw grib code for 2m temperature looks like this:
>>
>>
>> 167:13478868:d=12070100:2T:kpds5=167:kpds6=1:kpds7=0:TR=0:P1=3:P2=0:TimeU=1:sfc:3hr fcst:type=9:NAve=0
>>
>> In this case, both the level and parameter name are different than the ones that MET is expecting, and so MET will return a 'no parameter found' result. I had previously been using "set_grib" to actually manually copy and reset the ECMWF grib parameters to match up with those that MET is expected. However, with many variables, this is becoming quite time consuming. I would rather alter the mapping system for both the parameters (TMP) and the level (Z2), so that in this case, MET will find TMP/Z2 based on the grib code given above. Where in the MET source code is this mapping done, and how could I go about changing it?
>>
>> Thank you in advance!
>>
>> - Andrew
>>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #63032] Possible MET bug
From: John Halley Gotway
Time: Tue Sep 17 10:31:24 2013
Andrew,
Hmmm, very good question. I ran a couple of STAT-Analysis jobs using
the STAT data you sent. You're getting identical contingency table
counts (CTC line) whether you threshold at >0.10 or >=0.10.
When I use those thresholds, I get different results. I suspect this
is a precision issue.
Listed below are two STAT-Analysis jobs and the corresponding CTC
results I get. Can you try running those same jobs and let me know
what you see?
-------------------------------------------------------
METv4.1/bin/stat_analysis -lookin
PS\>0.10_point_stat_PCP_010000L_20130602_130000V.stat -job
aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh
\>0.10 -out_obs_thresh \>0.10
COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
CTC: 249 125 47 18 59
METv4.1/bin/stat_analysis -lookin
PS\>0.10_point_stat_PCP_010000L_20130602_130000V.stat -job
aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh
\>=0.10 -out_obs_thresh \>=0.10
COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
CTC: 249 135 37 23 54
-------------------------------------------------------
As you can see, the counts change based on the "out_fcst_thresh" and
"out_obs_thresh" I choose, >0.10 and >=0.10. I suspect that you'll
get identical results when you run these jobs. I'm guessing
that the test for "equality" isn't evaluating to true for those
matched pair lines (MPR) with an observation value of 0.10.
Please try running those two jobs above to see if they yield identical
results.
If they do, the next step is to try modifying the tolerance used when
checking for the equality of two numbers. This is set in the file
"METv4.1/src/basic/vx_math/math_constants.h". Try editing that
file as follows:
static const double default_tol = loose_tol;
The current setting requires that two numbers need to be within 10E-10
of each other to be considered equal. I'm having you change that
tolerance to 10E-5.
Then do a "make clean" followed by a "make". And run those 2 jobs
again. Do you still get identical results or do they differ, like
they should?
Thanks,
John
On 09/13/2013 04:10 AM, Andrew J. via RT wrote:
>
> Fri Sep 13 04:10:00 2013: Request 63032 was acted upon.
> Transaction: Ticket created by andrewwx at yahoo.com
> Queue: met_help
> Subject: Possible MET bug
> Owner: Nobody
> Requestors: andrewwx at yahoo.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=63032 >
>
>
> Hello, I was running point-stat and comparing values with my
colleague the other day, and we noticed an inconsistency n our data,
seemingly using the exact same model and dataset for
verification...the only difference being that he was running MET4.0
and I was running MET4.1. I dove a bit deeper into the problem, and
wanted to inquire about my findings...
>
> I have attached 4 documents. The first two are point_stat files,
containing a list of 249 matched pairs, which were used for evaluation
at forecast lead hour 1, and also CTS and CTC statistics for that time
point. I quickly produced an stat file, containing both aggregate CTC
and CTS statistics from the point_stat file. In one case, I used a
threshold of all forecasts and observations > 0.10 mm/hr. In the
second case, I used a threshold of >= 0.10 mm/hr. Both of these
stat files are also attached. I noticed that the aggregate CTC
numbers do not change depending on which threshold I am using, but the
probability of detection (and other scores in the aggregate CTS
output) have slightly changed. To me, this means that MET must have
an error in computing either its CTS or CTC statistics. I am leaning
towards the CTC statistics, since there are several data points that
lie exactly on 0.10 mm/hr, which should cause the CTC statistics to be
> different depending on whether > or >= is used for aggregation.
>
>
> Furthermore...the aggregate CTS and CTC statistics in the case of
the >0.10 threshold are NOT the same as the CTS and CTC scores in the
point_stat file, although this was the only time point that I used,
and so the scores should be exactly the same I would assume. I can
provide more information/data if necessary, but this was the furthest
that I could trace the problem. Has this error been documented or
examined before? Am I overlooking something in my analysis?
>
>
> ________________________________
> Von: John Halley Gotway via RT <met_help at ucar.edu>
> An: andrewwx at yahoo.com
> Gesendet: 18:32 Dienstag, 27.August 2013
> Betreff: Re: [rt.rap.ucar.edu #62586] MET question
>
>
> Andrew,
>
> I suspect that we have some problems applying the lat/lon polylines
to the global dataset. That's an open issue that we have yet to
resolve. And I suspect that's the source of the different counts
> that you're seeing.
>
> Specifying an explicit list of station id's is a great solution.
And METv4.1 actually does have that ability. This is mentioned in the
METv4.1 release notes about halfway down in the section for
> "Enhancements to Existing Tools":
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
>
> Here's how you'd set that up in the Point-Stat config file:
>
> mask = {
> grid = [ "FULL" ];
> poly = [];
> sid = [ "sid_list1.txt", "sid_list2.txt", "sid_list3.txt" ];
> };
>
> Where "sid_listn.txt" is a file containing (1) a name for the region
and (2) a list of station id's to be grouped together.
>
> Here's a selection from the METv4.1/data/config/README file
describing this:
> // - The "sid" entry is an array of station ID groups. Each
element is
> // either the name of a single station ID or a station ID group
file name.
> // A station ID group file consists of a name for the group
followed by a
> // list of station ID's.
>
> Please give that a shot and let me know if you run into any
problems.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On 08/27/2013 08:09 AM, Andrew J. via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=62586 >
>>
>> Hello, I have a few questions:
>>
>> First of all, I am using MET to verify a set of polyline masking
regions for a variety of different models (ECMWF, WRF, GFS).
Obviously, these models have different resolutions, but the region
that I am evaluating is not near the boundaries of any of the models.
The three models each find the same number of matched pairs in the
FULL domains, but for each polygon regions, different matched pairs
are found in GFS, ECMWF, and WRF. I do not know how this could be
possible, or how I could go about fixing this issue so that the same
matched pairs are evaluated for each model in each polygon....
>>
>> Secondly, as a potential solution to this problem, I though of
perhaps specifying the station id's that lie within the polygons and
compute matched pairs that way. However, MET currently only accepts
one sid file, and not a comma separated list for evaluating different
regions. I like the solution of using station ID's, but I'm not sure
how I can specify separate lists of station ID's, which MET will
sequentially evaluate. Would there be a way to accomplish this?
>>
>> Thank you in advance for your always helpful responses....
>>
>>
>>
>>
>> ________________________________
>> Von: John Halley Gotway via RT <met_help at ucar.edu>
>> An: andrewwx at yahoo.com
>> Gesendet: 15:42 Dienstag, 13.August 2013
>> Betreff: Re: [rt.rap.ucar.edu #62586] MET question
>>
>>
>> Andrew,
>>
>> I'll assume you're using METv4.1. Please let me know if that
assumption is incorrect. For METv4.1, the GRIB1 tables are contained
here:
>> METv4.1/data/table_files/nceptab_flat.txt
>>
>> The 5 columns of information are as follows:
>> (1) GRIB code number (integer)
>> (2) parameter table version number (integer)
>> (3) abbreviation (string)
>> (4) long name (string)
>> (5) units (string)
>>
>> You are correct, MET hasn't been well tested here using ECMWF data.
Unfortunately, we don't have ready access to ECMWF data and therefore
haven't exercised MET on it well. Looking at the example
>> GRIB record info you sent, I see the following:
>>
>> (1) The GRIB code being used is 167 (kpds5=167).
>> (2) I'm not sure of the parameter table version number. Try
running "wgrib -PDS10" on that GRIB file. The 4th entry of the
product description section will tell you the PTV.
>> (3) Use "2T" for the name.
>> (4) Specify a long name, perhaps "2-m Temperature".
>> (5) Specify units, likely "K".
>>
>> Do the conventions in use match the information contained here?
>> ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib/ectab_128
>>
>> Thanks,
>> John
>>
>>
>> On 08/12/2013 07:55 AM, Andrew J. via RT wrote:
>>>
>>> Mon Aug 12 07:55:31 2013: Request 62586 was acted upon.
>>> Transaction: Ticket created by andrewwx at yahoo.com
>>> Queue: met_help
>>> Subject: MET question
>>> Owner: Nobody
>>> Requestors: andrewwx at yahoo.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=62586 >
>>>
>>>
>>> Hello hello,
>>>
>>> I am trying to use MET to evaluate some ECMWF data and I am
running into a grib code problem. As you are probably already aware,
ECMWF grib codes are different than those used by WRF, GFS, etc.
Therefore, when a parameter is specified in the MET settings file, it
does not correctly match up with the grib code names and levels of
ECMWF. Lets take TMP/Z2 for example. In ECMWF, the raw grib code for
2m temperature looks like this:
>>>
>>>
>>>
167:13478868:d=12070100:2T:kpds5=167:kpds6=1:kpds7=0:TR=0:P1=3:P2=0:TimeU=1:sfc:3hr
fcst:type=9:NAve=0
>>>
>>> In this case, both the level and parameter name are different than
the ones that MET is expecting, and so MET will return a 'no parameter
found' result. I had previously been using "set_grib" to actually
manually copy and reset the ECMWF grib parameters to match up with
those that MET is expected. However, with many variables, this is
becoming quite time consuming. I would rather alter the mapping
system for both the parameters (TMP) and the level (Z2), so that in
this case, MET will find TMP/Z2 based on the grib code given above.
Where in the MET source code is this mapping done, and how could I go
about changing it?
>>>
>>> Thank you in advance!
>>>
>>> - Andrew
>>>
------------------------------------------------
More information about the Met_help
mailing list