[Met_help] [rt.rap.ucar.edu #65756] History for Point_Stat error
John Halley Gotway via RT
met_help at ucar.edu
Mon Jun 2 14:17:20 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I am getting an error running Point-Stat with a little_r observations file. Using MET V4.1, the log file shows the following dialog:
ERROR :
ERROR : do_vl1l2() -> the number of UGRD pairs != the number of VGRD pairs: 159 != 158
ERROR
I've attached the log file, but we feel that the information provided below in the flags for duplicate data may shed some light on the problem. We are curious if Point-Stat logging could provide a way to quickly identify the station whose data is causing the error. We performed searches in the llittle_r file to locate the observation which is causing the problem, but it is not easy or always successful. We have identified some things to consider which might alleviate the problem in the dialog below.
Do you have any suggestions on how to track down the offending observation or perhaps change the way the duplicate_flag works to avoid this problem?
Thanks.
________________________________
DEBUG 3: Duplicate point observations for -unique setting:
DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426, 00010426
DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495, 00010495
DEBUG 3:
DEBUG 3:
DEBUG 3: Duplicate point observations for -unique setting:
DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426, 00010426
DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495, 00010495
DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424, 00010424
I think that the first lines are the U observations that it considered duplicate for observation AIRCAR for P225-100, and the next few lines are the same for the V observations. So the third duplicate in the V does not have a matching duplicate in the U. Looking through the little_r files, I don't find two observations that exactly match that listing. However, I did find two that were pretty close, and perhaps close enough to be considered duplicate by MET? So the two observations are:
36.90000 -119.5900000010424 MADIS FM-96 -888888.00000 1 0 0 0 0 T F F -888888 -888888 20120207230900-888888.00000-888888-888888.00000-888888-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000-888888-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0
18734.89063 512 12192.00000 0 220.00000 0-888888.00000-888888 11.80000 0 274.00000 0 11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-888888
36.89000 -119.5800000010424 MADIS FM-96 -888888.00000 1 0 0 0 0 T F F -888888 -888888 20120207230900-888888.00000-888888-888888.00000-888888-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000-888888-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0-888888.00000 0
18734.89063 512 12192.00000 0 219.70000 0-888888.00000-888888 11.80000 0 274.00000 0 11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-888888
The latitudes and longitudes of the two observations are not identical but are 0.01 off of each other. Perhaps this is considered close enough by MET. For the first observation u,v is (11.77126,-0.82313) and for the second observation u,v is (11.77125,-0.82313). So the V's match but the U's differ by 0.00001 m/s, presumably due to rounding issues/machine precision. So my guess is that these observations are the source of the problem. Assuming that this is correct, the next question is how to fix this. A few possibilities come to mind:
1) Change MET to insist that for observations to be duplicate that the lat/lon match closer than 0.01 degrees (which is about 1km).
2) Change the duplicate checking to allow for machine precision type differences between observations.
3) Change the duplicate checking to check all observation components together, or at least U/V together. In other words, for an ob to be marked duplicate require that all observed fields match or at least require that U/V match.
I'm thinking we might want to send these two obs that MET appears to mark as duplicate to MET help to see what they think. For a quick fix you could go and edit one of the two observations so that the U's match, since a change of 0.00001 m/s is not meaningful.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
________________________________
From: Raby, John W CIV USARMY ARL (US)
Sent: Wednesday, March 05, 2014 2:31 PM
To: Reen, Brian P CIV USARMY ARL (US)
Subject: Point_Stat log file
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Fri Mar 07 11:57:03 2014
Hello John,
Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
238 // build a uniqueness test key
239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
240 lat, // lat
241 lon, // lon
242 lvl, // level
243 elv).text(); // elevation
As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude. So a difference of 0.01 would not be
"close enough".
If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
In the point_stat config file, set:
mask = {
grid = [];
poly = [];
sid = [ "station_list.txt" ];
};
Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424
That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.
I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Point_Stat error
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR :
> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together. In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think. For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Fri Mar 07 13:00:06 2014
John -
Great idea. Thanks for looking into this. I had not considered that
functionality of Point-Stat to narrow down the search through the use
of station identifiers. These little_r files are very large and
difficult to sort through and this should be a real help. I will give
it a try per your suggested technique below. If it works, then I might
be able to use the information to select the station(s) whose data
need to be deleted and fix the problem that way, too. Will be in touch
soon.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 07, 2014 11:57 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
Hello John,
Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
238 // build a uniqueness test key
239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
240 lat, // lat
241 lon, // lon
242 lvl, // level
243 elv).text(); // elevation
As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude. So a difference of 0.01 would not be
"close enough".
If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
In the point_stat config file, set:
mask = {
grid = [];
poly = [];
sid = [ "station_list.txt" ];
};
Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424
That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.
I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Point_Stat error
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR :
> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together. In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think. For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>
------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 08:47:24 2014
John -
I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no statistical results. I'm beginning to think
that for some reason it's not reading or performing the analysis with
the station ids. The log file acknowledges the presence of and
handling of the list as follows:
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
Let me know if I can provide you the data. The forecast file is 15mb,
and the little_r file is 6mb.
Thanks.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 07, 2014 11:57 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
Hello John,
Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
238 // build a uniqueness test key
239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
240 lat, // lat
241 lon, // lon
242 lvl, // level
243 elv).text(); // elevation
As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude. So a difference of 0.01 would not be
"close enough".
If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
In the point_stat config file, set:
mask = {
grid = [];
poly = [];
sid = [ "station_list.txt" ];
};
Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424
That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.
I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Point_Stat error
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR :
> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together. In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think. For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 09:43:22 2014
John,
Sure, go ahead and send the data:
- forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
You can post it to our anonymous ftp site following these
instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stati!
st!
> ical results. I'm beginning to think that for some reason it's not
reading or performing the analysis with the station ids. The log file
acknowledges the presence of and handling of the list as follows:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>
> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, March 07, 2014 11:57 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> Hello John,
>
> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>
> 238 // build a uniqueness test key
> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
> 240 lat, // lat
> 241 lon, // lon
> 242 lvl, // level
> 243 elv).text(); // elevation
>
> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>
> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>
> In the point_stat config file, set:
> mask = {
> grid = [];
> poly = [];
> sid = [ "station_list.txt" ];
> };
>
> Where station_list.txt is an ASCII file containing something like:
> TEST_STATIONS
> 00010426
> 00010495
> 00010424
>
> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
> narrow it down.
>
> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>
>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Point_Stat error
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>>
>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>
>> ERROR :
>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>> ERROR
>>
>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>
>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>
>> Thanks.
>> ________________________________
>>
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>> DEBUG 3:
>> DEBUG 3:
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>
>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>
>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>
>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>
>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>
>> R/
>> John
>>
>>
>> Mr John W. Raby, Meteorologist
>>
>> U.S. Army Research Laboratory
>>
>> White Sands Missile Range, NM 88002
>>
>> (575) 678-2004 DSN 258-2004
>>
>> FAX (575) 678-1230 DSN 258-1230
>>
>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>
>>
>>
>> ________________________________
>> From: Raby, John W CIV USARMY ARL (US)
>> Sent: Wednesday, March 05, 2014 2:31 PM
>> To: Reen, Brian P CIV USARMY ARL (US)
>> Subject: Point_Stat log file
>>
>>
>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 09:59:49 2014
John -
OK. Thanks.
I've had trouble lately with the WSMR firewall and may not be able to
send you the data via anonymous ftp, so I will give it a try and let
you know how it goes. Alternatively I can send the little_r file by
email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
You should have the log file and my configuration file already from my
previous email. Let me know if you didn't get it.
The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
I've attached the station_list.txt file to this email.
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 9:43 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
Sure, go ahead and send the data:
- forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
You can post it to our anonymous ftp site following these
instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stati!
st!
> ical results. I'm beginning to think that for some reason it's not
reading or performing the analysis with the station ids. The log file
acknowledges the presence of and handling of the list as follows:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>
> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, March 07, 2014 11:57 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> Hello John,
>
> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>
> 238 // build a uniqueness test key
> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
> 240 lat, // lat
> 241 lon, // lon
> 242 lvl, // level
> 243 elv).text(); // elevation
>
> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>
> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>
> In the point_stat config file, set:
> mask = {
> grid = [];
> poly = [];
> sid = [ "station_list.txt" ];
> };
>
> Where station_list.txt is an ASCII file containing something like:
> TEST_STATIONS
> 00010426
> 00010495
> 00010424
>
> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
> narrow it down.
>
> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>
>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Point_Stat error
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>>
>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>
>> ERROR :
>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>> ERROR
>>
>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>
>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>
>> Thanks.
>> ________________________________
>>
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>> DEBUG 3:
>> DEBUG 3:
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>
>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>
>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>
>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>
>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>
>> R/
>> John
>>
>>
>> Mr John W. Raby, Meteorologist
>>
>> U.S. Army Research Laboratory
>>
>> White Sands Missile Range, NM 88002
>>
>> (575) 678-2004 DSN 258-2004
>>
>> FAX (575) 678-1230 DSN 258-1230
>>
>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>
>>
>>
>> ________________________________
>> From: Raby, John W CIV USARMY ARL (US)
>> Sent: Wednesday, March 05, 2014 2:31 PM
>> To: Reen, Brian P CIV USARMY ARL (US)
>> Subject: Point_Stat log file
>>
>>
>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 09:59:49 2014
TEST_STATIONS
C6937
KELC1
QLGQ3
SRTC1
KFLG
TS776
QRCA3
KUKI
KNFL
KAJO
KINS
KNXP
KLAS
KALK
D8496
D2849
D8204
AT450
AU043
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 10:45:41 2014
John,
OK thanks. I now have the config file and station list. I just need
the forecast file and little_r file.
Thanks,
John
On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
i!
> st!
>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>> 238 // build a uniqueness test key
>> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>> 240 lat, // lat
>> 241 lon, // lon
>> 242 lvl, // level
>> 243 elv).text(); // elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>> grid = [];
>> poly = [];
>> sid = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Point_Stat error
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR :
>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 11:00:24 2014
John -
I attempted to ftp the files, but I'm blocked by WSMR from doing so
(connection refused). I confirmed that the block is a local one by
checking with our sysadm person.
Do you happen to have a sftp site you can point me to to accomplish
the transfer?
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
OK thanks. I now have the config file and station list. I just need
the forecast file and little_r file.
Thanks,
John
On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
i!
> st!
>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>> 238 // build a uniqueness test key
>> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>> 240 lat, // lat
>> 241 lon, // lon
>> 242 lvl, // level
>> 243 elv).text(); // elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>> grid = [];
>> poly = [];
>> sid = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Point_Stat error
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR :
>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 12:10:51 2014
John -
I just sent you the files using a file exchange system. Please let me
know if this works for you.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
OK thanks. I now have the config file and station list. I just need
the forecast file and little_r file.
Thanks,
John
On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
i!
> st!
>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>> 238 // build a uniqueness test key
>> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>> 240 lat, // lat
>> 241 lon, // lon
>> 242 lvl, // level
>> 243 elv).text(); // elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>> grid = [];
>> poly = [];
>> sid = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Point_Stat error
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR :
>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 12:17:34 2014
John -
Just in case, please delete the email you receive from the file
exchange system after you download the files.
Thanks.
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
OK thanks. I now have the config file and station list. I just need
the forecast file and little_r file.
Thanks,
John
On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
i!
> st!
>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>> 238 // build a uniqueness test key
>> 239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>> 240 lat, // lat
>> 241 lon, // lon
>> 242 lvl, // level
>> 243 elv).text(); // elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>> grid = [];
>> poly = [];
>> sid = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>> Queue: met_help
>>> Subject: Point_Stat error
>>> Owner: Nobody
>>> Requestors: john.w.raby2.civ at mail.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR :
>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 13:24:53 2014
John,
OK, I received the files and ran them through Point-Stat. And I was
able to get matched pair output. Upon closer inspection of my log
file versus yours, I see the problem. When I run Point-Stat, I
see the following line included in the log file:
DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
But in the log file you sent me, I see this instead:
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to include the full path for "station_list.txt" in the
Point-Stat config file. I believe that will solve your problem.
The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently. You listed the names
of the stations directly in the config file itself. Each station
name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair. After we added the
matched pair (MPR) output line type, this type of station id
masking was no longer needed. So we modified the logic. In METv4.1,
you list one or more strings in the "mask.sid" setting. Point-Stat
checks each string to see if it's a file name. If so, it
parses the group of station names from the file listed. If not, it
treats the string as the name of a single station to be used. That
enables us to maintain backward compatibility.
In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used. But you don't have any observations
for a station named "station_list.txt". Thus, it found 0 matched
pairs.
Point-Stat did tell us all the information in the log file, but it was
a pretty subtle point.
Please give that a shot and let me know how it goes.
Thanks,
John
On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks. I now have the config file and station list. I just
need the forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>
>> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>>
>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>
>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no sta!
t!
> i!
>> st!
>>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>> 238 // build a uniqueness test key
>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>> 240 lat, // lat
>>> 241 lon, // lon
>>> 242 lvl, // level
>>> 243 elv).text(); // elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>> grid = [];
>>> poly = [];
>>> sid = [ "station_list.txt" ];
>>> };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's. You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>> narrow it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>> Queue: met_help
>>>> Subject: Point_Stat error
>>>> Owner: Nobody
>>>> Requestors: john.w.raby2.civ at mail.mil
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>
>>>> ERROR :
>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>> ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:30:51 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for taking a look at this. I will now switch over to test the
solution
you recommend and report back.
Can you delete the email which delivered the info to download the
files?
Thanks.
R/
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:25 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
OK, I received the files and ran them through Point-Stat. And I was
able to
get matched pair output. Upon closer inspection of my log file versus
yours,
I see the problem. When I run Point-Stat, I see the following line
included
in the log file:
DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS"
mask from file "station_list.txt"
But in the log file you sent me, I see this instead:
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to
include the full path for "station_list.txt" in the Point-Stat config
file. I
believe that will solve your problem.
The long explanation is this... in earlier versions of MET, the
station id
setting functioned a bit differently. You listed the names of the
stations
directly in the config file itself. Each station name was treated as
it's own
masking region - so we were basically computing statistics over a
single
matched pair. After we added the matched pair (MPR) output line type,
this
type of station id masking was no longer needed. So we modified the
logic.
In METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name. If so, it
parses
the group of station names from the file listed. If not, it treats
the string
as the name of a single station to be used. That enables us to
maintain
backward compatibility.
In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the
station to be used. But you don't have any observations for a station
named
"station_list.txt". Thus, it found 0 matched pairs.
Point-Stat did tell us all the information in the log file, but it was
a
pretty subtle point.
Please give that a shot and let me know how it goes.
Thanks,
John
On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange
> system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks. I now have the config file and station list. I just
need the
> forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send
>> you the data via anonymous ftp, so I will give it a try and let you
know
>> how it goes. Alternatively I can send the little_r file by email
since it's
>> only 5.5 mb, but the forecast file at 15 mb may be too big for
email.
>>
>> You should have the log file and my configuration file already from
my
>> previous email. Let me know if you didn't get it.
>>
>> The obs filename is:
>> qc_obs_used_earth_relative.d01.2012-02-07_23:00:00.0000
>>
>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>> - forecast file, little_r file, Point-Stat config file, and the
>> "station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list,
>>> putting it in the same directory along with the config file and
then
>>> running Point-Stat. I changed the config file (attached) to match
your
>>> setting so it can read the station_list.txt file instead of a poly
file or
>>> the grid (FULL). I used the same 3 station ids we identified as
suspect
>>> per the email below. The log file showed that Point-Stat ran
without
>>> error, but the output files contain no numerical results, only
column
>>> headers. I experimented with adding to and subtracting from the id
list
>>> with known stations based on looking at the NetCDF obs file for
the same
>>> hour, but never produced output with actual statistics, just
headers. I
>>> removed those stations from the list and entered a new set of
surface obs
>>> (ADPSFC) which should have produced matched pairs at the Z2/Z10
levels,
>>> but no change. The log file (attached) shows no matched pairs in
every
>>> case. The output (CNT file) shows the headers, but no sta!
t!
> i!
>> st!
>>> ical results. I'm beginning to think that for some reason it's
not
>>> reading or performing the analysis with the station ids. The log
file
>>> acknowledges the presence of and handling of the list as follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
>>> "station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning
>>> how to use this functionality correctly in order to use it to
potentially
>>> solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and
>>> the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file
>>> METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>> 238 // build a uniqueness test key
>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>> 240 lat, // lat
>>> 241 lon, // lon
>>> 242 lvl, // level
>>> 243 elv).text(); // elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the
>>> latitude and longitude. So a difference of 0.01 would not be
"close
>>> enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
>>> figure out the smallest subset of littl_r data that reproduces
this
>>> problem. If there are just a couple of station id's that are
suspect, I'd
>>> suggest using the station id masking to narrow down exactly which
>>> station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>> grid = [];
>>> poly = [];
>>> sid = [ "station_list.txt" ]; };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's.
>>> You could check to see if just those 3 stations replicate the
error and
>>> then either add/subtract stations from that list to really narrow
it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to
>>> send me a sample little_r file, forecast file, and Point-Stat
>>> configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>> Queue: met_help
>>>> Subject: Point_Stat error
>>>> Owner: Nobody
>>>> Requestors: john.w.raby2.civ at mail.mil
>>>> Status: new
>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations
>>>> file. Using MET V4.1, the log file shows the following dialog:
>>>>
>>>> ERROR :
>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
>>>> VGRD pairs: 159 != 158 ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided
>>>> below in the flags for duplicate data may shed some light on the
problem.
>>>> We are curious if Point-Stat logging could provide a way to
quickly
>>>> identify the station whose data is causing the error. We
performed
>>>> searches in the llittle_r file to locate the observation which is
causing
>>>> the problem, but it is not easy or always successful. We have
identified
>>>> some things to consider which might alleviate the problem in the
dialog
>>>> below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
>>>> observation or perhaps change the way the duplicate_flag works to
avoid
>>>> this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev:
>>>> 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev:
>>>> 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev:
>>>> 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev:
>>>> 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev:
>>>> 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered
>>>> duplicate for observation AIRCAR for P225-100, and the next few
lines are
>>>> the same for the V observations. So the third duplicate in the V
does
>>>> not have a matching duplicate in the U. Looking through the
little_r
>>>> files, I don't find two observations that exactly match that
listing.
>>>> However, I did find two that were pretty close, and perhaps close
enough
>>>> to be considered duplicate by MET? So the two observations are:
>>>> 36.90000 -119.5900000010424
>>>> MADIS
>>>> -888888.00000 1
0
>>>> 0 0 0 T F F -888888
-888888
>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>> 0-888888.00000 0-888888.00000
>>>> 0-888888.00000-888888-888888.00000 0-888888.00000
>>>> 0-888888.00000 0-888888.00000 0-888888.00000
>>>> 0-888888.00000 0-888888.00000 0
>>>> 18734.89063 512 12192.00000 0 220.00000
>>>> 0-888888.00000-888888 11.80000 0 274.00000 0
>>>> 11.77126 0 -0.82313
>>>> 0-888888.00000-888888-888888.00000-888888
>>>>
>>>> 36.89000 -119.5800000010424
>>>> MADIS
>>>> -888888.00000 1
0
>>>> 0 0 0 T F F -888888
-888888
>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>> 0-888888.00000 0-888888.00000
>>>> 0-888888.00000-888888-888888.00000 0-888888.00000
>>>> 0-888888.00000 0-888888.00000 0-888888.00000
>>>> 0-888888.00000 0-888888.00000 0
>>>> 18734.89063 512 12192.00000 0 219.70000
>>>> 0-888888.00000-888888 11.80000 0 274.00000 0
>>>> 11.77125 0 -0.82313
>>>> 0-888888.00000-888888-888888.00000-888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical
>>>> but are 0.01 off of each other. Perhaps this is considered close
enough
>>>> by MET. For the first observation u,v is (11.77126,-0.82313) and
for the
>>>> second observation u,v is (11.77125,-0.82313). So the V's match
but the
>>>> U's differ by 0.00001 m/s, presumably due to rounding
issues/machine
>>>> precision. So my guess is that these observations are the source
of the
>>>> problem. Assuming that this is correct, the next question is how
to fix
>>>> this. A few possibilities come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the
>>>> lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type
>>>> differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components
>>>> together, or at least U/V together. In other words, for an ob to
be
>>>> marked duplicate require that all observed fields match or at
least
>>>> require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark
>>>> as duplicate to MET help to see what they think. For a quick fix
you
>>>> could go and edit one of the two observations so that the U's
match,
>>>> since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Mar 27 13:33:24 2014
Yep, already did.
Thanks,
John
On 03/27/2014 01:30 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for taking a look at this. I will now switch over to test the
solution
> you recommend and report back.
>
> Can you delete the email which delivered the info to download the
files?
>
> Thanks.
>
> R/
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:25 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat. And I was
able to
> get matched pair output. Upon closer inspection of my log file
versus yours,
> I see the problem. When I run Point-Stat, I see the following line
included
> in the log file:
>
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
> "station_list.txt"
> DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS"
> mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
> "station_list.txt"
>
> The short explanation is that point_stat can't find the file
> "station_list.txt" in the directory it's being run. The simple
solution is to
> include the full path for "station_list.txt" in the Point-Stat
config file. I
> believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id
> setting functioned a bit differently. You listed the names of the
stations
> directly in the config file itself. Each station name was treated
as it's own
> masking region - so we were basically computing statistics over a
single
> matched pair. After we added the matched pair (MPR) output line
type, this
> type of station id masking was no longer needed. So we modified the
logic.
> In METv4.1, you list one or more strings in the "mask.sid" setting.
> Point-Stat checks each string to see if it's a file name. If so, it
parses
> the group of station names from the file listed. If not, it treats
the string
> as the name of a single station to be used. That enables us to
maintain
> backward compatibility.
>
> In your case, you listed a file but didn't list the full path. When
> Point-Stat couldn't find that file, it assumed that was the name of
the
> station to be used. But you don't have any observations for a
station named
> "station_list.txt". Thus, it found 0 matched pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a
> pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange
>> system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks. I now have the config file and station list. I just
need the
>> forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send
>>> you the data via anonymous ftp, so I will give it a try and let
you know
>>> how it goes. Alternatively I can send the little_r file by email
since it's
>>> only 5.5 mb, but the forecast file at 15 mb may be too big for
email.
>>>
>>> You should have the log file and my configuration file already
from my
>>> previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is:
>>> qc_obs_used_earth_relative.d01.2012-02-07_23:00:00.0000
>>>
>>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>> - forecast file, little_r file, Point-Stat config file, and
the
>>> "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list,
>>>> putting it in the same directory along with the config file and
then
>>>> running Point-Stat. I changed the config file (attached) to match
your
>>>> setting so it can read the station_list.txt file instead of a
poly file or
>>>> the grid (FULL). I used the same 3 station ids we identified as
suspect
>>>> per the email below. The log file showed that Point-Stat ran
without
>>>> error, but the output files contain no numerical results, only
column
>>>> headers. I experimented with adding to and subtracting from the
id list
>>>> with known stations based on looking at the NetCDF obs file for
the same
>>>> hour, but never produced output with actual statistics, just
headers. I
>>>> removed those stations from the list and entered a new set of
surface obs
>>>> (ADPSFC) which should have produced matched pairs at the Z2/Z10
levels,
>>>> but no change. The log file (attached) shows no matched pairs in
every
>>>> case. The output (CNT file) shows the headers, but no sta!
> t!
>> i!
>>> st!
>>>> ical results. I'm beginning to think that for some reason
it's not
>>>> reading or performing the analysis with the station ids. The log
file
>>>> acknowledges the presence of and handling of the list as follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
>>>> "station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning
>>>> how to use this functionality correctly in order to use it to
potentially
>>>> solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and
>>>> the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question. Here is an excerpt from
this file
>>>> METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>> 238 // build a uniqueness test key
>>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>> 240 lat, // lat
>>>> 241 lon, // lon
>>>> 242 lvl, // level
>>>> 243 elv).text(); // elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the
>>>> latitude and longitude. So a difference of 0.01 would not be
"close
>>>> enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
>>>> figure out the smallest subset of littl_r data that reproduces
this
>>>> problem. If there are just a couple of station id's that are
suspect, I'd
>>>> suggest using the station id masking to narrow down exactly which
>>>> station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>> grid = [];
>>>> poly = [];
>>>> sid = [ "station_list.txt" ]; };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's.
>>>> You could check to see if just those 3 stations replicate the
error and
>>>> then either add/subtract stations from that list to really narrow
it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to
>>>> send me a sample little_r file, forecast file, and Point-Stat
>>>> configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>> Queue: met_help
>>>>> Subject: Point_Stat error
>>>>> Owner: Nobody
>>>>> Requestors: john.w.raby2.civ at mail.mil
>>>>> Status: new
>>>>> Ticket <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations
>>>>> file. Using MET V4.1, the log file shows the following dialog:
>>>>>
>>>>> ERROR :
>>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
>>>>> VGRD pairs: 159 != 158 ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided
>>>>> below in the flags for duplicate data may shed some light on the
problem.
>>>>> We are curious if Point-Stat logging could provide a way to
quickly
>>>>> identify the station whose data is causing the error. We
performed
>>>>> searches in the llittle_r file to locate the observation which
is causing
>>>>> the problem, but it is not easy or always successful. We have
identified
>>>>> some things to consider which might alleviate the problem in the
dialog
>>>>> below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
>>>>> observation or perhaps change the way the duplicate_flag works
to avoid
>>>>> this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev:
>>>>> 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev:
>>>>> 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev:
>>>>> 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev:
>>>>> 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev:
>>>>> 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered
>>>>> duplicate for observation AIRCAR for P225-100, and the next few
lines are
>>>>> the same for the V observations. So the third duplicate in the
V does
>>>>> not have a matching duplicate in the U. Looking through the
little_r
>>>>> files, I don't find two observations that exactly match that
listing.
>>>>> However, I did find two that were pretty close, and perhaps
close enough
>>>>> to be considered duplicate by MET? So the two observations are:
>>>>> 36.90000 -119.5900000010424
>>>>> MADIS
>>>>> -888888.00000 1
0
>>>>> 0 0 0 T F F -888888
-888888
>>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>>> 0-888888.00000 0-888888.00000
>>>>> 0-888888.00000-888888-888888.00000 0-888888.00000
>>>>> 0-888888.00000 0-888888.00000 0-888888.00000
>>>>> 0-888888.00000 0-888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 220.00000
>>>>> 0-888888.00000-888888 11.80000 0 274.00000 0
>>>>> 11.77126 0 -0.82313
>>>>> 0-888888.00000-888888-888888.00000-888888
>>>>>
>>>>> 36.89000 -119.5800000010424
>>>>> MADIS
>>>>> -888888.00000 1
0
>>>>> 0 0 0 T F F -888888
-888888
>>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>>> 0-888888.00000 0-888888.00000
>>>>> 0-888888.00000-888888-888888.00000 0-888888.00000
>>>>> 0-888888.00000 0-888888.00000 0-888888.00000
>>>>> 0-888888.00000 0-888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 219.70000
>>>>> 0-888888.00000-888888 11.80000 0 274.00000 0
>>>>> 11.77125 0 -0.82313
>>>>> 0-888888.00000-888888-888888.00000-888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical
>>>>> but are 0.01 off of each other. Perhaps this is considered
close enough
>>>>> by MET. For the first observation u,v is (11.77126,-0.82313)
and for the
>>>>> second observation u,v is (11.77125,-0.82313). So the V's match
but the
>>>>> U's differ by 0.00001 m/s, presumably due to rounding
issues/machine
>>>>> precision. So my guess is that these observations are the
source of the
>>>>> problem. Assuming that this is correct, the next question is
how to fix
>>>>> this. A few possibilities come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the
>>>>> lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type
>>>>> differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components
>>>>> together, or at least U/V together. In other words, for an ob
to be
>>>>> marked duplicate require that all observed fields match or at
least
>>>>> require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark
>>>>> as duplicate to MET help to see what they think. For a quick
fix you
>>>>> could go and edit one of the two observations so that the U's
match,
>>>>> since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:46:44 2014
John -
I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
DEBUG 1: Climatology File: none
DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
_23:00:00.0000.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for TMP/P225-100.
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 32
of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 41
of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 50
of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB records
matching VarInfo "TMP/P225-100" in GRIB f
ile "/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-
02-07_23:00:00".
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
ERROR :
ERROR : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
on_list.txt".
ERROR :
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:24 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
OK, I received the files and ran them through Point-Stat. And I was
able to get matched pair output. Upon closer inspection of my log
file versus yours, I see the problem. When I run Point-Stat, I
see the following line included in the log file:
DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
But in the log file you sent me, I see this instead:
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to include the full path for "station_list.txt" in the
Point-Stat config file. I believe that will solve your problem.
The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently. You listed the names
of the stations directly in the config file itself. Each station
name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair. After we added the
matched pair (MPR) output line type, this type of station id
masking was no longer needed. So we modified the logic. In METv4.1,
you list one or more strings in the "mask.sid" setting. Point-Stat
checks each string to see if it's a file name. If so, it
parses the group of station names from the file listed. If not, it
treats the string as the name of a single station to be used. That
enables us to maintain backward compatibility.
In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used. But you don't have any observations
for a station named "station_list.txt". Thus, it found 0 matched
pairs.
Point-Stat did tell us all the information in the log file, but it was
a pretty subtle point.
Please give that a shot and let me know how it goes.
Thanks,
John
On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks. I now have the config file and station list. I just
need the forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>
>> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>>
>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>
>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>> - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no sta!
t!
> i!
>> st!
>>> ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file acknowledges the presence of and handling of the list as
follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>> 238 // build a uniqueness test key
>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>> 240 lat, // lat
>>> 241 lon, // lon
>>> 242 lvl, // level
>>> 243 elv).text(); // elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude. So a difference of 0.01 would not be
"close enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>> grid = [];
>>> poly = [];
>>> sid = [ "station_list.txt" ];
>>> };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's. You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>> narrow it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>> Queue: met_help
>>>> Subject: Point_Stat error
>>>> Owner: Nobody
>>>> Requestors: john.w.raby2.civ at mail.mil
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>
>>>> ERROR :
>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>> ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think. For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 13:50:23 2014
John,
Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.
The ~ is something interpreted by the shell. You're passing this to
the MET Point-Stat tool, not the shell. That character is the source
of the error you're seeing.
John
On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR :
> ERROR : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat. And I was
able to get matched pair output. Upon closer inspection of my log
file versus yours, I see the problem. When I run Point-Stat, I
> see the following line included in the log file:
>
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
> DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file. I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently. You listed the names
of the stations directly in the config file itself. Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair. After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed. So we modified the logic. In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name. If so, it
> parses the group of station names from the file listed. If not, it
treats the string as the name of a single station to be used. That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used. But you don't have any observations
> for a station named "station_list.txt". Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks. I now have the config file and station list. I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>> - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
a!
> t!
>> i!
>>> st!
>>>> ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question. Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>> 238 // build a uniqueness test key
>>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>> 240 lat, // lat
>>>> 241 lon, // lon
>>>> 242 lvl, // level
>>>> 243 elv).text(); // elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude. So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>> grid = [];
>>>> poly = [];
>>>> sid = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's. You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>> Queue: met_help
>>>>> Subject: Point_Stat error
>>>>> Owner: Nobody
>>>>> Requestors: john.w.raby2.civ at mail.mil
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR :
>>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 220.00000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 219.70000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think. For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:53:33 2014
Ok. thanks for the quick response. I'll try that.
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:50 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.
The ~ is something interpreted by the shell. You're passing this to
the MET Point-Stat tool, not the shell. That character is the source
of the error you're seeing.
John
On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR :
> ERROR : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat. And I was
able to get matched pair output. Upon closer inspection of my log
file versus yours, I see the problem. When I run Point-Stat, I
> see the following line included in the log file:
>
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
> DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file. I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently. You listed the names
of the stations directly in the config file itself. Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair. After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed. So we modified the logic. In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name. If so, it
> parses the group of station names from the file listed. If not, it
treats the string as the name of a single station to be used. That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used. But you don't have any observations
> for a station named "station_list.txt". Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks. I now have the config file and station list. I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>> - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
a!
> t!
>> i!
>>> st!
>>>> ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question. Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>> 238 // build a uniqueness test key
>>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>> 240 lat, // lat
>>>> 241 lon, // lon
>>>> 242 lvl, // level
>>>> 243 elv).text(); // elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude. So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>> grid = [];
>>>> poly = [];
>>>> sid = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's. You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>> Queue: met_help
>>>>> Subject: Point_Stat error
>>>>> Owner: Nobody
>>>>> Requestors: john.w.raby2.civ at mail.mil
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR :
>>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 220.00000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 219.70000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think. For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 14:03:41 2014
John -
You are correct and that fixed the problem! Now I get the same
response you did when it reads the station list:
DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"/home/jraby/MET_PointStat/station_list.txt"
DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file
"/home/jraby/MET_PointStat/station_list.txt"
I have good output files now too.
Thanks for all the help.
I'll be testing his with the IDs which may have caused the U and V
problems now to see if can track that down.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:50 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
John,
Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.
The ~ is something interpreted by the shell. You're passing this to
the MET Point-Stat tool, not the shell. That character is the source
of the error you're seeing.
John
On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
> of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR :
> ERROR : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat. And I was
able to get matched pair output. Upon closer inspection of my log
file versus yours, I see the problem. When I run Point-Stat, I
> see the following line included in the log file:
>
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
> DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run. The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file. I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently. You listed the names
of the stations directly in the config file itself. Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair. After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed. So we modified the logic. In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name. If so, it
> parses the group of station names from the file listed. If not, it
treats the string as the name of a single station to be used. That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path. When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used. But you don't have any observations
> for a station named "station_list.txt". Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks. I now have the config file and station list. I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is: 131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>> - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
a!
> t!
>> i!
>>> st!
>>>> ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question. Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>> 238 // build a uniqueness test key
>>>> 239 string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>> 240 lat, // lat
>>>> 241 lon, // lon
>>>> 242 lvl, // level
>>>> 243 elv).text(); // elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude. So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem. If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>> grid = [];
>>>> poly = [];
>>>> sid = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's. You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>> Queue: met_help
>>>>> Subject: Point_Stat error
>>>>> Owner: Nobody
>>>>> Requestors: john.w.raby2.civ at mail.mil
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR :
>>>>> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid:
20120207_225800 lev: 187.38 elv: 12191.00 ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid:
20120207_230700 lev: 187.17 elv: 12198.00 ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid:
20120207_230900 lev: 187.35 elv: 12192.00 ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
>>>>> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 220.00000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
>>>>> 18734.89063 512 12192.00000 0 219.70000
0-888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together. In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think. For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 15:16:00 2014
John, -
Once again your assistance has really facilitated progress on our
project.
The advice on the problem which prevented me from running Point-Stat
with a station list was accurate and resulted in a quick resolution
with a minimum of delay enabling our work to proceed.
I was able to use the ability to run Point-Stat on specified stations
which were identified by Dr. Brian Reen as those which may be causing
a problem which prevented Point-Stat from running. This was due to
apparent issues with the U and V data.
I reran Point-Stat with the identifiers 00010426, 00010495, 00010424
in the systematic way you suggested and narrowed it down to one
station (00010424) which produces the error while the other two do
not. Next, I will open that little_r file and try and locate and then
edit or remove the offending data.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 07, 2014 11:57 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
Hello John,
Hmmm, sounds like a tricky question. Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
238 // build a uniqueness test key
239 string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
240 lat, // lat
241 lon, // lon
242 lvl, // level
243 elv).text(); // elevation
As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude. So a difference of 0.01 would not be
"close enough".
If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
In the point_stat config file, set:
mask = {
grid = [];
poly = [];
sid = [ "station_list.txt" ];
};
Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424
That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.
I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Point_Stat error
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR :
> ERROR : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3: [lat: 34.0000 lon: -118.3000 valid: 20120207_225800
lev: 187.38 elv: 12191.00 ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3: [lat: 34.7600 lon: -118.6500 valid: 20120207_230700
lev: 187.17 elv: 12198.00 ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3: [lat: 36.9000 lon: -119.5900 valid: 20120207_230900
lev: 187.35 elv: 12192.00 ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations. So the third duplicate
in the V does not have a matching duplicate in the U. Looking through
the little_r files, I don't find two observations that exactly match
that listing. However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET? So the two
observations are:
> 36.90000 -119.5900000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 220.00000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77126 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> 36.89000 -119.5800000010424
MADIS FM-96
-888888.00000 1 0 0 0 0
T F F -888888 -888888 20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000-888888-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0-888888.00000 0-888888.00000 0-
888888.00000 0
> 18734.89063 512 12192.00000 0 219.70000 0-
888888.00000-888888 11.80000 0 274.00000 0
11.77125 0 -0.82313 0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other. Perhaps this is considered
close enough by MET. For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision. So my guess is that these
observations are the source of the problem. Assuming that this is
correct, the next question is how to fix this. A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together. In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think. For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list