[Met_help] [rt.rap.ucar.edu #65756] History for Point_Stat error

John Halley Gotway via RT met_help at ucar.edu
Thu Mar 27 14:12:57 MDT 2014


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I am getting an error running Point-Stat with a little_r observations file. Using MET V4.1, the log file shows the following dialog:

ERROR  :
ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of VGRD pairs: 159 != 158
ERROR

I've attached the log file, but we feel that the information provided below in the flags for duplicate data may shed some light on the problem. We are curious if Point-Stat logging could provide a way to quickly identify the station whose data is causing the error. We performed searches in the llittle_r file to locate the observation which is causing the problem, but it is not easy or always successful. We have identified some things to consider which might alleviate the problem in the dialog below.

Do you have any suggestions on how to track down the offending observation or perhaps change the way the duplicate_flag works to avoid this problem?

Thanks.
________________________________

DEBUG 3: Duplicate point observations for -unique setting:
DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426, 00010426
DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495, 00010495
DEBUG 3:
DEBUG 3:
DEBUG 3: Duplicate point observations for -unique setting:
DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426, 00010426
DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495, 00010495
DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424, 00010424

I think that the first lines are the U observations that it considered duplicate for observation AIRCAR for P225-100, and the next few lines are the same for the V observations.  So the third duplicate in the V does not have a matching duplicate in the U.  Looking through the little_r files, I don't find two observations that exactly match that listing.  However, I did find two that were pretty close, and perhaps close enough to be considered duplicate by MET?  So the two observations are:
            36.90000          -119.5900000010424                                MADIS                                   FM-96                                                                                  -888888.00000         1         0         0         0         0         T         F         F   -888888   -888888      20120207230900-888888.00000-888888-888888.00000-888888-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000-888888-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0
  18734.89063    512  12192.00000      0    220.00000      0-888888.00000-888888     11.80000      0    274.00000      0     11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-888888

            36.89000          -119.5800000010424                                MADIS                                   FM-96                                                                                  -888888.00000         1         0         0         0         0         T         F         F   -888888   -888888      20120207230900-888888.00000-888888-888888.00000-888888-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000-888888-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0-888888.00000      0
  18734.89063    512  12192.00000      0    219.70000      0-888888.00000-888888     11.80000      0    274.00000      0     11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-888888

The latitudes and longitudes of the two observations are not identical but are 0.01 off of each other.  Perhaps this is considered close enough by MET.  For the first observation u,v is (11.77126,-0.82313) and for the second observation u,v is (11.77125,-0.82313).  So the V's match but the U's differ by 0.00001 m/s, presumably due to rounding issues/machine precision.  So my guess is that these observations are the source of the problem.  Assuming that this is correct, the next question is how to fix this.  A few possibilities come to mind:
1) Change MET to insist that for observations to be duplicate that the lat/lon match closer than 0.01 degrees (which is about 1km).
2) Change the duplicate checking to allow for machine precision type differences between observations.
3) Change the duplicate checking to check all observation components together, or at least U/V together.  In other words, for an ob to be marked duplicate require that all observed fields match or at least require that U/V match.

I'm thinking we might want to send these two obs that MET appears to mark as duplicate to MET help to see what they think.  For a quick fix you could go and edit one of the two observations so that the U's match, since a change of 0.00001 m/s is not meaningful.

R/
John


Mr John W. Raby, Meteorologist

U.S. Army Research Laboratory

White Sands Missile Range, NM 88002

(575) 678-2004 DSN 258-2004

FAX (575) 678-1230 DSN 258-1230

Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>



________________________________
From: Raby, John W CIV USARMY ARL (US)
Sent: Wednesday, March 05, 2014 2:31 PM
To: Reen, Brian P CIV USARMY ARL (US)
Subject: Point_Stat log file







----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Fri Mar 07 11:57:03 2014

Hello John,

Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:

     238       //  build a uniqueness test key
     239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
     240             lat,         //  lat
     241             lon,         //  lon
     242             lvl,         //  level
     243             elv).text(); //  elevation

As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude.  So a difference of 0.01 would not be
"close enough".

If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.

In the point_stat config file, set:
mask = {
    grid = [];
    poly = [];
    sid  = [ "station_list.txt" ];
};

Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424

That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.

I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.

Thanks,
John Halley Gotway
met_help at ucar.edu


On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>         Queue: met_help
>       Subject: Point_Stat error
>         Owner: Nobody
>    Requestors: john.w.raby2.civ at mail.mil
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR  :
> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>              36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
>              36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together.  In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think.  For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Fri Mar 07 13:00:06 2014

John -

Great idea. Thanks for looking into this. I had not considered that
functionality of Point-Stat to narrow down the search through the use
of station identifiers. These little_r files are very large and
difficult to sort through and this should be a real help. I will give
it a try per your suggested technique below. If it works, then I might
be able to use the information to select the station(s) whose data
need to be deleted and fix the problem that way, too. Will be in touch
soon.

R/
John

Mr John W. Raby, Meteorologist

U.S. Army Research Laboratory

White Sands Missile Range, NM 88002

(575) 678-2004 DSN 258-2004

FAX (575) 678-1230 DSN 258-1230

Email: john.w.raby2.civ at mail.mil

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 07, 2014 11:57 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

Hello John,

Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:

     238       //  build a uniqueness test key
     239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
     240             lat,         //  lat
     241             lon,         //  lon
     242             lvl,         //  level
     243             elv).text(); //  elevation

As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude.  So a difference of 0.01 would not be
"close enough".

If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.

In the point_stat config file, set:
mask = {
    grid = [];
    poly = [];
    sid  = [ "station_list.txt" ];
};

Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424

That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.

I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.

Thanks,
John Halley Gotway
met_help at ucar.edu


On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>         Queue: met_help
>       Subject: Point_Stat error
>         Owner: Nobody
>    Requestors: john.w.raby2.civ at mail.mil
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR  :
> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>              36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
>              36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together.  In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think.  For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>



------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 08:47:24 2014

John -

I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no statistical results. I'm beginning to think
that for some reason it's not reading or performing the analysis with
the station ids. The log file  acknowledges the presence of and
handling of the list as follows:

DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"

So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.

Let me know if I can provide you the data. The forecast file is 15mb,
and the little_r file is 6mb.

Thanks.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 07, 2014 11:57 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

Hello John,

Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:

     238       //  build a uniqueness test key
     239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
     240             lat,         //  lat
     241             lon,         //  lon
     242             lvl,         //  level
     243             elv).text(); //  elevation

As you can see, the uniqueness key is built using 3 decimal places for
the latitude and longitude.  So a difference of 0.01 would not be
"close enough".

If you'd like me to debug this some more, it would be ideal to figure
out the smallest subset of littl_r data that reproduces this problem.
If there are just a couple of station id's that are
suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.

In the point_stat config file, set:
mask = {
    grid = [];
    poly = [];
    sid  = [ "station_list.txt" ];
};

Where station_list.txt is an ASCII file containing something like:
TEST_STATIONS
00010426
00010495
00010424

That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
narrow it down.

I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.

Thanks,
John Halley Gotway
met_help at ucar.edu


On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>
> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>         Queue: met_help
>       Subject: Point_Stat error
>         Owner: Nobody
>    Requestors: john.w.raby2.civ at mail.mil
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
>
> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>
> ERROR  :
> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
> ERROR
>
> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>
> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>
> Thanks.
> ________________________________
>
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
> DEBUG 3:
> DEBUG 3:
> DEBUG 3: Duplicate point observations for -unique setting:
> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>
> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>              36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
>              36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>    18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>
> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
> 2) Change the duplicate checking to allow for machine precision type
differences between observations.
> 3) Change the duplicate checking to check all observation components
together, or at least U/V together.  In other words, for an ob to be
marked duplicate require that all observed fields match or at least
require that U/V match.
>
> I'm thinking we might want to send these two obs that MET appears to
mark as duplicate to MET help to see what they think.  For a quick fix
you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
>
> ________________________________
> From: Raby, John W CIV USARMY ARL (US)
> Sent: Wednesday, March 05, 2014 2:31 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Subject: Point_Stat log file
>
>
>
>
>
>


------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 09:43:22 2014

John,

Sure, go ahead and send the data:
  - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file

You can post it to our anonymous ftp site following these
instructions:
    http://www.dtcenter.org/met/users/support/met_help.php#ftp

Thanks,
John

On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stati!
 st!
>   ical results. I'm beginning to think that for some reason it's not
reading or performing the analysis with the station ids. The log file
acknowledges the presence of and handling of the list as follows:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>
> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, March 07, 2014 11:57 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> Hello John,
>
> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>
>       238       //  build a uniqueness test key
>       239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>       240             lat,         //  lat
>       241             lon,         //  lon
>       242             lvl,         //  level
>       243             elv).text(); //  elevation
>
> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>
> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>
> In the point_stat config file, set:
> mask = {
>      grid = [];
>      poly = [];
>      sid  = [ "station_list.txt" ];
> };
>
> Where station_list.txt is an ASCII file containing something like:
> TEST_STATIONS
> 00010426
> 00010495
> 00010424
>
> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
> narrow it down.
>
> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>
>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>          Queue: met_help
>>        Subject: Point_Stat error
>>          Owner: Nobody
>>     Requestors: john.w.raby2.civ at mail.mil
>>         Status: new
>>    Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>>
>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>
>> ERROR  :
>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>> ERROR
>>
>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>
>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>
>> Thanks.
>> ________________________________
>>
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>> DEBUG 3:
>> DEBUG 3:
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>
>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>               36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>     18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>
>>               36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>     18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>
>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>
>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>
>> R/
>> John
>>
>>
>> Mr John W. Raby, Meteorologist
>>
>> U.S. Army Research Laboratory
>>
>> White Sands Missile Range, NM 88002
>>
>> (575) 678-2004 DSN 258-2004
>>
>> FAX (575) 678-1230 DSN 258-1230
>>
>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>
>>
>>
>> ________________________________
>> From: Raby, John W CIV USARMY ARL (US)
>> Sent: Wednesday, March 05, 2014 2:31 PM
>> To: Reen, Brian P CIV USARMY ARL (US)
>> Subject: Point_Stat log file
>>
>>
>>
>>
>>
>>
>
>

------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 09:59:49 2014

John -

OK. Thanks.

I've had trouble lately with the WSMR firewall and may not be able to
send you the data via anonymous ftp, so I will give it a try and let
you know how it goes. Alternatively I can send the little_r file by
email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.

You should have the log file and my configuration file already from my
previous email. Let me know if you didn't get it.

The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000

The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00

I've attached the station_list.txt file to this email.





________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 9:43 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

Sure, go ahead and send the data:
  - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file

You can post it to our anonymous ftp site following these
instructions:
    http://www.dtcenter.org/met/users/support/met_help.php#ftp

Thanks,
John

On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stati!
 st!
>   ical results. I'm beginning to think that for some reason it's not
reading or performing the analysis with the station ids. The log file
acknowledges the presence of and handling of the list as follows:
>
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>
> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, March 07, 2014 11:57 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> Hello John,
>
> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>
>       238       //  build a uniqueness test key
>       239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>       240             lat,         //  lat
>       241             lon,         //  lon
>       242             lvl,         //  level
>       243             elv).text(); //  elevation
>
> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>
> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>
> In the point_stat config file, set:
> mask = {
>      grid = [];
>      poly = [];
>      sid  = [ "station_list.txt" ];
> };
>
> Where station_list.txt is an ASCII file containing something like:
> TEST_STATIONS
> 00010426
> 00010495
> 00010424
>
> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
> narrow it down.
>
> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>
>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>          Queue: met_help
>>        Subject: Point_Stat error
>>          Owner: Nobody
>>     Requestors: john.w.raby2.civ at mail.mil
>>         Status: new
>>    Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>>
>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>
>> ERROR  :
>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>> ERROR
>>
>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>
>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>
>> Thanks.
>> ________________________________
>>
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>> DEBUG 3:
>> DEBUG 3:
>> DEBUG 3: Duplicate point observations for -unique setting:
>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>
>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>               36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>     18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>
>>               36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>     18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>
>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>
>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>
>> R/
>> John
>>
>>
>> Mr John W. Raby, Meteorologist
>>
>> U.S. Army Research Laboratory
>>
>> White Sands Missile Range, NM 88002
>>
>> (575) 678-2004 DSN 258-2004
>>
>> FAX (575) 678-1230 DSN 258-1230
>>
>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>
>>
>>
>> ________________________________
>> From: Raby, John W CIV USARMY ARL (US)
>> Sent: Wednesday, March 05, 2014 2:31 PM
>> To: Reen, Brian P CIV USARMY ARL (US)
>> Subject: Point_Stat log file
>>
>>
>>
>>
>>
>>
>
>


------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 09:59:49 2014

TEST_STATIONS
C6937
KELC1
QLGQ3
SRTC1
KFLG
TS776
QRCA3
KUKI
KNFL
KAJO
KINS
KNXP
KLAS
KALK
D8496
D2849
D8204
AT450
AU043

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 10:45:41 2014

John,

OK thanks.  I now have the config file and station list.  I just need
the forecast file and little_r file.

Thanks,
John

On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
>    - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
>      http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
 i!
>   st!
>>    ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>>        238       //  build a uniqueness test key
>>        239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>>        240             lat,         //  lat
>>        241             lon,         //  lon
>>        242             lvl,         //  level
>>        243             elv).text(); //  elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>>       grid = [];
>>       poly = [];
>>       sid  = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>           Queue: met_help
>>>         Subject: Point_Stat error
>>>           Owner: Nobody
>>>      Requestors: john.w.raby2.civ at mail.mil
>>>          Status: new
>>>     Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR  :
>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>                36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>>                36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 11:00:24 2014

John -

I attempted to ftp the files, but I'm blocked by WSMR from doing so
(connection refused). I confirmed that the block is a local one by
checking with our sysadm person.

Do you happen to have a sftp site you can point me to to accomplish
the transfer?

R/
John



________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

OK thanks.  I now have the config file and station list.  I just need
the forecast file and little_r file.

Thanks,
John

On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
>    - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
>      http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
 i!
>   st!
>>    ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>>        238       //  build a uniqueness test key
>>        239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>>        240             lat,         //  lat
>>        241             lon,         //  lon
>>        242             lvl,         //  level
>>        243             elv).text(); //  elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>>       grid = [];
>>       poly = [];
>>       sid  = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>           Queue: met_help
>>>         Subject: Point_Stat error
>>>           Owner: Nobody
>>>      Requestors: john.w.raby2.civ at mail.mil
>>>          Status: new
>>>     Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR  :
>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>                36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>>                36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 12:10:51 2014

John -

I just sent you the files using a file exchange system. Please let me
know if this works for you.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

OK thanks.  I now have the config file and station list.  I just need
the forecast file and little_r file.

Thanks,
John

On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
>    - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
>      http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
 i!
>   st!
>>    ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>>        238       //  build a uniqueness test key
>>        239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>>        240             lat,         //  lat
>>        241             lon,         //  lon
>>        242             lvl,         //  level
>>        243             elv).text(); //  elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>>       grid = [];
>>       poly = [];
>>       sid  = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>           Queue: met_help
>>>         Subject: Point_Stat error
>>>           Owner: Nobody
>>>      Requestors: john.w.raby2.civ at mail.mil
>>>          Status: new
>>>     Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR  :
>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>                36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>>                36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 12:17:34 2014

John -

Just in case, please delete the email you receive from the file
exchange system after you download the files.

Thanks.

John

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 10:45 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

OK thanks.  I now have the config file and station list.  I just need
the forecast file and little_r file.

Thanks,
John

On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> OK. Thanks.
>
> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>
> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>
> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>
> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>
> I've attached the station_list.txt file to this email.
>
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 9:43 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> Sure, go ahead and send the data:
>    - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>
> You can post it to our anonymous ftp site following these
instructions:
>      http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> I tried the approach you suggested by creating the station id list,
putting it in the same directory along with the config file and then
running Point-Stat. I changed the config file (attached) to match your
setting so it can read the station_list.txt file instead of a poly
file or the grid (FULL). I used the same 3 station ids we identified
as suspect per the email below. The log file showed that Point-Stat
ran without error, but the output files contain no numerical results,
only column headers. I experimented with adding to and subtracting
from the id list with known stations based on looking at the NetCDF
obs file for the same hour, but never produced output with actual
statistics, just headers. I removed those stations from the list and
entered a new set of surface obs (ADPSFC) which should have produced
matched pairs at the Z2/Z10 levels, but no change. The log file
(attached) shows no matched pairs in every case. The output (CNT file)
shows the headers, but no stat!
 i!
>   st!
>>    ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>
>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>
>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>
>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>
>> Thanks.
>>
>> R/
>> John
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Friday, March 07, 2014 11:57 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> Hello John,
>>
>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>
>>        238       //  build a uniqueness test key
>>        239       string sng_key = str_format("%.3f:%.3f:%.2f:%.2f",
>>        240             lat,         //  lat
>>        241             lon,         //  lon
>>        242             lvl,         //  level
>>        243             elv).text(); //  elevation
>>
>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>
>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>
>> In the point_stat config file, set:
>> mask = {
>>       grid = [];
>>       poly = [];
>>       sid  = [ "station_list.txt" ];
>> };
>>
>> Where station_list.txt is an ASCII file containing something like:
>> TEST_STATIONS
>> 00010426
>> 00010495
>> 00010424
>>
>> That will restrict the analysis to only that group of station id's.
You could check to see if just those 3 stations replicate the error
and then either add/subtract stations from that list to really
>> narrow it down.
>>
>> I could probably take a closer look at this next week, but I'd need
you to send me a sample little_r file, forecast file, and Point-Stat
configuration file to replicate the error.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>
>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>           Queue: met_help
>>>         Subject: Point_Stat error
>>>           Owner: Nobody
>>>      Requestors: john.w.raby2.civ at mail.mil
>>>          Status: new
>>>     Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>>
>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>
>>> ERROR  :
>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>> ERROR
>>>
>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>
>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>
>>> Thanks.
>>> ________________________________
>>>
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>> DEBUG 3:
>>> DEBUG 3:
>>> DEBUG 3: Duplicate point observations for -unique setting:
>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid: 20120207_225800
lev: 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid: 20120207_230700
lev: 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid: 20120207_230900
lev: 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>
>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>                36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>>                36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>      18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>
>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>> 1) Change MET to insist that for observations to be duplicate that
the lat/lon match closer than 0.01 degrees (which is about 1km).
>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>
>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>
>>> R/
>>> John
>>>
>>>
>>> Mr John W. Raby, Meteorologist
>>>
>>> U.S. Army Research Laboratory
>>>
>>> White Sands Missile Range, NM 88002
>>>
>>> (575) 678-2004 DSN 258-2004
>>>
>>> FAX (575) 678-1230 DSN 258-1230
>>>
>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>
>>>
>>>
>>> ________________________________
>>> From: Raby, John W CIV USARMY ARL (US)
>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>> To: Reen, Brian P CIV USARMY ARL (US)
>>> Subject: Point_Stat log file
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 13:24:53 2014

John,

OK, I received the files and ran them through Point-Stat.  And I was
able to get matched pair output.  Upon closer inspection of my log
file versus yours, I see the problem.  When I run Point-Stat, I
see the following line included in the log file:

   DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
   DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"

But in the log file you sent me, I see this instead:

   DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"

The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to include the full path for "station_list.txt" in the
Point-Stat config file.  I believe that will solve your problem.

The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently.  You listed the names
of the stations directly in the config file itself.  Each station
name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair.  After we added the
matched pair (MPR) output line type, this type of station id
masking was no longer needed.  So we modified the logic.  In METv4.1,
you list one or more strings in the "mask.sid" setting.  Point-Stat
checks each string to see if it's a file name.  If so, it
parses the group of station names from the file listed.  If not, it
treats the string as the name of a single station to be used.  That
enables us to maintain backward compatibility.

In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used.  But you don't have any observations
for a station named "station_list.txt".  Thus, it found 0 matched
pairs.

Point-Stat did tell us all the information in the log file, but it was
a pretty subtle point.

Please give that a shot and let me know how it goes.

Thanks,
John



On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks.  I now have the config file and station list.  I just
need the forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>
>> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>>
>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>
>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>>     - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>>       http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no sta!
 t!
>   i!
>>    st!
>>>     ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>>         238       //  build a uniqueness test key
>>>         239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>         240             lat,         //  lat
>>>         241             lon,         //  lon
>>>         242             lvl,         //  level
>>>         243             elv).text(); //  elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>>        grid = [];
>>>        poly = [];
>>>        sid  = [ "station_list.txt" ];
>>> };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's.  You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>> narrow it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>            Queue: met_help
>>>>          Subject: Point_Stat error
>>>>            Owner: Nobody
>>>>       Requestors: john.w.raby2.civ at mail.mil
>>>>           Status: new
>>>>      Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>
>>>> ERROR  :
>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>> ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>>                 36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>       18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>
>>>>                 36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>       18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

------------------------------------------------
Subject: Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:30:51 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks for taking a look at this. I will now switch over to test the
solution
you recommend and report back.

Can you delete the email which delivered the info to download the
files?

Thanks.

R/


-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:25 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

OK, I received the files and ran them through Point-Stat.  And I was
able to
get matched pair output.  Upon closer inspection of my log file versus
yours,
I see the problem.  When I run Point-Stat, I see the following line
included
in the log file:

   DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
   DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS"
mask from file "station_list.txt"

But in the log file you sent me, I see this instead:

   DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"

The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to
include the full path for "station_list.txt" in the Point-Stat config
file.  I
believe that will solve your problem.

The long explanation is this... in earlier versions of MET, the
station id
setting functioned a bit differently.  You listed the names of the
stations
directly in the config file itself.  Each station name was treated as
it's own
masking region - so we were basically computing statistics over a
single
matched pair.  After we added the matched pair (MPR) output line type,
this
type of station id masking was no longer needed.  So we modified the
logic.
In METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name.  If so, it
parses
the group of station names from the file listed.  If not, it treats
the string
as the name of a single station to be used.  That enables us to
maintain
backward compatibility.

In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the
station to be used.  But you don't have any observations for a station
named
"station_list.txt".  Thus, it found 0 matched pairs.

Point-Stat did tell us all the information in the log file, but it was
a
pretty subtle point.

Please give that a shot and let me know how it goes.

Thanks,
John



On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange
> system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks.  I now have the config file and station list.  I just
need the
> forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send
>> you the data via anonymous ftp, so I will give it a try and let you
know
>> how it goes. Alternatively I can send the little_r file by email
since it's
>> only 5.5 mb, but the forecast file at 15 mb may be too big for
email.
>>
>> You should have the log file and my configuration file already from
my
>> previous email. Let me know if you didn't get it.
>>
>> The obs filename is:
>> qc_obs_used_earth_relative.d01.2012-02-07_23:00:00.0000
>>
>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>>     - forecast file, little_r file, Point-Stat config file, and the
>> "station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>>       http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list,
>>> putting it in the same directory along with the config file and
then
>>> running Point-Stat. I changed the config file (attached) to match
your
>>> setting so it can read the station_list.txt file instead of a poly
file or
>>> the grid (FULL). I used the same 3 station ids we identified as
suspect
>>> per the email below. The log file showed that Point-Stat ran
without
>>> error, but the output files contain no numerical results, only
column
>>> headers. I experimented with adding to and subtracting from the id
list
>>> with known stations based on looking at the NetCDF obs file for
the same
>>> hour, but never produced output with actual statistics, just
headers. I
>>> removed those stations from the list and entered a new set of
surface obs
>>> (ADPSFC) which should have produced matched pairs at the Z2/Z10
levels,
>>> but no change. The log file (attached) shows no matched pairs in
every
>>> case. The output (CNT file) shows the headers, but no sta!
 t!
>   i!
>>    st!
>>>     ical results. I'm beginning to think that for some reason it's
not
>>> reading or performing the analysis with the station ids. The log
file
>>> acknowledges the presence of and handling of the list as follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
>>> "station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning
>>> how to use this functionality correctly in order to use it to
potentially
>>> solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and
>>> the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file
>>> METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>>         238       //  build a uniqueness test key
>>>         239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>         240             lat,         //  lat
>>>         241             lon,         //  lon
>>>         242             lvl,         //  level
>>>         243             elv).text(); //  elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the
>>> latitude and longitude.  So a difference of 0.01 would not be
"close
>>> enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
>>> figure out the smallest subset of littl_r data that reproduces
this
>>> problem.  If there are just a couple of station id's that are
suspect, I'd
>>> suggest using the station id masking to narrow down exactly which
>>> station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>>        grid = [];
>>>        poly = [];
>>>        sid  = [ "station_list.txt" ]; };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's.
>>> You could check to see if just those 3 stations replicate the
error and
>>> then either add/subtract stations from that list to really narrow
it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to
>>> send me a sample little_r file, forecast file, and Point-Stat
>>> configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>            Queue: met_help
>>>>          Subject: Point_Stat error
>>>>            Owner: Nobody
>>>>       Requestors: john.w.raby2.civ at mail.mil
>>>>           Status: new
>>>>      Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations
>>>> file. Using MET V4.1, the log file shows the following dialog:
>>>>
>>>> ERROR  :
>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
>>>> VGRD pairs: 159 != 158 ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided
>>>> below in the flags for duplicate data may shed some light on the
problem.
>>>> We are curious if Point-Stat logging could provide a way to
quickly
>>>> identify the station whose data is causing the error. We
performed
>>>> searches in the llittle_r file to locate the observation which is
causing
>>>> the problem, but it is not easy or always successful. We have
identified
>>>> some things to consider which might alleviate the problem in the
dialog
>>>> below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
>>>> observation or perhaps change the way the duplicate_flag works to
avoid
>>>> this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev:
>>>> 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev:
>>>> 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev:
>>>> 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev:
>>>> 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev:
>>>> 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered
>>>> duplicate for observation AIRCAR for P225-100, and the next few
lines are
>>>> the same for the V observations.  So the third duplicate in the V
does
>>>> not have a matching duplicate in the U.  Looking through the
little_r
>>>> files, I don't find two observations that exactly match that
listing.
>>>> However, I did find two that were pretty close, and perhaps close
enough
>>>> to be considered duplicate by MET?  So the two observations are:
>>>>                 36.90000          -119.5900000010424
>>>> MADIS
>>>>                                         -888888.00000         1
0
>>>> 0         0         0         T         F         F   -888888
-888888
>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>> 0-888888.00000      0-888888.00000
>>>> 0-888888.00000-888888-888888.00000      0-888888.00000
>>>> 0-888888.00000      0-888888.00000      0-888888.00000
>>>> 0-888888.00000      0-888888.00000      0
>>>>       18734.89063    512  12192.00000      0    220.00000
>>>> 0-888888.00000-888888     11.80000      0    274.00000      0
>>>> 11.77126      0     -0.82313
>>>> 0-888888.00000-888888-888888.00000-888888
>>>>
>>>>                 36.89000          -119.5800000010424
>>>> MADIS
>>>>                                         -888888.00000         1
0
>>>> 0         0         0         T         F         F   -888888
-888888
>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>> 0-888888.00000      0-888888.00000
>>>> 0-888888.00000-888888-888888.00000      0-888888.00000
>>>> 0-888888.00000      0-888888.00000      0-888888.00000
>>>> 0-888888.00000      0-888888.00000      0
>>>>       18734.89063    512  12192.00000      0    219.70000
>>>> 0-888888.00000-888888     11.80000      0    274.00000      0
>>>> 11.77125      0     -0.82313
>>>> 0-888888.00000-888888-888888.00000-888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical
>>>> but are 0.01 off of each other.  Perhaps this is considered close
enough
>>>> by MET.  For the first observation u,v is (11.77126,-0.82313) and
for the
>>>> second observation u,v is (11.77125,-0.82313).  So the V's match
but the
>>>> U's differ by 0.00001 m/s, presumably due to rounding
issues/machine
>>>> precision.  So my guess is that these observations are the source
of the
>>>> problem.  Assuming that this is correct, the next question is how
to fix
>>>> this.  A few possibilities come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the
>>>> lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type
>>>> differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components
>>>> together, or at least U/V together.  In other words, for an ob to
be
>>>> marked duplicate require that all observed fields match or at
least
>>>> require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark
>>>> as duplicate to MET help to see what they think.  For a quick fix
you
>>>> could go and edit one of the two observations so that the U's
match,
>>>> since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Mar 27 13:33:24 2014

Yep, already did.

Thanks,
John

On 03/27/2014 01:30 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for taking a look at this. I will now switch over to test the
solution
> you recommend and report back.
>
> Can you delete the email which delivered the info to download the
files?
>
> Thanks.
>
> R/
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:25 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat.  And I was
able to
> get matched pair output.  Upon closer inspection of my log file
versus yours,
> I see the problem.  When I run Point-Stat, I see the following line
included
> in the log file:
>
>     DEBUG 4: parse_sid_mask() -> parsing station ID masking file
> "station_list.txt"
>     DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS"
> mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
>     DEBUG 4: parse_sid_mask() -> storing single station ID mask
> "station_list.txt"
>
> The short explanation is that point_stat can't find the file
> "station_list.txt" in the directory it's being run.  The simple
solution is to
> include the full path for "station_list.txt" in the Point-Stat
config file.  I
> believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id
> setting functioned a bit differently.  You listed the names of the
stations
> directly in the config file itself.  Each station name was treated
as it's own
> masking region - so we were basically computing statistics over a
single
> matched pair.  After we added the matched pair (MPR) output line
type, this
> type of station id masking was no longer needed.  So we modified the
logic.
> In METv4.1, you list one or more strings in the "mask.sid" setting.
> Point-Stat checks each string to see if it's a file name.  If so, it
parses
> the group of station names from the file listed.  If not, it treats
the string
> as the name of a single station to be used.  That enables us to
maintain
> backward compatibility.
>
> In your case, you listed a file but didn't list the full path.  When
> Point-Stat couldn't find that file, it assumed that was the name of
the
> station to be used.  But you don't have any observations for a
station named
> "station_list.txt".  Thus, it found 0 matched pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a
> pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange
>> system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks.  I now have the config file and station list.  I just
need the
>> forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send
>>> you the data via anonymous ftp, so I will give it a try and let
you know
>>> how it goes. Alternatively I can send the little_r file by email
since it's
>>> only 5.5 mb, but the forecast file at 15 mb may be too big for
email.
>>>
>>> You should have the log file and my configuration file already
from my
>>> previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is:
>>> qc_obs_used_earth_relative.d01.2012-02-07_23:00:00.0000
>>>
>>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>>      - forecast file, little_r file, Point-Stat config file, and
the
>>> "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>>        http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list,
>>>> putting it in the same directory along with the config file and
then
>>>> running Point-Stat. I changed the config file (attached) to match
your
>>>> setting so it can read the station_list.txt file instead of a
poly file or
>>>> the grid (FULL). I used the same 3 station ids we identified as
suspect
>>>> per the email below. The log file showed that Point-Stat ran
without
>>>> error, but the output files contain no numerical results, only
column
>>>> headers. I experimented with adding to and subtracting from the
id list
>>>> with known stations based on looking at the NetCDF obs file for
the same
>>>> hour, but never produced output with actual statistics, just
headers. I
>>>> removed those stations from the list and entered a new set of
surface obs
>>>> (ADPSFC) which should have produced matched pairs at the Z2/Z10
levels,
>>>> but no change. The log file (attached) shows no matched pairs in
every
>>>> case. The output (CNT file) shows the headers, but no sta!
>   t!
>>    i!
>>>     st!
>>>>      ical results. I'm beginning to think that for some reason
it's not
>>>> reading or performing the analysis with the station ids. The log
file
>>>> acknowledges the presence of and handling of the list as follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
>>>> "station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning
>>>> how to use this functionality correctly in order to use it to
potentially
>>>> solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and
>>>> the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question.  Here is an excerpt from
this file
>>>> METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>>          238       //  build a uniqueness test key
>>>>          239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>>          240             lat,         //  lat
>>>>          241             lon,         //  lon
>>>>          242             lvl,         //  level
>>>>          243             elv).text(); //  elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the
>>>> latitude and longitude.  So a difference of 0.01 would not be
"close
>>>> enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
>>>> figure out the smallest subset of littl_r data that reproduces
this
>>>> problem.  If there are just a couple of station id's that are
suspect, I'd
>>>> suggest using the station id masking to narrow down exactly which
>>>> station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>>         grid = [];
>>>>         poly = [];
>>>>         sid  = [ "station_list.txt" ]; };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's.
>>>> You could check to see if just those 3 stations replicate the
error and
>>>> then either add/subtract stations from that list to really narrow
it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to
>>>> send me a sample little_r file, forecast file, and Point-Stat
>>>> configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>>             Queue: met_help
>>>>>           Subject: Point_Stat error
>>>>>             Owner: Nobody
>>>>>        Requestors: john.w.raby2.civ at mail.mil
>>>>>            Status: new
>>>>>       Ticket <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations
>>>>> file. Using MET V4.1, the log file shows the following dialog:
>>>>>
>>>>> ERROR  :
>>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
>>>>> VGRD pairs: 159 != 158 ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided
>>>>> below in the flags for duplicate data may shed some light on the
problem.
>>>>> We are curious if Point-Stat logging could provide a way to
quickly
>>>>> identify the station whose data is causing the error. We
performed
>>>>> searches in the llittle_r file to locate the observation which
is causing
>>>>> the problem, but it is not easy or always successful. We have
identified
>>>>> some things to consider which might alleviate the problem in the
dialog
>>>>> below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
>>>>> observation or perhaps change the way the duplicate_flag works
to avoid
>>>>> this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev:
>>>>> 187.38  elv: 12191.00  ob_val: 14.9425] - Station IDs: 00010426,
00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev:
>>>>> 187.17  elv: 12198.00  ob_val: 15.9748] - Station IDs: 00010495,
00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev:
>>>>> 187.38  elv: 12191.00  ob_val: -3.7256] - Station IDs: 00010426,
00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev:
>>>>> 187.17  elv: 12198.00  ob_val: -5.8143] - Station IDs: 00010495,
00010495
>>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev:
>>>>> 187.35  elv: 12192.00  ob_val: -0.8231] - Station IDs: 00010424,
00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered
>>>>> duplicate for observation AIRCAR for P225-100, and the next few
lines are
>>>>> the same for the V observations.  So the third duplicate in the
V does
>>>>> not have a matching duplicate in the U.  Looking through the
little_r
>>>>> files, I don't find two observations that exactly match that
listing.
>>>>> However, I did find two that were pretty close, and perhaps
close enough
>>>>> to be considered duplicate by MET?  So the two observations are:
>>>>>                  36.90000          -119.5900000010424
>>>>> MADIS
>>>>>                                          -888888.00000         1
0
>>>>> 0         0         0         T         F         F   -888888
-888888
>>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>>> 0-888888.00000      0-888888.00000
>>>>> 0-888888.00000-888888-888888.00000      0-888888.00000
>>>>> 0-888888.00000      0-888888.00000      0-888888.00000
>>>>> 0-888888.00000      0-888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    220.00000
>>>>> 0-888888.00000-888888     11.80000      0    274.00000      0
>>>>> 11.77126      0     -0.82313
>>>>> 0-888888.00000-888888-888888.00000-888888
>>>>>
>>>>>                  36.89000          -119.5800000010424
>>>>> MADIS
>>>>>                                          -888888.00000         1
0
>>>>> 0         0         0         T         F         F   -888888
-888888
>>>>> 20120207230900-888888.00000-888888-888888.00000-888888-
888888.00000
>>>>> 0-888888.00000      0-888888.00000
>>>>> 0-888888.00000-888888-888888.00000      0-888888.00000
>>>>> 0-888888.00000      0-888888.00000      0-888888.00000
>>>>> 0-888888.00000      0-888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    219.70000
>>>>> 0-888888.00000-888888     11.80000      0    274.00000      0
>>>>> 11.77125      0     -0.82313
>>>>> 0-888888.00000-888888-888888.00000-888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical
>>>>> but are 0.01 off of each other.  Perhaps this is considered
close enough
>>>>> by MET.  For the first observation u,v is (11.77126,-0.82313)
and for the
>>>>> second observation u,v is (11.77125,-0.82313).  So the V's match
but the
>>>>> U's differ by 0.00001 m/s, presumably due to rounding
issues/machine
>>>>> precision.  So my guess is that these observations are the
source of the
>>>>> problem.  Assuming that this is correct, the next question is
how to fix
>>>>> this.  A few possibilities come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the
>>>>> lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type
>>>>> differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components
>>>>> together, or at least U/V together.  In other words, for an ob
to be
>>>>> marked duplicate require that all observed fields match or at
least
>>>>> require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark
>>>>> as duplicate to MET help to see what they think.  For a quick
fix you
>>>>> could go and edit one of the two observations so that the U's
match,
>>>>> since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:46:44 2014

John -

I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:

DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
DEBUG 1: Climatology File: none
DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
_23:00:00.0000.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for TMP/P225-100.
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 32
 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 41
 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/P225-100" in GRIB record 50
 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB records
matching VarInfo "TMP/P225-100" in GRIB f
ile "/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-
02-07_23:00:00".
DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
ERROR  :
ERROR  : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
on_list.txt".
ERROR  :




________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:24 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

OK, I received the files and ran them through Point-Stat.  And I was
able to get matched pair output.  Upon closer inspection of my log
file versus yours, I see the problem.  When I run Point-Stat, I
see the following line included in the log file:

   DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
   DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"

But in the log file you sent me, I see this instead:

   DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"

The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to include the full path for "station_list.txt" in the
Point-Stat config file.  I believe that will solve your problem.

The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently.  You listed the names
of the stations directly in the config file itself.  Each station
name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair.  After we added the
matched pair (MPR) output line type, this type of station id
masking was no longer needed.  So we modified the logic.  In METv4.1,
you list one or more strings in the "mask.sid" setting.  Point-Stat
checks each string to see if it's a file name.  If so, it
parses the group of station names from the file listed.  If not, it
treats the string as the name of a single station to be used.  That
enables us to maintain backward compatibility.

In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used.  But you don't have any observations
for a station named "station_list.txt".  Thus, it found 0 matched
pairs.

Point-Stat did tell us all the information in the log file, but it was
a pretty subtle point.

Please give that a shot and let me know how it goes.

Thanks,
John



On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>
> Thanks.
>
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 10:45 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK thanks.  I now have the config file and station list.  I just
need the forecast file and little_r file.
>
> Thanks,
> John
>
> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> OK. Thanks.
>>
>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>
>> You should have the log file and my configuration file already from
my previous email. Let me know if you didn't get it.
>>
>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>
>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>
>> I've attached the station_list.txt file to this email.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 9:43 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> Sure, go ahead and send the data:
>>     - forecast file, little_r file, Point-Stat config file, and the
"station_list.txt" masking file
>>
>> You can post it to our anonymous ftp site following these
instructions:
>>       http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no sta!
 t!
>   i!
>>    st!
>>>     ical results. I'm beginning to think that for some reason it's
not reading or performing the analysis with the station ids. The log
file  acknowledges the presence of and handling of the list as
follows:
>>>
>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>
>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>
>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Friday, March 07, 2014 11:57 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> Hello John,
>>>
>>> Hmmm, sounds like a tricky question.  Here is an excerpt from this
file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>
>>>         238       //  build a uniqueness test key
>>>         239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>         240             lat,         //  lat
>>>         241             lon,         //  lon
>>>         242             lvl,         //  level
>>>         243             elv).text(); //  elevation
>>>
>>> As you can see, the uniqueness key is built using 3 decimal places
for the latitude and longitude.  So a difference of 0.01 would not be
"close enough".
>>>
>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>
>>> In the point_stat config file, set:
>>> mask = {
>>>        grid = [];
>>>        poly = [];
>>>        sid  = [ "station_list.txt" ];
>>> };
>>>
>>> Where station_list.txt is an ASCII file containing something like:
>>> TEST_STATIONS
>>> 00010426
>>> 00010495
>>> 00010424
>>>
>>> That will restrict the analysis to only that group of station
id's.  You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>> narrow it down.
>>>
>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> met_help at ucar.edu
>>>
>>>
>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>            Queue: met_help
>>>>          Subject: Point_Stat error
>>>>            Owner: Nobody
>>>>       Requestors: john.w.raby2.civ at mail.mil
>>>>           Status: new
>>>>      Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>>
>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>
>>>> ERROR  :
>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>> ERROR
>>>>
>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>
>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>
>>>> Thanks.
>>>> ________________________________
>>>>
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:
>>>> DEBUG 3:
>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>
>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>>                 36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>       18734.89063    512  12192.00000      0    220.00000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>
>>>>                 36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>       18734.89063    512  12192.00000      0    219.70000      0-
888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>
>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>
>>>> I'm thinking we might want to send these two obs that MET appears
to mark as duplicate to MET help to see what they think.  For a quick
fix you could go and edit one of the two observations so that the U's
match, since a change of 0.00001 m/s is not meaningful.
>>>>
>>>> R/
>>>> John
>>>>
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>>
>>>> U.S. Army Research Laboratory
>>>>
>>>> White Sands Missile Range, NM 88002
>>>>
>>>> (575) 678-2004 DSN 258-2004
>>>>
>>>> FAX (575) 678-1230 DSN 258-1230
>>>>
>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Raby, John W CIV USARMY ARL (US)
>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>> Subject: Point_Stat log file
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
From: John Halley Gotway
Time: Thu Mar 27 13:50:23 2014

John,

Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.

The ~ is something interpreted by the shell.  You're passing this to
the MET Point-Stat tool, not the shell.  That character is the source
of the error you're seeing.

John

On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR  :
> ERROR  : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR  :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat.  And I was
able to get matched pair output.  Upon closer inspection of my log
file versus yours, I see the problem.  When I run Point-Stat, I
> see the following line included in the log file:
>
>     DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
>     DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
>     DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file.  I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently.  You listed the names
of the stations directly in the config file itself.  Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair.  After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed.  So we modified the logic.  In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name.  If so, it
> parses the group of station names from the file listed.  If not, it
treats the string as the name of a single station to be used.  That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used.  But you don't have any observations
> for a station named "station_list.txt".  Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks.  I now have the config file and station list.  I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>>      - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>>        http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
 a!
>   t!
>>    i!
>>>     st!
>>>>      ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file  acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question.  Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>>          238       //  build a uniqueness test key
>>>>          239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>>          240             lat,         //  lat
>>>>          241             lon,         //  lon
>>>>          242             lvl,         //  level
>>>>          243             elv).text(); //  elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude.  So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>>         grid = [];
>>>>         poly = [];
>>>>         sid  = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's.  You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>>             Queue: met_help
>>>>>           Subject: Point_Stat error
>>>>>             Owner: Nobody
>>>>>        Requestors: john.w.raby2.civ at mail.mil
>>>>>            Status: new
>>>>>       Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR  :
>>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>>>                  36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    220.00000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>>                  36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    219.70000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think.  For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 13:53:33 2014

Ok. thanks for the quick response. I'll try that.

John

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:50 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.

The ~ is something interpreted by the shell.  You're passing this to
the MET Point-Stat tool, not the shell.  That character is the source
of the error you're seeing.

John

On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR  :
> ERROR  : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR  :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat.  And I was
able to get matched pair output.  Upon closer inspection of my log
file versus yours, I see the problem.  When I run Point-Stat, I
> see the following line included in the log file:
>
>     DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
>     DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
>     DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file.  I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently.  You listed the names
of the stations directly in the config file itself.  Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair.  After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed.  So we modified the logic.  In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name.  If so, it
> parses the group of station names from the file listed.  If not, it
treats the string as the name of a single station to be used.  That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used.  But you don't have any observations
> for a station named "station_list.txt".  Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks.  I now have the config file and station list.  I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>>      - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>>        http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
 a!
>   t!
>>    i!
>>>     st!
>>>>      ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file  acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question.  Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>>          238       //  build a uniqueness test key
>>>>          239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>>          240             lat,         //  lat
>>>>          241             lon,         //  lon
>>>>          242             lvl,         //  level
>>>>          243             elv).text(); //  elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude.  So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>>         grid = [];
>>>>         poly = [];
>>>>         sid  = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's.  You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>>             Queue: met_help
>>>>>           Subject: Point_Stat error
>>>>>             Owner: Nobody
>>>>>        Requestors: john.w.raby2.civ at mail.mil
>>>>>            Status: new
>>>>>       Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR  :
>>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>>>                  36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    220.00000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>>                  36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    219.70000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think.  For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #65756] Point_Stat error
From: Raby, John W USA CIV
Time: Thu Mar 27 14:03:41 2014

John -

You are correct and that fixed the problem! Now I get the same
response you did when it reads the station list:

DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"/home/jraby/MET_PointStat/station_list.txt"
DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file
"/home/jraby/MET_PointStat/station_list.txt"

I have good output files now too.

Thanks for all the help.

I'll be testing his with the IDs which may have caused the U and V
problems now to see if  can track that down.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, March 27, 2014 1:50 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error

John,

Don't use the tilda in the full path - list it as
"/home/jraby/MET_PointStat/station_list.txt" instead.

The ~ is something interpreted by the shell.  You're passing this to
the MET Point-Stat tool, not the shell.  That character is the source
of the error you're seeing.

John

On 03/27/2014 01:46 PM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>
> John -
>
> I used the exact same station_list.txt file and provided the path to
it in the config file. Now it sees it, but thinks there is white space
in the IDs, but I can't find it. See the following error msg:
>
> DEBUG 1: Forecast File:
/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
/home/jraby/MET_obs/little_r/20120207_Reen/qc_obs_used_earth_relative.d01.2012-
02-07
> _23:00:00.0000.nc
> DEBUG 2:
> DEBUG 2:
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for TMP/P225-100.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 32
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 41
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for VarInfo "TMP/P225-100" in GRIB record 50
>   of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records matching VarInfo "TMP/P225-100" in GRIB f
> ile
"/home/jraby/MET_WRFpostprd/20120207_Reen/131127A_WRFPRS_d01_2012-02-
07_23:00:00".
> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"~/MET_PointStat/station_list.txt"
> ERROR  :
> ERROR  : parse_sid_mask() -> masking station ID name can't contain
whitespace or slashes "~/MET_PointStat/stati
> on_list.txt".
> ERROR  :
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, March 27, 2014 1:24 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>
> John,
>
> OK, I received the files and ran them through Point-Stat.  And I was
able to get matched pair output.  Upon closer inspection of my log
file versus yours, I see the problem.  When I run Point-Stat, I
> see the following line included in the log file:
>
>     DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"station_list.txt"
>     DEBUG 4: parse_sid_mask() -> parsed 19 station ID's for the
"TEST_STATIONS" mask from file "station_list.txt"
>
> But in the log file you sent me, I see this instead:
>
>     DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>
> The short explanation is that point_stat can't find the file
"station_list.txt" in the directory it's being run.  The simple
solution is to include the full path for "station_list.txt" in the
> Point-Stat config file.  I believe that will solve your problem.
>
> The long explanation is this... in earlier versions of MET, the
station id setting functioned a bit differently.  You listed the names
of the stations directly in the config file itself.  Each station
> name was treated as it's own masking region - so we were basically
computing statistics over a single matched pair.  After we added the
matched pair (MPR) output line type, this type of station id
> masking was no longer needed.  So we modified the logic.  In
METv4.1, you list one or more strings in the "mask.sid" setting.
Point-Stat checks each string to see if it's a file name.  If so, it
> parses the group of station names from the file listed.  If not, it
treats the string as the name of a single station to be used.  That
enables us to maintain backward compatibility.
>
> In your case, you listed a file but didn't list the full path.  When
Point-Stat couldn't find that file, it assumed that was the name of
the station to be used.  But you don't have any observations
> for a station named "station_list.txt".  Thus, it found 0 matched
pairs.
>
> Point-Stat did tell us all the information in the log file, but it
was a pretty subtle point.
>
> Please give that a shot and let me know how it goes.
>
> Thanks,
> John
>
>
>
> On 03/27/2014 12:17 PM, Raby, John W USA CIV via RT wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>
>> John -
>>
>> Just in case, please delete the email you receive from the file
exchange system after you download the files.
>>
>> Thanks.
>>
>> John
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Thursday, March 27, 2014 10:45 AM
>> To: Raby, John W CIV USARMY ARL (US)
>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>
>> John,
>>
>> OK thanks.  I now have the config file and station list.  I just
need the forecast file and little_r file.
>>
>> Thanks,
>> John
>>
>> On 03/27/2014 09:59 AM, Raby, John W USA CIV via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>
>>> John -
>>>
>>> OK. Thanks.
>>>
>>> I've had trouble lately with the WSMR firewall and may not be able
to send you the data via anonymous ftp, so I will give it a try and
let you know how it goes. Alternatively I can send the little_r file
by email since it's only 5.5 mb, but the forecast file at 15 mb may be
too big for email.
>>>
>>> You should have the log file and my configuration file already
from my previous email. Let me know if you didn't get it.
>>>
>>> The obs filename is: qc_obs_used_earth_relative.d01.2012-02-
07_23:00:00.0000
>>>
>>> The forecast filename is:  131127A_WRFPRS_d01_2012-02-07_23:00:00
>>>
>>> I've attached the station_list.txt file to this email.
>>>
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>> Sent: Thursday, March 27, 2014 9:43 AM
>>> To: Raby, John W CIV USARMY ARL (US)
>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>
>>> John,
>>>
>>> Sure, go ahead and send the data:
>>>      - forecast file, little_r file, Point-Stat config file, and
the "station_list.txt" masking file
>>>
>>> You can post it to our anonymous ftp site following these
instructions:
>>>        http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 03/27/2014 08:47 AM, Raby, John W USA CIV via RT wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>
>>>> John -
>>>>
>>>> I tried the approach you suggested by creating the station id
list, putting it in the same directory along with the config file and
then running Point-Stat. I changed the config file (attached) to match
your setting so it can read the station_list.txt file instead of a
poly file or the grid (FULL). I used the same 3 station ids we
identified as suspect per the email below. The log file showed that
Point-Stat ran without error, but the output files contain no
numerical results, only column headers. I experimented with adding to
and subtracting from the id list with known stations based on looking
at the NetCDF obs file for the same hour, but never produced output
with actual statistics, just headers. I removed those stations from
the list and entered a new set of surface obs (ADPSFC) which should
have produced matched pairs at the Z2/Z10 levels, but no change. The
log file (attached) shows no matched pairs in every case. The output
(CNT file) shows the headers, but no st!
 a!
>   t!
>>    i!
>>>     st!
>>>>      ical results. I'm beginning to think that for some reason
it's not reading or performing the analysis with the station ids. The
log file  acknowledges the presence of and handling of the list as
follows:
>>>>
>>>> DEBUG 4: parse_sid_mask() -> storing single station ID mask
"station_list.txt"
>>>>
>>>> So. I'm not sure where to go from here. I would like to pursue
learning how to use this functionality correctly in order to use it to
potentially solve the problem with the U and V data.
>>>>
>>>> Let me know if I can provide you the data. The forecast file is
15mb, and the little_r file is 6mb.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>> ________________________________________
>>>> From: John Halley Gotway via RT [met_help at ucar.edu]
>>>> Sent: Friday, March 07, 2014 11:57 AM
>>>> To: Raby, John W CIV USARMY ARL (US)
>>>> Subject: Re: [rt.rap.ucar.edu #65756] Point_Stat error
>>>>
>>>> Hello John,
>>>>
>>>> Hmmm, sounds like a tricky question.  Here is an excerpt from
this file METv4.1/src/libcode/vx_statistics/pair_base.cc:
>>>>
>>>>          238       //  build a uniqueness test key
>>>>          239       string sng_key =
str_format("%.3f:%.3f:%.2f:%.2f",
>>>>          240             lat,         //  lat
>>>>          241             lon,         //  lon
>>>>          242             lvl,         //  level
>>>>          243             elv).text(); //  elevation
>>>>
>>>> As you can see, the uniqueness key is built using 3 decimal
places for the latitude and longitude.  So a difference of 0.01 would
not be "close enough".
>>>>
>>>> If you'd like me to debug this some more, it would be ideal to
figure out the smallest subset of littl_r data that reproduces this
problem.  If there are just a couple of station id's that are
>>>> suspect, I'd suggest using the station id masking to narrow down
exactly which station(s) are causing the problem.
>>>>
>>>> In the point_stat config file, set:
>>>> mask = {
>>>>         grid = [];
>>>>         poly = [];
>>>>         sid  = [ "station_list.txt" ];
>>>> };
>>>>
>>>> Where station_list.txt is an ASCII file containing something
like:
>>>> TEST_STATIONS
>>>> 00010426
>>>> 00010495
>>>> 00010424
>>>>
>>>> That will restrict the analysis to only that group of station
id's.  You could check to see if just those 3 stations replicate the
error and then either add/subtract stations from that list to really
>>>> narrow it down.
>>>>
>>>> I could probably take a closer look at this next week, but I'd
need you to send me a sample little_r file, forecast file, and Point-
Stat configuration file to replicate the error.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>> met_help at ucar.edu
>>>>
>>>>
>>>> On 03/06/2014 02:12 PM, Raby, John W USA CIV via RT wrote:
>>>>>
>>>>> Thu Mar 06 14:12:55 2014: Request 65756 was acted upon.
>>>>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>>>>>             Queue: met_help
>>>>>           Subject: Point_Stat error
>>>>>             Owner: Nobody
>>>>>        Requestors: john.w.raby2.civ at mail.mil
>>>>>            Status: new
>>>>>       Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65756 >
>>>>>
>>>>>
>>>>> I am getting an error running Point-Stat with a little_r
observations file. Using MET V4.1, the log file shows the following
dialog:
>>>>>
>>>>> ERROR  :
>>>>> ERROR  : do_vl1l2() -> the number of UGRD pairs != the number of
VGRD pairs: 159 != 158
>>>>> ERROR
>>>>>
>>>>> I've attached the log file, but we feel that the information
provided below in the flags for duplicate data may shed some light on
the problem. We are curious if Point-Stat logging could provide a way
to quickly identify the station whose data is causing the error. We
performed searches in the llittle_r file to locate the observation
which is causing the problem, but it is not easy or always successful.
We have identified some things to consider which might alleviate the
problem in the dialog below.
>>>>>
>>>>> Do you have any suggestions on how to track down the offending
observation or perhaps change the way the duplicate_flag works to
avoid this problem?
>>>>>
>>>>> Thanks.
>>>>> ________________________________
>>>>>
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: 14.9425] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: 15.9748] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:
>>>>> DEBUG 3:
>>>>> DEBUG 3: Duplicate point observations for -unique setting:
>>>>> DEBUG 3:     [lat: 34.0000  lon: -118.3000  valid:
20120207_225800  lev: 187.38  elv: 12191.00  ob_val: -3.7256] -
Station IDs: 00010426, 00010426
>>>>> DEBUG 3:     [lat: 34.7600  lon: -118.6500  valid:
20120207_230700  lev: 187.17  elv: 12198.00  ob_val: -5.8143] -
Station IDs: 00010495, 00010495
>>>>> DEBUG 3:     [lat: 36.9000  lon: -119.5900  valid:
20120207_230900  lev: 187.35  elv: 12192.00  ob_val: -0.8231] -
Station IDs: 00010424, 00010424
>>>>>
>>>>> I think that the first lines are the U observations that it
considered duplicate for observation AIRCAR for P225-100, and the next
few lines are the same for the V observations.  So the third duplicate
in the V does not have a matching duplicate in the U.  Looking through
the little_r files, I don't find two observations that exactly match
that listing.  However, I did find two that were pretty close, and
perhaps close enough to be considered duplicate by MET?  So the two
observations are:
>>>>>                  36.90000          -119.5900000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    220.00000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77126      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>>                  36.89000          -119.5800000010424
MADIS                                   FM-96
-888888.00000         1         0         0         0         0
T         F         F   -888888   -888888      20120207230900-
888888.00000-888888-888888.00000-888888-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000-888888-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0-888888.00000      0-888888.00000      0-
888888.00000      0
>>>>>        18734.89063    512  12192.00000      0    219.70000
0-888888.00000-888888     11.80000      0    274.00000      0
11.77125      0     -0.82313      0-888888.00000-888888-888888.00000-
888888
>>>>>
>>>>> The latitudes and longitudes of the two observations are not
identical but are 0.01 off of each other.  Perhaps this is considered
close enough by MET.  For the first observation u,v is (11.77126,-
0.82313) and for the second observation u,v is (11.77125,-0.82313).
So the V's match but the U's differ by 0.00001 m/s, presumably due to
rounding issues/machine precision.  So my guess is that these
observations are the source of the problem.  Assuming that this is
correct, the next question is how to fix this.  A few possibilities
come to mind:
>>>>> 1) Change MET to insist that for observations to be duplicate
that the lat/lon match closer than 0.01 degrees (which is about 1km).
>>>>> 2) Change the duplicate checking to allow for machine precision
type differences between observations.
>>>>> 3) Change the duplicate checking to check all observation
components together, or at least U/V together.  In other words, for an
ob to be marked duplicate require that all observed fields match or at
least require that U/V match.
>>>>>
>>>>> I'm thinking we might want to send these two obs that MET
appears to mark as duplicate to MET help to see what they think.  For
a quick fix you could go and edit one of the two observations so that
the U's match, since a change of 0.00001 m/s is not meaningful.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>>
>>>>> U.S. Army Research Laboratory
>>>>>
>>>>> White Sands Missile Range, NM 88002
>>>>>
>>>>> (575) 678-2004 DSN 258-2004
>>>>>
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>>
>>>>> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Raby, John W CIV USARMY ARL (US)
>>>>> Sent: Wednesday, March 05, 2014 2:31 PM
>>>>> To: Reen, Brian P CIV USARMY ARL (US)
>>>>> Subject: Point_Stat log file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>



------------------------------------------------


More information about the Met_help mailing list