[Met_help] [rt.rap.ucar.edu #68016] History for FW: Bad SID Table (UNCLASSIFIED)

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I am using little_r formatted observation files from the MADIS source and
post-processed WRF output in Point-Stat (METV4.1) to generate MPR as well as
other line types.

My config file specifies "SINGLE" duplicate flag.

The MPR file contains Z2/TMP observations from AJMC1 (id) which are
duplicate in the sense that one id has two different locations (lat or long
or elev). I can trace the source of the duplicate obs back to the original
little_r file.

I also find in the MPR file a third instance (triple) of a report (Z2/TMP)
from AJMC1 which contains location info in a combination not found in the
duplicate reports for AJMC1 which are also contained in the little_r file.
The lat/long/elev info for this station seems to be a permutation of the
location data for the duplicate stations. See the following listings for
three different model lead times:

For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the following triple
for Z2/TMP obs:

AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268  290.92776  NA
(original dupe present in little_r file)
AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215  290.92776
NA(triple)
AJMC1   35.72780 -119.13250 NA         151.44901  288.47229  290.37222  NA
(original dupe present in little_r file)  

For 17 lead: same pattern is repeated

AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052   284.81668
NA
AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231   284.81668
NA (triple)
AJMC1    35.72780 -119.13250 NA         151.44901   285.73105   283.70557
NA

For 21 lead: same pattern is repeated

AJMC1    35.73420 -119.14890 NA         144.80000   282.80484   282.03888
NA
AJMC1    35.72780 -119.13250 NA         151.44901   282.74350   282.59445
NA
AJMC1    35.73420 -119.14890 NA         151.44901   282.80484   282.03888
NA (triple)

For 08 lead: exception  - missing one of the original dupes

AJMC1    35.73420 -119.14890 NA         144.80000   291.99427   292.03888
NA
AJMC1    35.73420 -119.14890 NA         151.44901   291.99427   292.03888
NA (triple)

It appears that Point-Stat is generating a fictitious ob (triple) whose
combination of lat/long/elev cannot be traced to any of the input obs in the
little_r files.

I've attached one of the MPR text files (02 lead time) to this email for
your inspection.

Could you determine why this triple ob shows up in the MPR file and verify
that this may or may not be a bug in the Point-Stat software?

Thanks.

R/
John
________________________________________
From: Smith, Jeffrey A CIV USARMY ARL (US)
Sent: Wednesday, July 02, 2014 1:40 PM
To: Reen, Brian P CIV USARMY ARL (US)
Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
Subject: RE: Bad SID Table (UNCLASSIFIED)

Classification: UNCLASSIFIED
Caveats: NONE

Thanks Brian.  John and I are looking into this, but its.. intriguing to say
the least.  In a quick scan of my duplicate table for the 02072012 case day
(attached), sheet Case_20120207_TMP_Dups for OBS_SID=AJMC1, I find triples
at FCS_LEAD = 02, 17 and 21.  For FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18,
20, 22, 23 and 24 I find just doubles.  For that station ID, I find a total
of 33 records.   In each of the triples, two of the OBS_VALID_END times are
the same and one is different.  It appears from this limited sample, the one
element of the triple you excluded was one of the two 'SAME' OBS_VALID_END
times.  Furthermore, it appears that OBS should have been either part of the
previous hour or the subsequent hour forecast (i.e., not within +/- 15
minutes of the top of the hour).

Jeffrey



-----Original Message-----
From: Reen, Brian P CIV USARMY ARL (US)
Sent: Wednesday, July 02, 2014 8:03 AM
To: Smith, Jeffrey A CIV USARMY ARL (US)
Cc: Raby, John W CIV USARMY ARL (US)
Subject: RE: Bad SID Table (UNCLASSIFIED)

Classification: UNCLASSIFIED
Caveats: NONE

Jeffrey,
Your choices on which obs to include looks good to me.

Following are some answers regarding the questions I could not answer
yesterday on the phone about how WRF deals with potentially duplicate obs.
It is most likely more detailed than you are looking for, but I thought it
would be good to have at least for reference sake.

The Obsgrid software deals with quality control and dealing with duplicate
obs.  Obsgrid is used both to create the obs files I gave to John Raby for
ingestion into MET and the obs files I use for data assimilation in WRF.
Therefore, the obs I provided to you should be the same as the obs that I
used for data assimilation in WRF.  When I describe how Obsgrid works, I'm
referring to the version that I use that I have modified from the standard
version.

The observation files use a quality control flag that is additive powers of
2.  For example, 2^5=32 means one thing and 2^8=256 means another thing, so
if the quality control flag is 288=256+32 we know that the condition
represented by 32 is true as is the condition represented by 256 is true.
The most serious flags are the largest flags.  I have Obsgrid set to not
output any obs with a QC flag of 32768 or greater, and WRF is hardcoded to
ignore obs with a QC flag of 30000 or greater, so any obs in the obs file I
gave should be being used by WRF (assuming that they fall within the WRF
domain). In theory, obs with QC flags between 30000 and 32767 could be
included in the obs file but rejected by WRF, but I'm doubtful that any obs
would have QC flags in that range.

In Obsgrid it determines obs are duplicates if the following is true:
1) The latitude and longitudes are both less than 0.01 degrees apart
2) The station ID and name match exactly (note that for MADIS data the
station ID is OBS_SID in your spreadsheet, but the name is always "MADIS")
3) The times of the observations are within 30 minutes of one another

If the obs are duplicate then it tries to merge the two obs and uses the
time of the ob closer to the time for which we are looking for obs (Obsgrid
is being used to create hourly files that contain all obs closest to that
hour so this means it will use the time of the ob closest to the current
hour) as the time of the ob.  For each variable in the ob it tries to pick
the best from the two obs.  If does so using the following:
1) Use the non-missing value if one and only one of the obs is missing
2) Use the ob with the lower QC flag
3) Use the overall ob (i.e., all variables and I believe all levels) with
more valid observations
4) [I don't think the following flags are ever set so I suspect these never
come into play] User the overall ob with fewer errors noted, fewer warnings,
and higher sequence number
5) All else being equal, use the ob that is closer to the target time for
which we are looking for an observation

Based on all this, it may appear that stations like AIDC1 should not be in
the obs file from both CA-Hydro and MesoWest since they have identical
lat/lons.  It looks like obs from AIDC1 are not in the ob file from both
sources at the same time.  At least for the February 7th case, all but one
of the AIDC1 obs is from CA-Hydro, and at the time of the MesoWest AIDC1 ob
there is no matching ob from CA-Hydro.  So perhaps, CA-Hydro for some reason
did not report an ob from AIDC1 at that time to MADIS or perhaps the
MesoWest version happened to be earlier in the file than the CA-Hydro
version at that time, and the observations being identical Obsgrid chose the
first ob.

Brian

-----Original Message-----
From: Smith, Jeffrey A CIV USARMY ARL (US)
Sent: Tuesday, July 01, 2014 2:45 PM
To: Reen, Brian P CIV USARMY ARL (US)
Cc: Raby, John W CIV USARMY ARL (US)
Subject: RE: Bad SID Table (UNCLASSIFIED)

Classification: UNCLASSIFIED
Caveats: NONE

Brian.

Once again, thank you for your time.  I appreciate your help.

>From your comments to the original spreadsheet, I added a column called
"INCLUDE" that is Boolean -  true to include that data, false if to ignore.
Universally, if it was a choice between CA-Hydro and another net, I choose
CA-Hydro because of the precision in the data.

WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We  have no idea
which, if either, is the correct data so absent better information, I think
it's better to ignore both.

WRK KIYK, I chose the NonFED one for the same reason as you recommended.

For the greyed out triple values, I'll ignore those and John and I will take
a crack at figuring out what is going on there with MET.

Regards,

Jeffrey



-----Original Message-----
From: Reen, Brian P CIV USARMY ARL (US)
Sent: Friday, June 27, 2014 11:47 AM
To: Smith, Jeffrey A CIV USARMY ARL (US)
Cc: Raby, John W CIV USARMY ARL (US)
Subject: RE: Bad SID Table (UNCLASSIFIED)

Classification: UNCLASSIFIED
Caveats: NONE


Jeffrey,
I've went through the spreadsheet and found that whenever there are three
different lat/lon/elevation triplets for a given station, I only find two of
the combinations in the observation files I'm looking at.  The third
combination, the one I do not have, has a lat/lon/elevation combination that
can be constructed from taking 2 of the fields from one of the other
triplets and the other field from the other triplet. For example, if we have
three listings for a station called YYY:
1. YYY lat=lat1, lon=lon1, elev=elev1
2. YYY lat=lat2, lon=lon2, elev=elev2
3. YYY lat=lat3, lon=lon3, elev=elev3
Then if lines 1 and 2 are the versions I have in my file while I am missing
version 3, lat3 equals either lat1 or lat2, lon3 equals either lon1 or lon2,
elev3 equals either elev1 or elev2 [and line 3 is not a duplicate of either
line 1 or line 2]. This suggests that some processing step may be merging
the station information in certain cases.  In light of this, in the
spreadsheet I grayed out the lines that did not seem to occur in the
observation files that I have.

I've added columns showing the source of the observation, according to
MADIS.  I also added columns showing the difference in latitude, longitude,
and elevation between the two versions of each observation location.  I
calculated an estimated distance between the two locations as well.  I
estimated the elevation at the station location according to the 1/3
arc-second NED dataset via http://ned.usgs.gov/epqs/ and compared this to
the elevation reported in the obs file to find an elevation "error".   Of
course we do not know what the true elevation is exactly at the observation
location so this is not really necessarily an error.  Finally I put a
summary column with some text regarding the comparison.

Most of the differences are differences between how CA-Hydro and MesoWest
report the lat/lon/elevation of stations. In general it is hard to know
which is correct.  Often CA-Hydro matches better with the terrain database
than MesoWest, but the number of times that CA-Hydro exactly matches the
terrain database makes me suspicious that they might be directly pulling it
out of the same terrain database.  Beside the CA-Hydro / MesoWest
discrepencies, the following were found:

For KIYK, the Inyokern Airport in California, OTHER-MTR and NonFedAWOS
report different locations (~1.2km) and elevations (~0.7 m).  Since
NonFedAWOS reports the values with more precision, perhaps it is more likely
to be right?; also, the location it reports is right off one of the runways
of the airport.

For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and elevation
(~6 m).  The elevations reported by the two networks only differ by 8m, but
the terrain database estimated elevation for the two locations differs by
~113 m, resulting in the CA-Hydro elevation matching the terrain database
estimated elevation much better (~0.01m difference compared to 121.08 m).
This suggests that CA-Hydro is better.

For WSPC1 and YRKC1, RAWS reports identical latitude/longitude, and an
elevation that differs by 0.9 m.  Although the elevation matches the terrain
database at this location pretty well, the location is far from those
reported by HADS (by about 110 km for WSPC1 and 156 km for YRKC1).  The HADS
website lists descriptions of the two sites ("Whispering Pines near
Middletown 6NW" and "Yorkville 1WNW") that are consistent with the HADS
locations but not the RAWS locations.    This suggests that the HADS values
are correct.  However, note that the HADS elevation value for WSPC1 is about
200 m lower than expected by the terrain database.


Brian





-----Original Message-----
From: Smith, Jeffrey A CIV USARMY ARL (US)
Sent: Thursday, June 26, 2014 3:51 PM
To: Reen, Brian P CIV USARMY ARL (US)
Cc: Raby, John W CIV USARMY ARL (US)
Subject: Bad SID Table (UNCLASSIFIED)

Classification: UNCLASSIFIED
Caveats: NONE

Brian.

Here are the SID's I was speaking to you about.  They are all SID's that
produced at least one multiple ob, in a least one case hour of the five case
days.  After John reprocessed the LA Basin data with the new switch, I ran
my scripts against the data to produce the table.

Jeffrey



Classification: UNCLASSIFIED
Caveats: NONE




Classification: UNCLASSIFIED
Caveats: NONE



Classification: UNCLASSIFIED
Caveats: NONE



Classification: UNCLASSIFIED
Caveats: NONE



Classification: UNCLASSIFIED
Caveats: NONE



Classification: UNCLASSIFIED
Caveats: NONE




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: FW: Bad SID Table (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Jul 04 11:53:16 2014

John,

I'm having a difficult time exactly following the logic here, but I'll
do
my best.  If you have more questions, please send me an input little_r
file, a sample forecast file, and your Point-Stat config file.
Running it
here through the debugger is the easiest way for me to trace through
the
logic.

The "single" logic in the code is handled by a map called
"map_single".  A
map is a data structure that takes an input "key" and maps it to a
"value".
 For example, you might have a map that stores an baseball team's home
state.  The "Rockies" key would map to a value of "Colorado", while
the
"Dodgers" key would map to a value of "California".

Keys for the single map are constructed using the observation's
latitude,
longitude, level, and elevation.  The value is stored as the station
id,
the valid time, and the observation value.  For each observation
Point-Stat
processes, it constructs that key and checks to see if we've already
encountered it.  If not, we add a new entry.  If so, we check to see
if
it's value should be updated.

But there's a wrinkle when we're verifying surface observations, such
as
2-m temperature.  This is described in the METv4.1 release notes in
the 5th
bullet from the bottom:

http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php

When verifying surface obs, we reset the level value to NA so that
it's not
used in the duplicate logic.

Looking at the MPR file you sent, for the AJMC1 station, for TMP/Z2, I
extracted the lat, lon, level, and elevation columns, and here's what
I see:

OBS_LAT OBS_LON OBS_LVL OBS_ELV

35.73420 -119.14890 NA 144.80000

35.73420 -119.14890 NA 151.44901

35.72780 -119.13250 NA 151.44901
Stringing these values together to create the "single" key, you can
see
that all the keys are unique.  So the current logic won't flag these
as
being duplicates.

Does that help clarify?  I suspect the note included in the METv4.1
release
note helps explain the question you had.

Hope that helps and have a happy 4th.

Thanks,
John


On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: FW: Bad SID Table (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am using little_r formatted observation files from the MADIS
source and
> post-processed WRF output in Point-Stat (METV4.1) to generate MPR as
well
> as
> other line types.
>
> My config file specifies "SINGLE" duplicate flag.
>
> The MPR file contains Z2/TMP observations from AJMC1 (id) which are
> duplicate in the sense that one id has two different locations (lat
or long
> or elev). I can trace the source of the duplicate obs back to the
original
> little_r file.
>
> I also find in the MPR file a third instance (triple) of a report
(Z2/TMP)
> from AJMC1 which contains location info in a combination not found
in the
> duplicate reports for AJMC1 which are also contained in the little_r
file.
> The lat/long/elev info for this station seems to be a permutation of
the
> location data for the duplicate stations. See the following listings
for
> three different model lead times:
>
> For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the following
triple
> for Z2/TMP obs:
>
> AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776  NA
> (original dupe present in little_r file)
> AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> NA(triple)
> AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222  NA
> (original dupe present in little_r file)
>
> For 17 lead: same pattern is repeated
>
> AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
284.81668
> NA
> AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
284.81668
> NA (triple)
> AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
283.70557
> NA
>
> For 21 lead: same pattern is repeated
>
> AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
282.03888
> NA
> AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
282.59445
> NA
> AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
282.03888
> NA (triple)
>
> For 08 lead: exception  - missing one of the original dupes
>
> AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
292.03888
> NA
> AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
292.03888
> NA (triple)
>
> It appears that Point-Stat is generating a fictitious ob (triple)
whose
> combination of lat/long/elev cannot be traced to any of the input
obs in
> the
> little_r files.
>
> I've attached one of the MPR text files (02 lead time) to this email
for
> your inspection.
>
> Could you determine why this triple ob shows up in the MPR file and
verify
> that this may or may not be a bug in the Point-Stat software?
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Wednesday, July 02, 2014 1:40 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Thanks Brian.  John and I are looking into this, but its..
intriguing to
> say
> the least.  In a quick scan of my duplicate table for the 02072012
case day
> (attached), sheet Case_20120207_TMP_Dups for OBS_SID=AJMC1, I find
triples
> at FCS_LEAD = 02, 17 and 21.  For FCST_LEAD= 01, 03, 04, 08, 11, 13,
15 18,
> 20, 22, 23 and 24 I find just doubles.  For that station ID, I find
a total
> of 33 records.   In each of the triples, two of the OBS_VALID_END
times are
> the same and one is different.  It appears from this limited sample,
the
> one
> element of the triple you excluded was one of the two 'SAME'
OBS_VALID_END
> times.  Furthermore, it appears that OBS should have been either
part of
> the
> previous hour or the subsequent hour forecast (i.e., not within +/-
15
> minutes of the top of the hour).
>
> Jeffrey
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY ARL (US)
> Sent: Wednesday, July 02, 2014 8:03 AM
> To: Smith, Jeffrey A CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Jeffrey,
> Your choices on which obs to include looks good to me.
>
> Following are some answers regarding the questions I could not
answer
> yesterday on the phone about how WRF deals with potentially
duplicate obs.
> It is most likely more detailed than you are looking for, but I
thought it
> would be good to have at least for reference sake.
>
> The Obsgrid software deals with quality control and dealing with
duplicate
> obs.  Obsgrid is used both to create the obs files I gave to John
Raby for
> ingestion into MET and the obs files I use for data assimilation in
WRF.
> Therefore, the obs I provided to you should be the same as the obs
that I
> used for data assimilation in WRF.  When I describe how Obsgrid
works, I'm
> referring to the version that I use that I have modified from the
standard
> version.
>
> The observation files use a quality control flag that is additive
powers of
> 2.  For example, 2^5=32 means one thing and 2^8=256 means another
thing, so
> if the quality control flag is 288=256+32 we know that the condition
> represented by 32 is true as is the condition represented by 256 is
true.
> The most serious flags are the largest flags.  I have Obsgrid set to
not
> output any obs with a QC flag of 32768 or greater, and WRF is
hardcoded to
> ignore obs with a QC flag of 30000 or greater, so any obs in the obs
file I
> gave should be being used by WRF (assuming that they fall within the
WRF
> domain). In theory, obs with QC flags between 30000 and 32767 could
be
> included in the obs file but rejected by WRF, but I'm doubtful that
any obs
> would have QC flags in that range.
>
> In Obsgrid it determines obs are duplicates if the following is
true:
> 1) The latitude and longitudes are both less than 0.01 degrees apart
> 2) The station ID and name match exactly (note that for MADIS data
the
> station ID is OBS_SID in your spreadsheet, but the name is always
"MADIS")
> 3) The times of the observations are within 30 minutes of one
another
>
> If the obs are duplicate then it tries to merge the two obs and uses
the
> time of the ob closer to the time for which we are looking for obs
(Obsgrid
> is being used to create hourly files that contain all obs closest to
that
> hour so this means it will use the time of the ob closest to the
current
> hour) as the time of the ob.  For each variable in the ob it tries
to pick
> the best from the two obs.  If does so using the following:
> 1) Use the non-missing value if one and only one of the obs is
missing
> 2) Use the ob with the lower QC flag
> 3) Use the overall ob (i.e., all variables and I believe all levels)
with
> more valid observations
> 4) [I don't think the following flags are ever set so I suspect
these never
> come into play] User the overall ob with fewer errors noted, fewer
> warnings,
> and higher sequence number
> 5) All else being equal, use the ob that is closer to the target
time for
> which we are looking for an observation
>
> Based on all this, it may appear that stations like AIDC1 should not
be in
> the obs file from both CA-Hydro and MesoWest since they have
identical
> lat/lons.  It looks like obs from AIDC1 are not in the ob file from
both
> sources at the same time.  At least for the February 7th case, all
but one
> of the AIDC1 obs is from CA-Hydro, and at the time of the MesoWest
AIDC1 ob
> there is no matching ob from CA-Hydro.  So perhaps, CA-Hydro for
some
> reason
> did not report an ob from AIDC1 at that time to MADIS or perhaps the
> MesoWest version happened to be earlier in the file than the CA-
Hydro
> version at that time, and the observations being identical Obsgrid
chose
> the
> first ob.
>
> Brian
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Tuesday, July 01, 2014 2:45 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Brian.
>
> Once again, thank you for your time.  I appreciate your help.
>
> From your comments to the original spreadsheet, I added a column
called
> "INCLUDE" that is Boolean -  true to include that data, false if to
ignore.
> Universally, if it was a choice between CA-Hydro and another net, I
choose
> CA-Hydro because of the precision in the data.
>
> WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We  have
no
> idea
> which, if either, is the correct data so absent better information,
I think
> it's better to ignore both.
>
> WRK KIYK, I chose the NonFED one for the same reason as you
recommended.
>
> For the greyed out triple values, I'll ignore those and John and I
will
> take
> a crack at figuring out what is going on there with MET.
>
> Regards,
>
> Jeffrey
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY ARL (US)
> Sent: Friday, June 27, 2014 11:47 AM
> To: Smith, Jeffrey A CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
> Jeffrey,
> I've went through the spreadsheet and found that whenever there are
three
> different lat/lon/elevation triplets for a given station, I only
find two
> of
> the combinations in the observation files I'm looking at.  The third
> combination, the one I do not have, has a lat/lon/elevation
combination
> that
> can be constructed from taking 2 of the fields from one of the other
> triplets and the other field from the other triplet. For example, if
we
> have
> three listings for a station called YYY:
> 1. YYY lat=lat1, lon=lon1, elev=elev1
> 2. YYY lat=lat2, lon=lon2, elev=elev2
> 3. YYY lat=lat3, lon=lon3, elev=elev3
> Then if lines 1 and 2 are the versions I have in my file while I am
missing
> version 3, lat3 equals either lat1 or lat2, lon3 equals either lon1
or
> lon2,
> elev3 equals either elev1 or elev2 [and line 3 is not a duplicate of
either
> line 1 or line 2]. This suggests that some processing step may be
merging
> the station information in certain cases.  In light of this, in the
> spreadsheet I grayed out the lines that did not seem to occur in the
> observation files that I have.
>
> I've added columns showing the source of the observation, according
to
> MADIS.  I also added columns showing the difference in latitude,
longitude,
> and elevation between the two versions of each observation location.
I
> calculated an estimated distance between the two locations as well.
I
> estimated the elevation at the station location according to the 1/3
> arc-second NED dataset via http://ned.usgs.gov/epqs/ and compared
this to
> the elevation reported in the obs file to find an elevation "error".
Of
> course we do not know what the true elevation is exactly at the
observation
> location so this is not really necessarily an error.  Finally I put
a
> summary column with some text regarding the comparison.
>
> Most of the differences are differences between how CA-Hydro and
MesoWest
> report the lat/lon/elevation of stations. In general it is hard to
know
> which is correct.  Often CA-Hydro matches better with the terrain
database
> than MesoWest, but the number of times that CA-Hydro exactly matches
the
> terrain database makes me suspicious that they might be directly
pulling it
> out of the same terrain database.  Beside the CA-Hydro / MesoWest
> discrepencies, the following were found:
>
> For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> report different locations (~1.2km) and elevations (~0.7 m).  Since
> NonFedAWOS reports the values with more precision, perhaps it is
more
> likely
> to be right?; also, the location it reports is right off one of the
runways
> of the airport.
>
> For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and
> elevation
> (~6 m).  The elevations reported by the two networks only differ by
8m, but
> the terrain database estimated elevation for the two locations
differs by
> ~113 m, resulting in the CA-Hydro elevation matching the terrain
database
> estimated elevation much better (~0.01m difference compared to
121.08 m).
> This suggests that CA-Hydro is better.
>
> For WSPC1 and YRKC1, RAWS reports identical latitude/longitude, and
an
> elevation that differs by 0.9 m.  Although the elevation matches the
> terrain
> database at this location pretty well, the location is far from
those
> reported by HADS (by about 110 km for WSPC1 and 156 km for YRKC1).
The
> HADS
> website lists descriptions of the two sites ("Whispering Pines near
> Middletown 6NW" and "Yorkville 1WNW") that are consistent with the
HADS
> locations but not the RAWS locations.    This suggests that the HADS
values
> are correct.  However, note that the HADS elevation value for WSPC1
is
> about
> 200 m lower than expected by the terrain database.
>
>
> Brian
>
>
>
>
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Thursday, June 26, 2014 3:51 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Brian.
>
> Here are the SID's I was speaking to you about.  They are all SID's
that
> produced at least one multiple ob, in a least one case hour of the
five
> case
> days.  After John reprocessed the LA Basin data with the new switch,
I ran
> my scripts against the data to produce the table.
>
> Jeffrey
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jul 08 13:51:30 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks. I will send you the data you request, but, from my last
attempt to
transfer large files (15mb), ftp is blocked locally (unless you have a
sftp
server you can point me to) and I will have to resort to using the ARL
SAFE
file exchange system. The problem is that it sends you the email with
login
info to provide you secure access for downloading the files, but using
the
MET_HELP address potentially publishes this info to the open internet.
Do you
have an alternate email address I could use to facilitate this
transfer? I
have your address from UCAR if that would work for you.

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, July 04, 2014 11:53 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)

John,

I'm having a difficult time exactly following the logic here, but I'll
do my
best.  If you have more questions, please send me an input little_r
file, a
sample forecast file, and your Point-Stat config file.  Running it
here
through the debugger is the easiest way for me to trace through the
logic.

The "single" logic in the code is handled by a map called
"map_single".  A map
is a data structure that takes an input "key" and maps it to a
"value".
 For example, you might have a map that stores an baseball team's home
state.
The "Rockies" key would map to a value of "Colorado", while the
"Dodgers" key
would map to a value of "California".

Keys for the single map are constructed using the observation's
latitude,
longitude, level, and elevation.  The value is stored as the station
id, the
valid time, and the observation value.  For each observation Point-
Stat
processes, it constructs that key and checks to see if we've already
encountered it.  If not, we add a new entry.  If so, we check to see
if it's
value should be updated.

But there's a wrinkle when we're verifying surface observations, such
as 2-m
temperature.  This is described in the METv4.1 release notes in the
5th bullet
from the bottom:

http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php

When verifying surface obs, we reset the level value to NA so that
it's not
used in the duplicate logic.

Looking at the MPR file you sent, for the AJMC1 station, for TMP/Z2, I
extracted the lat, lon, level, and elevation columns, and here's what
I see:

OBS_LAT OBS_LON OBS_LVL OBS_ELV

35.73420 -119.14890 NA 144.80000

35.73420 -119.14890 NA 151.44901

35.72780 -119.13250 NA 151.44901
Stringing these values together to create the "single" key, you can
see that
all the keys are unique.  So the current logic won't flag these as
being
duplicates.

Does that help clarify?  I suspect the note included in the METv4.1
release
note helps explain the question you had.

Hope that helps and have a happy 4th.

Thanks,
John


On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: FW: Bad SID Table (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am using little_r formatted observation files from the MADIS
source
> and post-processed WRF output in Point-Stat (METV4.1) to generate
MPR
> as well as other line types.
>
> My config file specifies "SINGLE" duplicate flag.
>
> The MPR file contains Z2/TMP observations from AJMC1 (id) which are
> duplicate in the sense that one id has two different locations (lat
or
> long or elev). I can trace the source of the duplicate obs back to
the
> original little_r file.
>
> I also find in the MPR file a third instance (triple) of a report
> (Z2/TMP) from AJMC1 which contains location info in a combination
not
> found in the duplicate reports for AJMC1 which are also contained in
the
> little_r file.
> The lat/long/elev info for this station seems to be a permutation of
> the location data for the duplicate stations. See the following
> listings for three different model lead times:
>
> For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the following
> triple for Z2/TMP obs:
>
> AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776  NA
> (original dupe present in little_r file)
> AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> NA(triple)
> AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222  NA
> (original dupe present in little_r file)
>
> For 17 lead: same pattern is repeated
>
> AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
284.81668
> NA
> AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
284.81668
> NA (triple)
> AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
283.70557
> NA
>
> For 21 lead: same pattern is repeated
>
> AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
282.03888
> NA
> AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
282.59445
> NA
> AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
282.03888
> NA (triple)
>
> For 08 lead: exception  - missing one of the original dupes
>
> AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
292.03888
> NA
> AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
292.03888
> NA (triple)
>
> It appears that Point-Stat is generating a fictitious ob (triple)
> whose combination of lat/long/elev cannot be traced to any of the
> input obs in the little_r files.
>
> I've attached one of the MPR text files (02 lead time) to this email
> for your inspection.
>
> Could you determine why this triple ob shows up in the MPR file and
> verify that this may or may not be a bug in the Point-Stat software?
>
> Thanks.
>
> R/
> John
> ________________________________________
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Wednesday, July 02, 2014 1:40 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Thanks Brian.  John and I are looking into this, but its..
intriguing
> to say the least.  In a quick scan of my duplicate table for the
> 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I find
> just doubles.  For that station ID, I find a total
> of 33 records.   In each of the triples, two of the OBS_VALID_END
times are
> the same and one is different.  It appears from this limited sample,
> the one element of the triple you excluded was one of the two 'SAME'
> OBS_VALID_END times.  Furthermore, it appears that OBS should have
> been either part of the previous hour or the subsequent hour
forecast
> (i.e., not within +/- 15 minutes of the top of the hour).
>
> Jeffrey
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY ARL (US)
> Sent: Wednesday, July 02, 2014 8:03 AM
> To: Smith, Jeffrey A CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Jeffrey,
> Your choices on which obs to include looks good to me.
>
> Following are some answers regarding the questions I could not
answer
> yesterday on the phone about how WRF deals with potentially
duplicate obs.
> It is most likely more detailed than you are looking for, but I
> thought it would be good to have at least for reference sake.
>
> The Obsgrid software deals with quality control and dealing with
> duplicate obs.  Obsgrid is used both to create the obs files I gave
to
> John Raby for ingestion into MET and the obs files I use for data
> assimilation in WRF.
> Therefore, the obs I provided to you should be the same as the obs
> that I used for data assimilation in WRF.  When I describe how
Obsgrid
> works, I'm referring to the version that I use that I have modified
> from the standard version.
>
> The observation files use a quality control flag that is additive
> powers of 2.  For example, 2^5=32 means one thing and 2^8=256 means
> another thing, so if the quality control flag is 288=256+32 we know
> that the condition represented by 32 is true as is the condition
represented
> by 256 is true.
> The most serious flags are the largest flags.  I have Obsgrid set to
> not output any obs with a QC flag of 32768 or greater, and WRF is
> hardcoded to ignore obs with a QC flag of 30000 or greater, so any
obs
> in the obs file I gave should be being used by WRF (assuming that
they
> fall within the WRF domain). In theory, obs with QC flags between
> 30000 and 32767 could be included in the obs file but rejected by
WRF,
> but I'm doubtful that any obs would have QC flags in that range.
>
> In Obsgrid it determines obs are duplicates if the following is
true:
> 1) The latitude and longitudes are both less than 0.01 degrees apart
> 2) The station ID and name match exactly (note that for MADIS data
the
> station ID is OBS_SID in your spreadsheet, but the name is always
> "MADIS")
> 3) The times of the observations are within 30 minutes of one
another
>
> If the obs are duplicate then it tries to merge the two obs and uses
> the time of the ob closer to the time for which we are looking for
obs
> (Obsgrid is being used to create hourly files that contain all obs
> closest to that hour so this means it will use the time of the ob
> closest to the current
> hour) as the time of the ob.  For each variable in the ob it tries
to
> pick the best from the two obs.  If does so using the following:
> 1) Use the non-missing value if one and only one of the obs is
missing
> 2) Use the ob with the lower QC flag
> 3) Use the overall ob (i.e., all variables and I believe all levels)
> with more valid observations
> 4) [I don't think the following flags are ever set so I suspect
these
> never come into play] User the overall ob with fewer errors noted,
> fewer warnings, and higher sequence number
> 5) All else being equal, use the ob that is closer to the target
time
> for which we are looking for an observation
>
> Based on all this, it may appear that stations like AIDC1 should not
> be in the obs file from both CA-Hydro and MesoWest since they have
> identical lat/lons.  It looks like obs from AIDC1 are not in the ob
> file from both sources at the same time.  At least for the February
> 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at the
> time of the MesoWest AIDC1 ob there is no matching ob from CA-Hydro.
> So perhaps, CA-Hydro for some reason did not report an ob from AIDC1
> at that time to MADIS or perhaps the MesoWest version happened to be
> earlier in the file than the CA-Hydro version at that time, and the
> observations being identical Obsgrid chose the first ob.
>
> Brian
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Tuesday, July 01, 2014 2:45 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Brian.
>
> Once again, thank you for your time.  I appreciate your help.
>
> From your comments to the original spreadsheet, I added a column
> called "INCLUDE" that is Boolean -  true to include that data, false
if to
> ignore.
> Universally, if it was a choice between CA-Hydro and another net, I
> choose CA-Hydro because of the precision in the data.
>
> WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We  have
> no idea which, if either, is the correct data so absent better
> information, I think it's better to ignore both.
>
> WRK KIYK, I chose the NonFED one for the same reason as you
recommended.
>
> For the greyed out triple values, I'll ignore those and John and I
> will take a crack at figuring out what is going on there with MET.
>
> Regards,
>
> Jeffrey
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY ARL (US)
> Sent: Friday, June 27, 2014 11:47 AM
> To: Smith, Jeffrey A CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: RE: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
> Jeffrey,
> I've went through the spreadsheet and found that whenever there are
> three different lat/lon/elevation triplets for a given station, I
only
> find two of the combinations in the observation files I'm looking
at.
> The third combination, the one I do not have, has a
lat/lon/elevation
> combination that can be constructed from taking 2 of the fields from
> one of the other triplets and the other field from the other
triplet.
> For example, if we have three listings for a station called YYY:
> 1. YYY lat=lat1, lon=lon1, elev=elev1
> 2. YYY lat=lat2, lon=lon2, elev=elev2
> 3. YYY lat=lat3, lon=lon3, elev=elev3
> Then if lines 1 and 2 are the versions I have in my file while I am
> missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> lon1 or lon2,
> elev3 equals either elev1 or elev2 [and line 3 is not a duplicate of
> either line 1 or line 2]. This suggests that some processing step
may
> be merging the station information in certain cases.  In light of
> this, in the spreadsheet I grayed out the lines that did not seem to
> occur in the observation files that I have.
>
> I've added columns showing the source of the observation, according
to
> MADIS.  I also added columns showing the difference in latitude,
> longitude, and elevation between the two versions of each
observation
> location.  I calculated an estimated distance between the two
> locations as well.  I estimated the elevation at the station
location
> according to the 1/3 arc-second NED dataset via
http://ned.usgs.gov/epqs/
> and compared this to
> the elevation reported in the obs file to find an elevation "error".
Of
> course we do not know what the true elevation is exactly at the
> observation location so this is not really necessarily an error.
> Finally I put a summary column with some text regarding the
comparison.
>
> Most of the differences are differences between how CA-Hydro and
> MesoWest report the lat/lon/elevation of stations. In general it is
> hard to know which is correct.  Often CA-Hydro matches better with
the
> terrain database than MesoWest, but the number of times that CA-
Hydro
> exactly matches the terrain database makes me suspicious that they
> might be directly pulling it out of the same terrain database.
Beside
> the CA-Hydro / MesoWest discrepencies, the following were found:
>
> For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> report different locations (~1.2km) and elevations (~0.7 m).  Since
> NonFedAWOS reports the values with more precision, perhaps it is
more
> likely to be right?; also, the location it reports is right off one
of
> the runways of the airport.
>
> For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and
> elevation
> (~6 m).  The elevations reported by the two networks only differ by
> 8m, but the terrain database estimated elevation for the two
locations
> differs by
> ~113 m, resulting in the CA-Hydro elevation matching the terrain
> database estimated elevation much better (~0.01m difference compared
to
> 121.08 m).
> This suggests that CA-Hydro is better.
>
> For WSPC1 and YRKC1, RAWS reports identical latitude/longitude, and
an
> elevation that differs by 0.9 m.  Although the elevation matches the
> terrain database at this location pretty well, the location is far
> from those reported by HADS (by about 110 km for WSPC1 and 156 km
for
> YRKC1).  The HADS website lists descriptions of the two sites
> ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW") that
are
> consistent with the HADS
> locations but not the RAWS locations.    This suggests that the HADS
values
> are correct.  However, note that the HADS elevation value for WSPC1
is
> about
> 200 m lower than expected by the terrain database.
>
>
> Brian
>
>
>
>
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY ARL (US)
> Sent: Thursday, June 26, 2014 3:51 PM
> To: Reen, Brian P CIV USARMY ARL (US)
> Cc: Raby, John W CIV USARMY ARL (US)
> Subject: Bad SID Table (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Brian.
>
> Here are the SID's I was speaking to you about.  They are all SID's
> that produced at least one multiple ob, in a least one case hour of
> the five case days.  After John reprocessed the LA Basin data with
the
> new switch, I ran my scripts against the data to produce the table.
>
> Jeffrey
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Jul 08 14:13:17 2014

John,

Sure that would work, you can send it to "johnhg at ucar.edu".

Thanks,
John


On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks. I will send you the data you request, but, from my last
attempt to
> transfer large files (15mb), ftp is blocked locally (unless you have
a sftp
> server you can point me to) and I will have to resort to using the
ARL SAFE
> file exchange system. The problem is that it sends you the email
with login
> info to provide you secure access for downloading the files, but
using the
> MET_HELP address potentially publishes this info to the open
internet. Do
> you
> have an alternate email address I could use to facilitate this
transfer? I
> have your address from UCAR if that would work for you.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, July 04, 2014 11:53 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> I'm having a difficult time exactly following the logic here, but
I'll do
> my
> best.  If you have more questions, please send me an input little_r
file, a
> sample forecast file, and your Point-Stat config file.  Running it
here
> through the debugger is the easiest way for me to trace through the
logic.
>
> The "single" logic in the code is handled by a map called
"map_single".  A
> map
> is a data structure that takes an input "key" and maps it to a
"value".
>  For example, you might have a map that stores an baseball team's
home
> state.
> The "Rockies" key would map to a value of "Colorado", while the
"Dodgers"
> key
> would map to a value of "California".
>
> Keys for the single map are constructed using the observation's
latitude,
> longitude, level, and elevation.  The value is stored as the station
id,
> the
> valid time, and the observation value.  For each observation Point-
Stat
> processes, it constructs that key and checks to see if we've already
> encountered it.  If not, we add a new entry.  If so, we check to see
if
> it's
> value should be updated.
>
> But there's a wrinkle when we're verifying surface observations,
such as
> 2-m
> temperature.  This is described in the METv4.1 release notes in the
5th
> bullet
> from the bottom:
>
>
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
>
> When verifying surface obs, we reset the level value to NA so that
it's not
> used in the duplicate logic.
>
> Looking at the MPR file you sent, for the AJMC1 station, for TMP/Z2,
I
> extracted the lat, lon, level, and elevation columns, and here's
what I
> see:
>
> OBS_LAT OBS_LON OBS_LVL OBS_ELV
>
> 35.73420 -119.14890 NA 144.80000
>
> 35.73420 -119.14890 NA 151.44901
>
> 35.72780 -119.13250 NA 151.44901
> Stringing these values together to create the "single" key, you can
see
> that
> all the keys are unique.  So the current logic won't flag these as
being
> duplicates.
>
> Does that help clarify?  I suspect the note included in the METv4.1
release
> note helps explain the question you had.
>
> Hope that helps and have a happy 4th.
>
> Thanks,
> John
>
>
> On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I am using little_r formatted observation files from the MADIS
source
> > and post-processed WRF output in Point-Stat (METV4.1) to generate
MPR
> > as well as other line types.
> >
> > My config file specifies "SINGLE" duplicate flag.
> >
> > The MPR file contains Z2/TMP observations from AJMC1 (id) which
are
> > duplicate in the sense that one id has two different locations
(lat or
> > long or elev). I can trace the source of the duplicate obs back to
the
> > original little_r file.
> >
> > I also find in the MPR file a third instance (triple) of a report
> > (Z2/TMP) from AJMC1 which contains location info in a combination
not
> > found in the duplicate reports for AJMC1 which are also contained
in the
> > little_r file.
> > The lat/long/elev info for this station seems to be a permutation
of
> > the location data for the duplicate stations. See the following
> > listings for three different model lead times:
> >
> > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > triple for Z2/TMP obs:
> >
> > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776
>  NA
> > (original dupe present in little_r file)
> > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> > NA(triple)
> > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222
>  NA
> > (original dupe present in little_r file)
> >
> > For 17 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
284.81668
> > NA
> > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
284.81668
> > NA (triple)
> > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
283.70557
> > NA
> >
> > For 21 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
282.03888
> > NA
> > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
282.59445
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
282.03888
> > NA (triple)
> >
> > For 08 lead: exception  - missing one of the original dupes
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
292.03888
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
292.03888
> > NA (triple)
> >
> > It appears that Point-Stat is generating a fictitious ob (triple)
> > whose combination of lat/long/elev cannot be traced to any of the
> > input obs in the little_r files.
> >
> > I've attached one of the MPR text files (02 lead time) to this
email
> > for your inspection.
> >
> > Could you determine why this triple ob shows up in the MPR file
and
> > verify that this may or may not be a bug in the Point-Stat
software?
> >
> > Thanks.
> >
> > R/
> > John
> > ________________________________________
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 1:40 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > to say the least.  In a quick scan of my duplicate table for the
> > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I find
> > just doubles.  For that station ID, I find a total
> > of 33 records.   In each of the triples, two of the OBS_VALID_END
times
> are
> > the same and one is different.  It appears from this limited
sample,
> > the one element of the triple you excluded was one of the two
'SAME'
> > OBS_VALID_END times.  Furthermore, it appears that OBS should have
> > been either part of the previous hour or the subsequent hour
forecast
> > (i.e., not within +/- 15 minutes of the top of the hour).
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 8:03 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Jeffrey,
> > Your choices on which obs to include looks good to me.
> >
> > Following are some answers regarding the questions I could not
answer
> > yesterday on the phone about how WRF deals with potentially
duplicate
> obs.
> > It is most likely more detailed than you are looking for, but I
> > thought it would be good to have at least for reference sake.
> >
> > The Obsgrid software deals with quality control and dealing with
> > duplicate obs.  Obsgrid is used both to create the obs files I
gave to
> > John Raby for ingestion into MET and the obs files I use for data
> > assimilation in WRF.
> > Therefore, the obs I provided to you should be the same as the obs
> > that I used for data assimilation in WRF.  When I describe how
Obsgrid
> > works, I'm referring to the version that I use that I have
modified
> > from the standard version.
> >
> > The observation files use a quality control flag that is additive
> > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > another thing, so if the quality control flag is 288=256+32 we
know
> > that the condition represented by 32 is true as is the condition
> represented
> > by 256 is true.
> > The most serious flags are the largest flags.  I have Obsgrid set
to
> > not output any obs with a QC flag of 32768 or greater, and WRF is
> > hardcoded to ignore obs with a QC flag of 30000 or greater, so any
obs
> > in the obs file I gave should be being used by WRF (assuming that
they
> > fall within the WRF domain). In theory, obs with QC flags between
> > 30000 and 32767 could be included in the obs file but rejected by
WRF,
> > but I'm doubtful that any obs would have QC flags in that range.
> >
> > In Obsgrid it determines obs are duplicates if the following is
true:
> > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > 2) The station ID and name match exactly (note that for MADIS data
the
> > station ID is OBS_SID in your spreadsheet, but the name is always
> > "MADIS")
> > 3) The times of the observations are within 30 minutes of one
another
> >
> > If the obs are duplicate then it tries to merge the two obs and
uses
> > the time of the ob closer to the time for which we are looking for
obs
> > (Obsgrid is being used to create hourly files that contain all obs
> > closest to that hour so this means it will use the time of the ob
> > closest to the current
> > hour) as the time of the ob.  For each variable in the ob it tries
to
> > pick the best from the two obs.  If does so using the following:
> > 1) Use the non-missing value if one and only one of the obs is
missing
> > 2) Use the ob with the lower QC flag
> > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > with more valid observations
> > 4) [I don't think the following flags are ever set so I suspect
these
> > never come into play] User the overall ob with fewer errors noted,
> > fewer warnings, and higher sequence number
> > 5) All else being equal, use the ob that is closer to the target
time
> > for which we are looking for an observation
> >
> > Based on all this, it may appear that stations like AIDC1 should
not
> > be in the obs file from both CA-Hydro and MesoWest since they have
> > identical lat/lons.  It looks like obs from AIDC1 are not in the
ob
> > file from both sources at the same time.  At least for the
February
> > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at
the
> > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > at that time to MADIS or perhaps the MesoWest version happened to
be
> > earlier in the file than the CA-Hydro version at that time, and
the
> > observations being identical Obsgrid chose the first ob.
> >
> > Brian
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Tuesday, July 01, 2014 2:45 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Once again, thank you for your time.  I appreciate your help.
> >
> > From your comments to the original spreadsheet, I added a column
> > called "INCLUDE" that is Boolean -  true to include that data,
false if
> to
> > ignore.
> > Universally, if it was a choice between CA-Hydro and another net,
I
> > choose CA-Hydro because of the precision in the data.
> >
> > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > no idea which, if either, is the correct data so absent better
> > information, I think it's better to ignore both.
> >
> > WRK KIYK, I chose the NonFED one for the same reason as you
recommended.
> >
> > For the greyed out triple values, I'll ignore those and John and I
> > will take a crack at figuring out what is going on there with MET.
> >
> > Regards,
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Friday, June 27, 2014 11:47 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> > Jeffrey,
> > I've went through the spreadsheet and found that whenever there
are
> > three different lat/lon/elevation triplets for a given station, I
only
> > find two of the combinations in the observation files I'm looking
at.
> > The third combination, the one I do not have, has a
lat/lon/elevation
> > combination that can be constructed from taking 2 of the fields
from
> > one of the other triplets and the other field from the other
triplet.
> > For example, if we have three listings for a station called YYY:
> > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > Then if lines 1 and 2 are the versions I have in my file while I
am
> > missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> > lon1 or lon2,
> > elev3 equals either elev1 or elev2 [and line 3 is not a duplicate
of
> > either line 1 or line 2]. This suggests that some processing step
may
> > be merging the station information in certain cases.  In light of
> > this, in the spreadsheet I grayed out the lines that did not seem
to
> > occur in the observation files that I have.
> >
> > I've added columns showing the source of the observation,
according to
> > MADIS.  I also added columns showing the difference in latitude,
> > longitude, and elevation between the two versions of each
observation
> > location.  I calculated an estimated distance between the two
> > locations as well.  I estimated the elevation at the station
location
> > according to the 1/3 arc-second NED dataset via
> http://ned.usgs.gov/epqs/
> > and compared this to
> > the elevation reported in the obs file to find an elevation
"error".   Of
> > course we do not know what the true elevation is exactly at the
> > observation location so this is not really necessarily an error.
> > Finally I put a summary column with some text regarding the
comparison.
> >
> > Most of the differences are differences between how CA-Hydro and
> > MesoWest report the lat/lon/elevation of stations. In general it
is
> > hard to know which is correct.  Often CA-Hydro matches better with
the
> > terrain database than MesoWest, but the number of times that CA-
Hydro
> > exactly matches the terrain database makes me suspicious that they
> > might be directly pulling it out of the same terrain database.
Beside
> > the CA-Hydro / MesoWest discrepencies, the following were found:
> >
> > For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > NonFedAWOS reports the values with more precision, perhaps it is
more
> > likely to be right?; also, the location it reports is right off
one of
> > the runways of the airport.
> >
> > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and
> > elevation
> > (~6 m).  The elevations reported by the two networks only differ
by
> > 8m, but the terrain database estimated elevation for the two
locations
> > differs by
> > ~113 m, resulting in the CA-Hydro elevation matching the terrain
> > database estimated elevation much better (~0.01m difference
compared to
> > 121.08 m).
> > This suggests that CA-Hydro is better.
> >
> > For WSPC1 and YRKC1, RAWS reports identical latitude/longitude,
and an
> > elevation that differs by 0.9 m.  Although the elevation matches
the
> > terrain database at this location pretty well, the location is far
> > from those reported by HADS (by about 110 km for WSPC1 and 156 km
for
> > YRKC1).  The HADS website lists descriptions of the two sites
> > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW") that
are
> > consistent with the HADS
> > locations but not the RAWS locations.    This suggests that the
HADS
> values
> > are correct.  However, note that the HADS elevation value for
WSPC1 is
> > about
> > 200 m lower than expected by the terrain database.
> >
> >
> > Brian
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Thursday, June 26, 2014 3:51 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Here are the SID's I was speaking to you about.  They are all
SID's
> > that produced at least one multiple ob, in a least one case hour
of
> > the five case days.  After John reprocessed the LA Basin data with
the
> > new switch, I ran my scripts against the data to produce the
table.
> >
> > Jeffrey
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jul 08 14:27:51 2014

Classification: UNCLASSIFIED
Caveats: NONE

Thanks. I don't have access to the workstation where the data is at
this
moment due to some troubleshooting, but I'll send it as soon as I can
today.

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, July 08, 2014 2:13 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)

John,

Sure that would work, you can send it to "johnhg at ucar.edu".

Thanks,
John


On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks. I will send you the data you request, but, from my last
> attempt to transfer large files (15mb), ftp is blocked locally
(unless
> you have a sftp server you can point me to) and I will have to
resort
> to using the ARL SAFE file exchange system. The problem is that it
> sends you the email with login info to provide you secure access for
> downloading the files, but using the MET_HELP address potentially
> publishes this info to the open internet. Do you have an alternate
> email address I could use to facilitate this transfer? I have your
> address from UCAR if that would work for you.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, July 04, 2014 11:53 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> I'm having a difficult time exactly following the logic here, but
I'll
> do my best.  If you have more questions, please send me an input
> little_r file, a sample forecast file, and your Point-Stat config
> file.  Running it here through the debugger is the easiest way for
me
> to trace through the logic.
>
> The "single" logic in the code is handled by a map called
> "map_single".  A map is a data structure that takes an input "key"
and
> maps it to a "value".
>  For example, you might have a map that stores an baseball team's
home
> state.
> The "Rockies" key would map to a value of "Colorado", while the
"Dodgers"
> key
> would map to a value of "California".
>
> Keys for the single map are constructed using the observation's
> latitude, longitude, level, and elevation.  The value is stored as
the
> station id, the valid time, and the observation value.  For each
> observation Point-Stat processes, it constructs that key and checks
to
> see if we've already encountered it.  If not, we add a new entry.
If
> so, we check to see if it's value should be updated.
>
> But there's a wrinkle when we're verifying surface observations,
such
> as 2-m temperature.  This is described in the METv4.1 release notes
in
> the 5th bullet from the bottom:
>
>
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
>
> When verifying surface obs, we reset the level value to NA so that
it's not
> used in the duplicate logic.
>
> Looking at the MPR file you sent, for the AJMC1 station, for TMP/Z2,
I
> extracted the lat, lon, level, and elevation columns, and here's
what I
> see:
>
> OBS_LAT OBS_LON OBS_LVL OBS_ELV
>
> 35.73420 -119.14890 NA 144.80000
>
> 35.73420 -119.14890 NA 151.44901
>
> 35.72780 -119.13250 NA 151.44901
> Stringing these values together to create the "single" key, you can
see
> that
> all the keys are unique.  So the current logic won't flag these as
being
> duplicates.
>
> Does that help clarify?  I suspect the note included in the METv4.1
release
> note helps explain the question you had.
>
> Hope that helps and have a happy 4th.
>
> Thanks,
> John
>
>
> On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I am using little_r formatted observation files from the MADIS
source
> > and post-processed WRF output in Point-Stat (METV4.1) to generate
MPR
> > as well as other line types.
> >
> > My config file specifies "SINGLE" duplicate flag.
> >
> > The MPR file contains Z2/TMP observations from AJMC1 (id) which
are
> > duplicate in the sense that one id has two different locations
(lat or
> > long or elev). I can trace the source of the duplicate obs back to
the
> > original little_r file.
> >
> > I also find in the MPR file a third instance (triple) of a report
> > (Z2/TMP) from AJMC1 which contains location info in a combination
not
> > found in the duplicate reports for AJMC1 which are also contained
in the
> > little_r file.
> > The lat/long/elev info for this station seems to be a permutation
of
> > the location data for the duplicate stations. See the following
> > listings for three different model lead times:
> >
> > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > triple for Z2/TMP obs:
> >
> > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776
>  NA
> > (original dupe present in little_r file)
> > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> > NA(triple)
> > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222
>  NA
> > (original dupe present in little_r file)
> >
> > For 17 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
284.81668
> > NA
> > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
284.81668
> > NA (triple)
> > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
283.70557
> > NA
> >
> > For 21 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
282.03888
> > NA
> > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
282.59445
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
282.03888
> > NA (triple)
> >
> > For 08 lead: exception  - missing one of the original dupes
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
292.03888
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
292.03888
> > NA (triple)
> >
> > It appears that Point-Stat is generating a fictitious ob (triple)
> > whose combination of lat/long/elev cannot be traced to any of the
> > input obs in the little_r files.
> >
> > I've attached one of the MPR text files (02 lead time) to this
email
> > for your inspection.
> >
> > Could you determine why this triple ob shows up in the MPR file
and
> > verify that this may or may not be a bug in the Point-Stat
software?
> >
> > Thanks.
> >
> > R/
> > John
> > ________________________________________
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 1:40 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > to say the least.  In a quick scan of my duplicate table for the
> > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I find
> > just doubles.  For that station ID, I find a total
> > of 33 records.   In each of the triples, two of the OBS_VALID_END
times
> are
> > the same and one is different.  It appears from this limited
sample,
> > the one element of the triple you excluded was one of the two
'SAME'
> > OBS_VALID_END times.  Furthermore, it appears that OBS should have
> > been either part of the previous hour or the subsequent hour
forecast
> > (i.e., not within +/- 15 minutes of the top of the hour).
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 8:03 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Jeffrey,
> > Your choices on which obs to include looks good to me.
> >
> > Following are some answers regarding the questions I could not
answer
> > yesterday on the phone about how WRF deals with potentially
duplicate
> obs.
> > It is most likely more detailed than you are looking for, but I
> > thought it would be good to have at least for reference sake.
> >
> > The Obsgrid software deals with quality control and dealing with
> > duplicate obs.  Obsgrid is used both to create the obs files I
gave to
> > John Raby for ingestion into MET and the obs files I use for data
> > assimilation in WRF.
> > Therefore, the obs I provided to you should be the same as the obs
> > that I used for data assimilation in WRF.  When I describe how
Obsgrid
> > works, I'm referring to the version that I use that I have
modified
> > from the standard version.
> >
> > The observation files use a quality control flag that is additive
> > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > another thing, so if the quality control flag is 288=256+32 we
know
> > that the condition represented by 32 is true as is the condition
> represented
> > by 256 is true.
> > The most serious flags are the largest flags.  I have Obsgrid set
to
> > not output any obs with a QC flag of 32768 or greater, and WRF is
> > hardcoded to ignore obs with a QC flag of 30000 or greater, so any
obs
> > in the obs file I gave should be being used by WRF (assuming that
they
> > fall within the WRF domain). In theory, obs with QC flags between
> > 30000 and 32767 could be included in the obs file but rejected by
WRF,
> > but I'm doubtful that any obs would have QC flags in that range.
> >
> > In Obsgrid it determines obs are duplicates if the following is
true:
> > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > 2) The station ID and name match exactly (note that for MADIS data
the
> > station ID is OBS_SID in your spreadsheet, but the name is always
> > "MADIS")
> > 3) The times of the observations are within 30 minutes of one
another
> >
> > If the obs are duplicate then it tries to merge the two obs and
uses
> > the time of the ob closer to the time for which we are looking for
obs
> > (Obsgrid is being used to create hourly files that contain all obs
> > closest to that hour so this means it will use the time of the ob
> > closest to the current
> > hour) as the time of the ob.  For each variable in the ob it tries
to
> > pick the best from the two obs.  If does so using the following:
> > 1) Use the non-missing value if one and only one of the obs is
missing
> > 2) Use the ob with the lower QC flag
> > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > with more valid observations
> > 4) [I don't think the following flags are ever set so I suspect
these
> > never come into play] User the overall ob with fewer errors noted,
> > fewer warnings, and higher sequence number
> > 5) All else being equal, use the ob that is closer to the target
time
> > for which we are looking for an observation
> >
> > Based on all this, it may appear that stations like AIDC1 should
not
> > be in the obs file from both CA-Hydro and MesoWest since they have
> > identical lat/lons.  It looks like obs from AIDC1 are not in the
ob
> > file from both sources at the same time.  At least for the
February
> > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at
the
> > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > at that time to MADIS or perhaps the MesoWest version happened to
be
> > earlier in the file than the CA-Hydro version at that time, and
the
> > observations being identical Obsgrid chose the first ob.
> >
> > Brian
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Tuesday, July 01, 2014 2:45 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Once again, thank you for your time.  I appreciate your help.
> >
> > From your comments to the original spreadsheet, I added a column
> > called "INCLUDE" that is Boolean -  true to include that data,
false if
> to
> > ignore.
> > Universally, if it was a choice between CA-Hydro and another net,
I
> > choose CA-Hydro because of the precision in the data.
> >
> > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > no idea which, if either, is the correct data so absent better
> > information, I think it's better to ignore both.
> >
> > WRK KIYK, I chose the NonFED one for the same reason as you
recommended.
> >
> > For the greyed out triple values, I'll ignore those and John and I
> > will take a crack at figuring out what is going on there with MET.
> >
> > Regards,
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Friday, June 27, 2014 11:47 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> > Jeffrey,
> > I've went through the spreadsheet and found that whenever there
are
> > three different lat/lon/elevation triplets for a given station, I
only
> > find two of the combinations in the observation files I'm looking
at.
> > The third combination, the one I do not have, has a
lat/lon/elevation
> > combination that can be constructed from taking 2 of the fields
from
> > one of the other triplets and the other field from the other
triplet.
> > For example, if we have three listings for a station called YYY:
> > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > Then if lines 1 and 2 are the versions I have in my file while I
am
> > missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> > lon1 or lon2,
> > elev3 equals either elev1 or elev2 [and line 3 is not a duplicate
of
> > either line 1 or line 2]. This suggests that some processing step
may
> > be merging the station information in certain cases.  In light of
> > this, in the spreadsheet I grayed out the lines that did not seem
to
> > occur in the observation files that I have.
> >
> > I've added columns showing the source of the observation,
according to
> > MADIS.  I also added columns showing the difference in latitude,
> > longitude, and elevation between the two versions of each
observation
> > location.  I calculated an estimated distance between the two
> > locations as well.  I estimated the elevation at the station
location
> > according to the 1/3 arc-second NED dataset via
> http://ned.usgs.gov/epqs/
> > and compared this to
> > the elevation reported in the obs file to find an elevation
"error".   Of
> > course we do not know what the true elevation is exactly at the
> > observation location so this is not really necessarily an error.
> > Finally I put a summary column with some text regarding the
comparison.
> >
> > Most of the differences are differences between how CA-Hydro and
> > MesoWest report the lat/lon/elevation of stations. In general it
is
> > hard to know which is correct.  Often CA-Hydro matches better with
the
> > terrain database than MesoWest, but the number of times that CA-
Hydro
> > exactly matches the terrain database makes me suspicious that they
> > might be directly pulling it out of the same terrain database.
Beside
> > the CA-Hydro / MesoWest discrepencies, the following were found:
> >
> > For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > NonFedAWOS reports the values with more precision, perhaps it is
more
> > likely to be right?; also, the location it reports is right off
one of
> > the runways of the airport.
> >
> > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and
> > elevation
> > (~6 m).  The elevations reported by the two networks only differ
by
> > 8m, but the terrain database estimated elevation for the two
locations
> > differs by
> > ~113 m, resulting in the CA-Hydro elevation matching the terrain
> > database estimated elevation much better (~0.01m difference
compared to
> > 121.08 m).
> > This suggests that CA-Hydro is better.
> >
> > For WSPC1 and YRKC1, RAWS reports identical latitude/longitude,
and an
> > elevation that differs by 0.9 m.  Although the elevation matches
the
> > terrain database at this location pretty well, the location is far
> > from those reported by HADS (by about 110 km for WSPC1 and 156 km
for
> > YRKC1).  The HADS website lists descriptions of the two sites
> > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW") that
are
> > consistent with the HADS
> > locations but not the RAWS locations.    This suggests that the
HADS
> values
> > are correct.  However, note that the HADS elevation value for
WSPC1 is
> > about
> > 200 m lower than expected by the terrain database.
> >
> >
> > Brian
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Thursday, June 26, 2014 3:51 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Here are the SID's I was speaking to you about.  They are all
SID's
> > that produced at least one multiple ob, in a least one case hour
of
> > the five case days.  After John reprocessed the LA Basin data with
the
> > new switch, I ran my scripts against the data to produce the
table.
> >
> > Jeffrey
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Jul 08 15:40:31 2014

John  -

I just dropped off the files using the ARL SAFE. You should receive an
email notification soon.

I searched for the string "AJMC1" in the little_r file and it comes in
two flavors :>

AJMC1   35.73420 -119.14890 144.80000
AJMC1   35.72780 -119.13250 151.44901

The above is the duplicate which the MADIS data source is providing
and represents a type of duplicate which isn't supposed to occur, that
is "one station identifier having two different locations. I will be
taking this problem up with the MADIS administrators soon.

I found no instances of the following in the little_r file:
AJMC1   35.73420 -119.14890 992.55835  151.44901
This one seems to be a fictitious ob that may be generated by Point-
Stat. I only see this in the MPR file.

R/
John

_______________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, July 08, 2014 2:13 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)

John,

Sure that would work, you can send it to "johnhg at ucar.edu".

Thanks,
John


On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks. I will send you the data you request, but, from my last
attempt to
> transfer large files (15mb), ftp is blocked locally (unless you have
a sftp
> server you can point me to) and I will have to resort to using the
ARL SAFE
> file exchange system. The problem is that it sends you the email
with login
> info to provide you secure access for downloading the files, but
using the
> MET_HELP address potentially publishes this info to the open
internet. Do
> you
> have an alternate email address I could use to facilitate this
transfer? I
> have your address from UCAR if that would work for you.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, July 04, 2014 11:53 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> I'm having a difficult time exactly following the logic here, but
I'll do
> my
> best.  If you have more questions, please send me an input little_r
file, a
> sample forecast file, and your Point-Stat config file.  Running it
here
> through the debugger is the easiest way for me to trace through the
logic.
>
> The "single" logic in the code is handled by a map called
"map_single".  A
> map
> is a data structure that takes an input "key" and maps it to a
"value".
>  For example, you might have a map that stores an baseball team's
home
> state.
> The "Rockies" key would map to a value of "Colorado", while the
"Dodgers"
> key
> would map to a value of "California".
>
> Keys for the single map are constructed using the observation's
latitude,
> longitude, level, and elevation.  The value is stored as the station
id,
> the
> valid time, and the observation value.  For each observation Point-
Stat
> processes, it constructs that key and checks to see if we've already
> encountered it.  If not, we add a new entry.  If so, we check to see
if
> it's
> value should be updated.
>
> But there's a wrinkle when we're verifying surface observations,
such as
> 2-m
> temperature.  This is described in the METv4.1 release notes in the
5th
> bullet
> from the bottom:
>
>
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
>
> When verifying surface obs, we reset the level value to NA so that
it's not
> used in the duplicate logic.
>
> Looking at the MPR file you sent, for the AJMC1 station, for TMP/Z2,
I
> extracted the lat, lon, level, and elevation columns, and here's
what I
> see:
>
> OBS_LAT OBS_LON OBS_LVL OBS_ELV
>
> 35.73420 -119.14890 NA 144.80000
>
> 35.73420 -119.14890 NA 151.44901
>
> 35.72780 -119.13250 NA 151.44901
> Stringing these values together to create the "single" key, you can
see
> that
> all the keys are unique.  So the current logic won't flag these as
being
> duplicates.
>
> Does that help clarify?  I suspect the note included in the METv4.1
release
> note helps explain the question you had.
>
> Hope that helps and have a happy 4th.
>
> Thanks,
> John
>
>
> On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I am using little_r formatted observation files from the MADIS
source
> > and post-processed WRF output in Point-Stat (METV4.1) to generate
MPR
> > as well as other line types.
> >
> > My config file specifies "SINGLE" duplicate flag.
> >
> > The MPR file contains Z2/TMP observations from AJMC1 (id) which
are
> > duplicate in the sense that one id has two different locations
(lat or
> > long or elev). I can trace the source of the duplicate obs back to
the
> > original little_r file.
> >
> > I also find in the MPR file a third instance (triple) of a report
> > (Z2/TMP) from AJMC1 which contains location info in a combination
not
> > found in the duplicate reports for AJMC1 which are also contained
in the
> > little_r file.
> > The lat/long/elev info for this station seems to be a permutation
of
> > the location data for the duplicate stations. See the following
> > listings for three different model lead times:
> >
> > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > triple for Z2/TMP obs:
> >
> > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776
>  NA
> > (original dupe present in little_r file)
> > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> > NA(triple)
> > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222
>  NA
> > (original dupe present in little_r file)
> >
> > For 17 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
284.81668
> > NA
> > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
284.81668
> > NA (triple)
> > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
283.70557
> > NA
> >
> > For 21 lead: same pattern is repeated
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
282.03888
> > NA
> > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
282.59445
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
282.03888
> > NA (triple)
> >
> > For 08 lead: exception  - missing one of the original dupes
> >
> > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
292.03888
> > NA
> > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
292.03888
> > NA (triple)
> >
> > It appears that Point-Stat is generating a fictitious ob (triple)
> > whose combination of lat/long/elev cannot be traced to any of the
> > input obs in the little_r files.
> >
> > I've attached one of the MPR text files (02 lead time) to this
email
> > for your inspection.
> >
> > Could you determine why this triple ob shows up in the MPR file
and
> > verify that this may or may not be a bug in the Point-Stat
software?
> >
> > Thanks.
> >
> > R/
> > John
> > ________________________________________
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 1:40 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US); <jeffrey.a.smith at zianet.com>
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > to say the least.  In a quick scan of my duplicate table for the
> > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I find
> > just doubles.  For that station ID, I find a total
> > of 33 records.   In each of the triples, two of the OBS_VALID_END
times
> are
> > the same and one is different.  It appears from this limited
sample,
> > the one element of the triple you excluded was one of the two
'SAME'
> > OBS_VALID_END times.  Furthermore, it appears that OBS should have
> > been either part of the previous hour or the subsequent hour
forecast
> > (i.e., not within +/- 15 minutes of the top of the hour).
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Wednesday, July 02, 2014 8:03 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Jeffrey,
> > Your choices on which obs to include looks good to me.
> >
> > Following are some answers regarding the questions I could not
answer
> > yesterday on the phone about how WRF deals with potentially
duplicate
> obs.
> > It is most likely more detailed than you are looking for, but I
> > thought it would be good to have at least for reference sake.
> >
> > The Obsgrid software deals with quality control and dealing with
> > duplicate obs.  Obsgrid is used both to create the obs files I
gave to
> > John Raby for ingestion into MET and the obs files I use for data
> > assimilation in WRF.
> > Therefore, the obs I provided to you should be the same as the obs
> > that I used for data assimilation in WRF.  When I describe how
Obsgrid
> > works, I'm referring to the version that I use that I have
modified
> > from the standard version.
> >
> > The observation files use a quality control flag that is additive
> > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > another thing, so if the quality control flag is 288=256+32 we
know
> > that the condition represented by 32 is true as is the condition
> represented
> > by 256 is true.
> > The most serious flags are the largest flags.  I have Obsgrid set
to
> > not output any obs with a QC flag of 32768 or greater, and WRF is
> > hardcoded to ignore obs with a QC flag of 30000 or greater, so any
obs
> > in the obs file I gave should be being used by WRF (assuming that
they
> > fall within the WRF domain). In theory, obs with QC flags between
> > 30000 and 32767 could be included in the obs file but rejected by
WRF,
> > but I'm doubtful that any obs would have QC flags in that range.
> >
> > In Obsgrid it determines obs are duplicates if the following is
true:
> > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > 2) The station ID and name match exactly (note that for MADIS data
the
> > station ID is OBS_SID in your spreadsheet, but the name is always
> > "MADIS")
> > 3) The times of the observations are within 30 minutes of one
another
> >
> > If the obs are duplicate then it tries to merge the two obs and
uses
> > the time of the ob closer to the time for which we are looking for
obs
> > (Obsgrid is being used to create hourly files that contain all obs
> > closest to that hour so this means it will use the time of the ob
> > closest to the current
> > hour) as the time of the ob.  For each variable in the ob it tries
to
> > pick the best from the two obs.  If does so using the following:
> > 1) Use the non-missing value if one and only one of the obs is
missing
> > 2) Use the ob with the lower QC flag
> > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > with more valid observations
> > 4) [I don't think the following flags are ever set so I suspect
these
> > never come into play] User the overall ob with fewer errors noted,
> > fewer warnings, and higher sequence number
> > 5) All else being equal, use the ob that is closer to the target
time
> > for which we are looking for an observation
> >
> > Based on all this, it may appear that stations like AIDC1 should
not
> > be in the obs file from both CA-Hydro and MesoWest since they have
> > identical lat/lons.  It looks like obs from AIDC1 are not in the
ob
> > file from both sources at the same time.  At least for the
February
> > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at
the
> > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > at that time to MADIS or perhaps the MesoWest version happened to
be
> > earlier in the file than the CA-Hydro version at that time, and
the
> > observations being identical Obsgrid chose the first ob.
> >
> > Brian
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Tuesday, July 01, 2014 2:45 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Once again, thank you for your time.  I appreciate your help.
> >
> > From your comments to the original spreadsheet, I added a column
> > called "INCLUDE" that is Boolean -  true to include that data,
false if
> to
> > ignore.
> > Universally, if it was a choice between CA-Hydro and another net,
I
> > choose CA-Hydro because of the precision in the data.
> >
> > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > no idea which, if either, is the correct data so absent better
> > information, I think it's better to ignore both.
> >
> > WRK KIYK, I chose the NonFED one for the same reason as you
recommended.
> >
> > For the greyed out triple values, I'll ignore those and John and I
> > will take a crack at figuring out what is going on there with MET.
> >
> > Regards,
> >
> > Jeffrey
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY ARL (US)
> > Sent: Friday, June 27, 2014 11:47 AM
> > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: RE: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> > Jeffrey,
> > I've went through the spreadsheet and found that whenever there
are
> > three different lat/lon/elevation triplets for a given station, I
only
> > find two of the combinations in the observation files I'm looking
at.
> > The third combination, the one I do not have, has a
lat/lon/elevation
> > combination that can be constructed from taking 2 of the fields
from
> > one of the other triplets and the other field from the other
triplet.
> > For example, if we have three listings for a station called YYY:
> > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > Then if lines 1 and 2 are the versions I have in my file while I
am
> > missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> > lon1 or lon2,
> > elev3 equals either elev1 or elev2 [and line 3 is not a duplicate
of
> > either line 1 or line 2]. This suggests that some processing step
may
> > be merging the station information in certain cases.  In light of
> > this, in the spreadsheet I grayed out the lines that did not seem
to
> > occur in the observation files that I have.
> >
> > I've added columns showing the source of the observation,
according to
> > MADIS.  I also added columns showing the difference in latitude,
> > longitude, and elevation between the two versions of each
observation
> > location.  I calculated an estimated distance between the two
> > locations as well.  I estimated the elevation at the station
location
> > according to the 1/3 arc-second NED dataset via
> http://ned.usgs.gov/epqs/
> > and compared this to
> > the elevation reported in the obs file to find an elevation
"error".   Of
> > course we do not know what the true elevation is exactly at the
> > observation location so this is not really necessarily an error.
> > Finally I put a summary column with some text regarding the
comparison.
> >
> > Most of the differences are differences between how CA-Hydro and
> > MesoWest report the lat/lon/elevation of stations. In general it
is
> > hard to know which is correct.  Often CA-Hydro matches better with
the
> > terrain database than MesoWest, but the number of times that CA-
Hydro
> > exactly matches the terrain database makes me suspicious that they
> > might be directly pulling it out of the same terrain database.
Beside
> > the CA-Hydro / MesoWest discrepencies, the following were found:
> >
> > For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > NonFedAWOS reports the values with more precision, perhaps it is
more
> > likely to be right?; also, the location it reports is right off
one of
> > the runways of the airport.
> >
> > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m) and
> > elevation
> > (~6 m).  The elevations reported by the two networks only differ
by
> > 8m, but the terrain database estimated elevation for the two
locations
> > differs by
> > ~113 m, resulting in the CA-Hydro elevation matching the terrain
> > database estimated elevation much better (~0.01m difference
compared to
> > 121.08 m).
> > This suggests that CA-Hydro is better.
> >
> > For WSPC1 and YRKC1, RAWS reports identical latitude/longitude,
and an
> > elevation that differs by 0.9 m.  Although the elevation matches
the
> > terrain database at this location pretty well, the location is far
> > from those reported by HADS (by about 110 km for WSPC1 and 156 km
for
> > YRKC1).  The HADS website lists descriptions of the two sites
> > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW") that
are
> > consistent with the HADS
> > locations but not the RAWS locations.    This suggests that the
HADS
> values
> > are correct.  However, note that the HADS elevation value for
WSPC1 is
> > about
> > 200 m lower than expected by the terrain database.
> >
> >
> > Brian
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > Sent: Thursday, June 26, 2014 3:51 PM
> > To: Reen, Brian P CIV USARMY ARL (US)
> > Cc: Raby, John W CIV USARMY ARL (US)
> > Subject: Bad SID Table (UNCLASSIFIED)
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Brian.
> >
> > Here are the SID's I was speaking to you about.  They are all
SID's
> > that produced at least one multiple ob, in a least one case hour
of
> > the five case days.  After John reprocessed the LA Basin data with
the
> > new switch, I ran my scripts against the data to produce the
table.
> >
> > Jeffrey
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>



------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jul 09 12:44:49 2014

John,

Thanks for sending this.  Unfortunately, I don't have time to work on
this
today, but will take a look tomorrow.

Thanks,
John


On Tue, Jul 8, 2014 at 3:40 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> John  -
>
> I just dropped off the files using the ARL SAFE. You should receive
an
> email notification soon.
>
> I searched for the string "AJMC1" in the little_r file and it comes
in two
> flavors :>
>
> AJMC1   35.73420 -119.14890 144.80000
> AJMC1   35.72780 -119.13250 151.44901
>
> The above is the duplicate which the MADIS data source is providing
and
> represents a type of duplicate which isn't supposed to occur, that
is "one
> station identifier having two different locations. I will be taking
this
> problem up with the MADIS administrators soon.
>
> I found no instances of the following in the little_r file:
> AJMC1   35.73420 -119.14890 992.55835  151.44901
> This one seems to be a fictitious ob that may be generated by Point-
Stat.
> I only see this in the MPR file.
>
> R/
> John
>
> _______________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, July 08, 2014 2:13 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> Sure that would work, you can send it to "johnhg at ucar.edu".
>
> Thanks,
> John
>
>
> On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks. I will send you the data you request, but, from my last
attempt
> to
> > transfer large files (15mb), ftp is blocked locally (unless you
have a
> sftp
> > server you can point me to) and I will have to resort to using the
ARL
> SAFE
> > file exchange system. The problem is that it sends you the email
with
> login
> > info to provide you secure access for downloading the files, but
using
> the
> > MET_HELP address potentially publishes this info to the open
internet. Do
> > you
> > have an alternate email address I could use to facilitate this
transfer?
> I
> > have your address from UCAR if that would work for you.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, July 04, 2014 11:53 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> >
> > John,
> >
> > I'm having a difficult time exactly following the logic here, but
I'll do
> > my
> > best.  If you have more questions, please send me an input
little_r
> file, a
> > sample forecast file, and your Point-Stat config file.  Running it
here
> > through the debugger is the easiest way for me to trace through
the
> logic.
> >
> > The "single" logic in the code is handled by a map called
"map_single".
>  A
> > map
> > is a data structure that takes an input "key" and maps it to a
"value".
> >  For example, you might have a map that stores an baseball team's
home
> > state.
> > The "Rockies" key would map to a value of "Colorado", while the
"Dodgers"
> > key
> > would map to a value of "California".
> >
> > Keys for the single map are constructed using the observation's
latitude,
> > longitude, level, and elevation.  The value is stored as the
station id,
> > the
> > valid time, and the observation value.  For each observation
Point-Stat
> > processes, it constructs that key and checks to see if we've
already
> > encountered it.  If not, we add a new entry.  If so, we check to
see if
> > it's
> > value should be updated.
> >
> > But there's a wrinkle when we're verifying surface observations,
such as
> > 2-m
> > temperature.  This is described in the METv4.1 release notes in
the 5th
> > bullet
> > from the bottom:
> >
> >
> >
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
> >
> > When verifying surface obs, we reset the level value to NA so that
it's
> not
> > used in the duplicate logic.
> >
> > Looking at the MPR file you sent, for the AJMC1 station, for
TMP/Z2, I
> > extracted the lat, lon, level, and elevation columns, and here's
what I
> > see:
> >
> > OBS_LAT OBS_LON OBS_LVL OBS_ELV
> >
> > 35.73420 -119.14890 NA 144.80000
> >
> > 35.73420 -119.14890 NA 151.44901
> >
> > 35.72780 -119.13250 NA 151.44901
> > Stringing these values together to create the "single" key, you
can see
> > that
> > all the keys are unique.  So the current logic won't flag these as
being
> > duplicates.
> >
> > Does that help clarify?  I suspect the note included in the
METv4.1
> release
> > note helps explain the question you had.
> >
> > Hope that helps and have a happy 4th.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I am using little_r formatted observation files from the MADIS
source
> > > and post-processed WRF output in Point-Stat (METV4.1) to
generate MPR
> > > as well as other line types.
> > >
> > > My config file specifies "SINGLE" duplicate flag.
> > >
> > > The MPR file contains Z2/TMP observations from AJMC1 (id) which
are
> > > duplicate in the sense that one id has two different locations
(lat or
> > > long or elev). I can trace the source of the duplicate obs back
to the
> > > original little_r file.
> > >
> > > I also find in the MPR file a third instance (triple) of a
report
> > > (Z2/TMP) from AJMC1 which contains location info in a
combination not
> > > found in the duplicate reports for AJMC1 which are also
contained in
> the
> > > little_r file.
> > > The lat/long/elev info for this station seems to be a
permutation of
> > > the location data for the duplicate stations. See the following
> > > listings for three different model lead times:
> > >
> > > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > > triple for Z2/TMP obs:
> > >
> > > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776
> >  NA
> > > (original dupe present in little_r file)
> > > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> > > NA(triple)
> > > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222
> >  NA
> > > (original dupe present in little_r file)
> > >
> > > For 17 lead: same pattern is repeated
> > >
> > > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
> 284.81668
> > > NA
> > > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
> 284.81668
> > > NA (triple)
> > > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
> 283.70557
> > > NA
> > >
> > > For 21 lead: same pattern is repeated
> > >
> > > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
> 282.03888
> > > NA
> > > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
> 282.59445
> > > NA
> > > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
> 282.03888
> > > NA (triple)
> > >
> > > For 08 lead: exception  - missing one of the original dupes
> > >
> > > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
> 292.03888
> > > NA
> > > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
> 292.03888
> > > NA (triple)
> > >
> > > It appears that Point-Stat is generating a fictitious ob
(triple)
> > > whose combination of lat/long/elev cannot be traced to any of
the
> > > input obs in the little_r files.
> > >
> > > I've attached one of the MPR text files (02 lead time) to this
email
> > > for your inspection.
> > >
> > > Could you determine why this triple ob shows up in the MPR file
and
> > > verify that this may or may not be a bug in the Point-Stat
software?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Wednesday, July 02, 2014 1:40 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US);
<jeffrey.a.smith at zianet.com>
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > > to say the least.  In a quick scan of my duplicate table for the
> > > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> > > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I
find
> > > just doubles.  For that station ID, I find a total
> > > of 33 records.   In each of the triples, two of the
OBS_VALID_END times
> > are
> > > the same and one is different.  It appears from this limited
sample,
> > > the one element of the triple you excluded was one of the two
'SAME'
> > > OBS_VALID_END times.  Furthermore, it appears that OBS should
have
> > > been either part of the previous hour or the subsequent hour
forecast
> > > (i.e., not within +/- 15 minutes of the top of the hour).
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY ARL (US)
> > > Sent: Wednesday, July 02, 2014 8:03 AM
> > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Jeffrey,
> > > Your choices on which obs to include looks good to me.
> > >
> > > Following are some answers regarding the questions I could not
answer
> > > yesterday on the phone about how WRF deals with potentially
duplicate
> > obs.
> > > It is most likely more detailed than you are looking for, but I
> > > thought it would be good to have at least for reference sake.
> > >
> > > The Obsgrid software deals with quality control and dealing with
> > > duplicate obs.  Obsgrid is used both to create the obs files I
gave to
> > > John Raby for ingestion into MET and the obs files I use for
data
> > > assimilation in WRF.
> > > Therefore, the obs I provided to you should be the same as the
obs
> > > that I used for data assimilation in WRF.  When I describe how
Obsgrid
> > > works, I'm referring to the version that I use that I have
modified
> > > from the standard version.
> > >
> > > The observation files use a quality control flag that is
additive
> > > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > > another thing, so if the quality control flag is 288=256+32 we
know
> > > that the condition represented by 32 is true as is the condition
> > represented
> > > by 256 is true.
> > > The most serious flags are the largest flags.  I have Obsgrid
set to
> > > not output any obs with a QC flag of 32768 or greater, and WRF
is
> > > hardcoded to ignore obs with a QC flag of 30000 or greater, so
any obs
> > > in the obs file I gave should be being used by WRF (assuming
that they
> > > fall within the WRF domain). In theory, obs with QC flags
between
> > > 30000 and 32767 could be included in the obs file but rejected
by WRF,
> > > but I'm doubtful that any obs would have QC flags in that range.
> > >
> > > In Obsgrid it determines obs are duplicates if the following is
true:
> > > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > > 2) The station ID and name match exactly (note that for MADIS
data the
> > > station ID is OBS_SID in your spreadsheet, but the name is
always
> > > "MADIS")
> > > 3) The times of the observations are within 30 minutes of one
another
> > >
> > > If the obs are duplicate then it tries to merge the two obs and
uses
> > > the time of the ob closer to the time for which we are looking
for obs
> > > (Obsgrid is being used to create hourly files that contain all
obs
> > > closest to that hour so this means it will use the time of the
ob
> > > closest to the current
> > > hour) as the time of the ob.  For each variable in the ob it
tries to
> > > pick the best from the two obs.  If does so using the following:
> > > 1) Use the non-missing value if one and only one of the obs is
missing
> > > 2) Use the ob with the lower QC flag
> > > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > > with more valid observations
> > > 4) [I don't think the following flags are ever set so I suspect
these
> > > never come into play] User the overall ob with fewer errors
noted,
> > > fewer warnings, and higher sequence number
> > > 5) All else being equal, use the ob that is closer to the target
time
> > > for which we are looking for an observation
> > >
> > > Based on all this, it may appear that stations like AIDC1 should
not
> > > be in the obs file from both CA-Hydro and MesoWest since they
have
> > > identical lat/lons.  It looks like obs from AIDC1 are not in the
ob
> > > file from both sources at the same time.  At least for the
February
> > > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at
the
> > > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > > at that time to MADIS or perhaps the MesoWest version happened
to be
> > > earlier in the file than the CA-Hydro version at that time, and
the
> > > observations being identical Obsgrid chose the first ob.
> > >
> > > Brian
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Tuesday, July 01, 2014 2:45 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Brian.
> > >
> > > Once again, thank you for your time.  I appreciate your help.
> > >
> > > From your comments to the original spreadsheet, I added a column
> > > called "INCLUDE" that is Boolean -  true to include that data,
false if
> > to
> > > ignore.
> > > Universally, if it was a choice between CA-Hydro and another
net, I
> > > choose CA-Hydro because of the precision in the data.
> > >
> > > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > > no idea which, if either, is the correct data so absent better
> > > information, I think it's better to ignore both.
> > >
> > > WRK KIYK, I chose the NonFED one for the same reason as you
> recommended.
> > >
> > > For the greyed out triple values, I'll ignore those and John and
I
> > > will take a crack at figuring out what is going on there with
MET.
> > >
> > > Regards,
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY ARL (US)
> > > Sent: Friday, June 27, 2014 11:47 AM
> > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > > Jeffrey,
> > > I've went through the spreadsheet and found that whenever there
are
> > > three different lat/lon/elevation triplets for a given station,
I only
> > > find two of the combinations in the observation files I'm
looking at.
> > > The third combination, the one I do not have, has a
lat/lon/elevation
> > > combination that can be constructed from taking 2 of the fields
from
> > > one of the other triplets and the other field from the other
triplet.
> > > For example, if we have three listings for a station called YYY:
> > > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > > Then if lines 1 and 2 are the versions I have in my file while I
am
> > > missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> > > lon1 or lon2,
> > > elev3 equals either elev1 or elev2 [and line 3 is not a
duplicate of
> > > either line 1 or line 2]. This suggests that some processing
step may
> > > be merging the station information in certain cases.  In light
of
> > > this, in the spreadsheet I grayed out the lines that did not
seem to
> > > occur in the observation files that I have.
> > >
> > > I've added columns showing the source of the observation,
according to
> > > MADIS.  I also added columns showing the difference in latitude,
> > > longitude, and elevation between the two versions of each
observation
> > > location.  I calculated an estimated distance between the two
> > > locations as well.  I estimated the elevation at the station
location
> > > according to the 1/3 arc-second NED dataset via
> > http://ned.usgs.gov/epqs/
> > > and compared this to
> > > the elevation reported in the obs file to find an elevation
"error".
> Of
> > > course we do not know what the true elevation is exactly at the
> > > observation location so this is not really necessarily an error.
> > > Finally I put a summary column with some text regarding the
comparison.
> > >
> > > Most of the differences are differences between how CA-Hydro and
> > > MesoWest report the lat/lon/elevation of stations. In general it
is
> > > hard to know which is correct.  Often CA-Hydro matches better
with the
> > > terrain database than MesoWest, but the number of times that CA-
Hydro
> > > exactly matches the terrain database makes me suspicious that
they
> > > might be directly pulling it out of the same terrain database.
Beside
> > > the CA-Hydro / MesoWest discrepencies, the following were found:
> > >
> > > For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> > > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > > NonFedAWOS reports the values with more precision, perhaps it is
more
> > > likely to be right?; also, the location it reports is right off
one of
> > > the runways of the airport.
> > >
> > > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m)
and
> > > elevation
> > > (~6 m).  The elevations reported by the two networks only differ
by
> > > 8m, but the terrain database estimated elevation for the two
locations
> > > differs by
> > > ~113 m, resulting in the CA-Hydro elevation matching the terrain
> > > database estimated elevation much better (~0.01m difference
compared to
> > > 121.08 m).
> > > This suggests that CA-Hydro is better.
> > >
> > > For WSPC1 and YRKC1, RAWS reports identical latitude/longitude,
and an
> > > elevation that differs by 0.9 m.  Although the elevation matches
the
> > > terrain database at this location pretty well, the location is
far
> > > from those reported by HADS (by about 110 km for WSPC1 and 156
km for
> > > YRKC1).  The HADS website lists descriptions of the two sites
> > > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW")
that are
> > > consistent with the HADS
> > > locations but not the RAWS locations.    This suggests that the
HADS
> > values
> > > are correct.  However, note that the HADS elevation value for
WSPC1 is
> > > about
> > > 200 m lower than expected by the terrain database.
> > >
> > >
> > > Brian
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Thursday, June 26, 2014 3:51 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Brian.
> > >
> > > Here are the SID's I was speaking to you about.  They are all
SID's
> > > that produced at least one multiple ob, in a least one case hour
of
> > > the five case days.  After John reprocessed the LA Basin data
with the
> > > new switch, I ran my scripts against the data to produce the
table.
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 09 13:40:03 2014

John -

No problem.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, July 09, 2014 12:44 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)

John,

Thanks for sending this.  Unfortunately, I don't have time to work on
this
today, but will take a look tomorrow.

Thanks,
John


On Tue, Jul 8, 2014 at 3:40 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> John  -
>
> I just dropped off the files using the ARL SAFE. You should receive
an
> email notification soon.
>
> I searched for the string "AJMC1" in the little_r file and it comes
in two
> flavors :>
>
> AJMC1   35.73420 -119.14890 144.80000
> AJMC1   35.72780 -119.13250 151.44901
>
> The above is the duplicate which the MADIS data source is providing
and
> represents a type of duplicate which isn't supposed to occur, that
is "one
> station identifier having two different locations. I will be taking
this
> problem up with the MADIS administrators soon.
>
> I found no instances of the following in the little_r file:
> AJMC1   35.73420 -119.14890 992.55835  151.44901
> This one seems to be a fictitious ob that may be generated by Point-
Stat.
> I only see this in the MPR file.
>
> R/
> John
>
> _______________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, July 08, 2014 2:13 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> Sure that would work, you can send it to "johnhg at ucar.edu".
>
> Thanks,
> John
>
>
> On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks. I will send you the data you request, but, from my last
attempt
> to
> > transfer large files (15mb), ftp is blocked locally (unless you
have a
> sftp
> > server you can point me to) and I will have to resort to using the
ARL
> SAFE
> > file exchange system. The problem is that it sends you the email
with
> login
> > info to provide you secure access for downloading the files, but
using
> the
> > MET_HELP address potentially publishes this info to the open
internet. Do
> > you
> > have an alternate email address I could use to facilitate this
transfer?
> I
> > have your address from UCAR if that would work for you.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, July 04, 2014 11:53 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> >
> > John,
> >
> > I'm having a difficult time exactly following the logic here, but
I'll do
> > my
> > best.  If you have more questions, please send me an input
little_r
> file, a
> > sample forecast file, and your Point-Stat config file.  Running it
here
> > through the debugger is the easiest way for me to trace through
the
> logic.
> >
> > The "single" logic in the code is handled by a map called
"map_single".
>  A
> > map
> > is a data structure that takes an input "key" and maps it to a
"value".
> >  For example, you might have a map that stores an baseball team's
home
> > state.
> > The "Rockies" key would map to a value of "Colorado", while the
"Dodgers"
> > key
> > would map to a value of "California".
> >
> > Keys for the single map are constructed using the observation's
latitude,
> > longitude, level, and elevation.  The value is stored as the
station id,
> > the
> > valid time, and the observation value.  For each observation
Point-Stat
> > processes, it constructs that key and checks to see if we've
already
> > encountered it.  If not, we add a new entry.  If so, we check to
see if
> > it's
> > value should be updated.
> >
> > But there's a wrinkle when we're verifying surface observations,
such as
> > 2-m
> > temperature.  This is described in the METv4.1 release notes in
the 5th
> > bullet
> > from the bottom:
> >
> >
> >
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
> >
> > When verifying surface obs, we reset the level value to NA so that
it's
> not
> > used in the duplicate logic.
> >
> > Looking at the MPR file you sent, for the AJMC1 station, for
TMP/Z2, I
> > extracted the lat, lon, level, and elevation columns, and here's
what I
> > see:
> >
> > OBS_LAT OBS_LON OBS_LVL OBS_ELV
> >
> > 35.73420 -119.14890 NA 144.80000
> >
> > 35.73420 -119.14890 NA 151.44901
> >
> > 35.72780 -119.13250 NA 151.44901
> > Stringing these values together to create the "single" key, you
can see
> > that
> > all the keys are unique.  So the current logic won't flag these as
being
> > duplicates.
> >
> > Does that help clarify?  I suspect the note included in the
METv4.1
> release
> > note helps explain the question you had.
> >
> > Hope that helps and have a happy 4th.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I am using little_r formatted observation files from the MADIS
source
> > > and post-processed WRF output in Point-Stat (METV4.1) to
generate MPR
> > > as well as other line types.
> > >
> > > My config file specifies "SINGLE" duplicate flag.
> > >
> > > The MPR file contains Z2/TMP observations from AJMC1 (id) which
are
> > > duplicate in the sense that one id has two different locations
(lat or
> > > long or elev). I can trace the source of the duplicate obs back
to the
> > > original little_r file.
> > >
> > > I also find in the MPR file a third instance (triple) of a
report
> > > (Z2/TMP) from AJMC1 which contains location info in a
combination not
> > > found in the duplicate reports for AJMC1 which are also
contained in
> the
> > > little_r file.
> > > The lat/long/elev info for this station seems to be a
permutation of
> > > the location data for the duplicate stations. See the following
> > > listings for three different model lead times:
> > >
> > > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > > triple for Z2/TMP obs:
> > >
> > > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
290.92776
> >  NA
> > > (original dupe present in little_r file)
> > > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
290.92776
> > > NA(triple)
> > > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
290.37222
> >  NA
> > > (original dupe present in little_r file)
> > >
> > > For 17 lead: same pattern is repeated
> > >
> > > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
> 284.81668
> > > NA
> > > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
> 284.81668
> > > NA (triple)
> > > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
> 283.70557
> > > NA
> > >
> > > For 21 lead: same pattern is repeated
> > >
> > > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
> 282.03888
> > > NA
> > > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
> 282.59445
> > > NA
> > > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
> 282.03888
> > > NA (triple)
> > >
> > > For 08 lead: exception  - missing one of the original dupes
> > >
> > > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
> 292.03888
> > > NA
> > > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
> 292.03888
> > > NA (triple)
> > >
> > > It appears that Point-Stat is generating a fictitious ob
(triple)
> > > whose combination of lat/long/elev cannot be traced to any of
the
> > > input obs in the little_r files.
> > >
> > > I've attached one of the MPR text files (02 lead time) to this
email
> > > for your inspection.
> > >
> > > Could you determine why this triple ob shows up in the MPR file
and
> > > verify that this may or may not be a bug in the Point-Stat
software?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Wednesday, July 02, 2014 1:40 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US);
<jeffrey.a.smith at zianet.com>
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > > to say the least.  In a quick scan of my duplicate table for the
> > > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.  For
> > > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I
find
> > > just doubles.  For that station ID, I find a total
> > > of 33 records.   In each of the triples, two of the
OBS_VALID_END times
> > are
> > > the same and one is different.  It appears from this limited
sample,
> > > the one element of the triple you excluded was one of the two
'SAME'
> > > OBS_VALID_END times.  Furthermore, it appears that OBS should
have
> > > been either part of the previous hour or the subsequent hour
forecast
> > > (i.e., not within +/- 15 minutes of the top of the hour).
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY ARL (US)
> > > Sent: Wednesday, July 02, 2014 8:03 AM
> > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Jeffrey,
> > > Your choices on which obs to include looks good to me.
> > >
> > > Following are some answers regarding the questions I could not
answer
> > > yesterday on the phone about how WRF deals with potentially
duplicate
> > obs.
> > > It is most likely more detailed than you are looking for, but I
> > > thought it would be good to have at least for reference sake.
> > >
> > > The Obsgrid software deals with quality control and dealing with
> > > duplicate obs.  Obsgrid is used both to create the obs files I
gave to
> > > John Raby for ingestion into MET and the obs files I use for
data
> > > assimilation in WRF.
> > > Therefore, the obs I provided to you should be the same as the
obs
> > > that I used for data assimilation in WRF.  When I describe how
Obsgrid
> > > works, I'm referring to the version that I use that I have
modified
> > > from the standard version.
> > >
> > > The observation files use a quality control flag that is
additive
> > > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > > another thing, so if the quality control flag is 288=256+32 we
know
> > > that the condition represented by 32 is true as is the condition
> > represented
> > > by 256 is true.
> > > The most serious flags are the largest flags.  I have Obsgrid
set to
> > > not output any obs with a QC flag of 32768 or greater, and WRF
is
> > > hardcoded to ignore obs with a QC flag of 30000 or greater, so
any obs
> > > in the obs file I gave should be being used by WRF (assuming
that they
> > > fall within the WRF domain). In theory, obs with QC flags
between
> > > 30000 and 32767 could be included in the obs file but rejected
by WRF,
> > > but I'm doubtful that any obs would have QC flags in that range.
> > >
> > > In Obsgrid it determines obs are duplicates if the following is
true:
> > > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > > 2) The station ID and name match exactly (note that for MADIS
data the
> > > station ID is OBS_SID in your spreadsheet, but the name is
always
> > > "MADIS")
> > > 3) The times of the observations are within 30 minutes of one
another
> > >
> > > If the obs are duplicate then it tries to merge the two obs and
uses
> > > the time of the ob closer to the time for which we are looking
for obs
> > > (Obsgrid is being used to create hourly files that contain all
obs
> > > closest to that hour so this means it will use the time of the
ob
> > > closest to the current
> > > hour) as the time of the ob.  For each variable in the ob it
tries to
> > > pick the best from the two obs.  If does so using the following:
> > > 1) Use the non-missing value if one and only one of the obs is
missing
> > > 2) Use the ob with the lower QC flag
> > > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > > with more valid observations
> > > 4) [I don't think the following flags are ever set so I suspect
these
> > > never come into play] User the overall ob with fewer errors
noted,
> > > fewer warnings, and higher sequence number
> > > 5) All else being equal, use the ob that is closer to the target
time
> > > for which we are looking for an observation
> > >
> > > Based on all this, it may appear that stations like AIDC1 should
not
> > > be in the obs file from both CA-Hydro and MesoWest since they
have
> > > identical lat/lons.  It looks like obs from AIDC1 are not in the
ob
> > > file from both sources at the same time.  At least for the
February
> > > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and at
the
> > > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > > at that time to MADIS or perhaps the MesoWest version happened
to be
> > > earlier in the file than the CA-Hydro version at that time, and
the
> > > observations being identical Obsgrid chose the first ob.
> > >
> > > Brian
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Tuesday, July 01, 2014 2:45 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Brian.
> > >
> > > Once again, thank you for your time.  I appreciate your help.
> > >
> > > From your comments to the original spreadsheet, I added a column
> > > called "INCLUDE" that is Boolean -  true to include that data,
false if
> > to
> > > ignore.
> > > Universally, if it was a choice between CA-Hydro and another
net, I
> > > choose CA-Hydro because of the precision in the data.
> > >
> > > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > > no idea which, if either, is the correct data so absent better
> > > information, I think it's better to ignore both.
> > >
> > > WRK KIYK, I chose the NonFED one for the same reason as you
> recommended.
> > >
> > > For the greyed out triple values, I'll ignore those and John and
I
> > > will take a crack at figuring out what is going on there with
MET.
> > >
> > > Regards,
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY ARL (US)
> > > Sent: Friday, June 27, 2014 11:47 AM
> > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > > Jeffrey,
> > > I've went through the spreadsheet and found that whenever there
are
> > > three different lat/lon/elevation triplets for a given station,
I only
> > > find two of the combinations in the observation files I'm
looking at.
> > > The third combination, the one I do not have, has a
lat/lon/elevation
> > > combination that can be constructed from taking 2 of the fields
from
> > > one of the other triplets and the other field from the other
triplet.
> > > For example, if we have three listings for a station called YYY:
> > > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > > Then if lines 1 and 2 are the versions I have in my file while I
am
> > > missing version 3, lat3 equals either lat1 or lat2, lon3 equals
either
> > > lon1 or lon2,
> > > elev3 equals either elev1 or elev2 [and line 3 is not a
duplicate of
> > > either line 1 or line 2]. This suggests that some processing
step may
> > > be merging the station information in certain cases.  In light
of
> > > this, in the spreadsheet I grayed out the lines that did not
seem to
> > > occur in the observation files that I have.
> > >
> > > I've added columns showing the source of the observation,
according to
> > > MADIS.  I also added columns showing the difference in latitude,
> > > longitude, and elevation between the two versions of each
observation
> > > location.  I calculated an estimated distance between the two
> > > locations as well.  I estimated the elevation at the station
location
> > > according to the 1/3 arc-second NED dataset via
> > http://ned.usgs.gov/epqs/
> > > and compared this to
> > > the elevation reported in the obs file to find an elevation
"error".
> Of
> > > course we do not know what the true elevation is exactly at the
> > > observation location so this is not really necessarily an error.
> > > Finally I put a summary column with some text regarding the
comparison.
> > >
> > > Most of the differences are differences between how CA-Hydro and
> > > MesoWest report the lat/lon/elevation of stations. In general it
is
> > > hard to know which is correct.  Often CA-Hydro matches better
with the
> > > terrain database than MesoWest, but the number of times that CA-
Hydro
> > > exactly matches the terrain database makes me suspicious that
they
> > > might be directly pulling it out of the same terrain database.
Beside
> > > the CA-Hydro / MesoWest discrepencies, the following were found:
> > >
> > > For KIYK, the Inyokern Airport in California, OTHER-MTR and
NonFedAWOS
> > > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > > NonFedAWOS reports the values with more precision, perhaps it is
more
> > > likely to be right?; also, the location it reports is right off
one of
> > > the runways of the airport.
> > >
> > > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m)
and
> > > elevation
> > > (~6 m).  The elevations reported by the two networks only differ
by
> > > 8m, but the terrain database estimated elevation for the two
locations
> > > differs by
> > > ~113 m, resulting in the CA-Hydro elevation matching the terrain
> > > database estimated elevation much better (~0.01m difference
compared to
> > > 121.08 m).
> > > This suggests that CA-Hydro is better.
> > >
> > > For WSPC1 and YRKC1, RAWS reports identical latitude/longitude,
and an
> > > elevation that differs by 0.9 m.  Although the elevation matches
the
> > > terrain database at this location pretty well, the location is
far
> > > from those reported by HADS (by about 110 km for WSPC1 and 156
km for
> > > YRKC1).  The HADS website lists descriptions of the two sites
> > > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW")
that are
> > > consistent with the HADS
> > > locations but not the RAWS locations.    This suggests that the
HADS
> > values
> > > are correct.  However, note that the HADS elevation value for
WSPC1 is
> > > about
> > > 200 m lower than expected by the terrain database.
> > >
> > >
> > > Brian
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > Sent: Thursday, June 26, 2014 3:51 PM
> > > To: Reen, Brian P CIV USARMY ARL (US)
> > > Cc: Raby, John W CIV USARMY ARL (US)
> > > Subject: Bad SID Table (UNCLASSIFIED)
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > Brian.
> > >
> > > Here are the SID's I was speaking to you about.  They are all
SID's
> > > that produced at least one multiple ob, in a least one case hour
of
> > > the five case days.  After John reprocessed the LA Basin data
with the
> > > new switch, I ran my scripts against the data to produce the
table.
> > >
> > > Jeffrey
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>



------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jul 10 11:35:49 2014

John,

I ran METv4.1 ascii2nc and point_stat on the data you sent me.  And I
looked at the AJMC1 station in the output MPR file.  I see 2 lines for
TMP/P1010-910, 2 lines for HGT/P1010-910, and 2 lines for TMP/Z2.

To simplify things, I defined a single mask using just the AJMC1
station.
The resulting *_mpr.txt file is attached.  It contains the 6 MPR lines
described above.  For each of the "duplicates" notice that the lat/lon
values differ:
   AJMC1   35.73420   -119.14890
   AJMC1   35.72780   -119.13250

In your Point-Stat config file, "duplicate_flag" is set to "SINGLE".
As I
described before, for this logic point_stat looks at the combination
of
lat/lon/level/elevation.  It keeps track of the observation values for
each
unique combination of those fields.

Since the combination of lat/lon/level/elevation differ between the
duplicates you're seeing in the output they are treated as different
observations.

And for TMP/Z2, since we're verifying at the surface, the level is set
to
bad data to make the vertical level matching logic simpler.

I don't see any "triples", as you described it, in that output.

>From my point of view, it's working as it should.  I don't understand
where
the problem is.  Can you please look at the attached MPR file and
restate
your question?

Thanks,
John





On Wed, Jul 9, 2014 at 1:40 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> John -
>
> No problem.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, July 09, 2014 12:44 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> Thanks for sending this.  Unfortunately, I don't have time to work
on this
> today, but will take a look tomorrow.
>
> Thanks,
> John
>
>
> On Tue, Jul 8, 2014 at 3:40 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> >
> > John  -
> >
> > I just dropped off the files using the ARL SAFE. You should
receive an
> > email notification soon.
> >
> > I searched for the string "AJMC1" in the little_r file and it
comes in
> two
> > flavors :>
> >
> > AJMC1   35.73420 -119.14890 144.80000
> > AJMC1   35.72780 -119.13250 151.44901
> >
> > The above is the duplicate which the MADIS data source is
providing and
> > represents a type of duplicate which isn't supposed to occur, that
is
> "one
> > station identifier having two different locations. I will be
taking this
> > problem up with the MADIS administrators soon.
> >
> > I found no instances of the following in the little_r file:
> > AJMC1   35.73420 -119.14890 992.55835  151.44901
> > This one seems to be a fictitious ob that may be generated by
Point-Stat.
> > I only see this in the MPR file.
> >
> > R/
> > John
> >
> > _______________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, July 08, 2014 2:13 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> >
> > John,
> >
> > Sure that would work, you can send it to "johnhg at ucar.edu".
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks. I will send you the data you request, but, from my last
attempt
> > to
> > > transfer large files (15mb), ftp is blocked locally (unless you
have a
> > sftp
> > > server you can point me to) and I will have to resort to using
the ARL
> > SAFE
> > > file exchange system. The problem is that it sends you the email
with
> > login
> > > info to provide you secure access for downloading the files, but
using
> > the
> > > MET_HELP address potentially publishes this info to the open
internet.
> Do
> > > you
> > > have an alternate email address I could use to facilitate this
> transfer?
> > I
> > > have your address from UCAR if that would work for you.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, July 04, 2014 11:53 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I'm having a difficult time exactly following the logic here,
but I'll
> do
> > > my
> > > best.  If you have more questions, please send me an input
little_r
> > file, a
> > > sample forecast file, and your Point-Stat config file.  Running
it here
> > > through the debugger is the easiest way for me to trace through
the
> > logic.
> > >
> > > The "single" logic in the code is handled by a map called
"map_single".
> >  A
> > > map
> > > is a data structure that takes an input "key" and maps it to a
"value".
> > >  For example, you might have a map that stores an baseball
team's home
> > > state.
> > > The "Rockies" key would map to a value of "Colorado", while the
> "Dodgers"
> > > key
> > > would map to a value of "California".
> > >
> > > Keys for the single map are constructed using the observation's
> latitude,
> > > longitude, level, and elevation.  The value is stored as the
station
> id,
> > > the
> > > valid time, and the observation value.  For each observation
Point-Stat
> > > processes, it constructs that key and checks to see if we've
already
> > > encountered it.  If not, we add a new entry.  If so, we check to
see if
> > > it's
> > > value should be updated.
> > >
> > > But there's a wrinkle when we're verifying surface observations,
such
> as
> > > 2-m
> > > temperature.  This is described in the METv4.1 release notes in
the 5th
> > > bullet
> > > from the bottom:
> > >
> > >
> > >
> >
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
> > >
> > > When verifying surface obs, we reset the level value to NA so
that it's
> > not
> > > used in the duplicate logic.
> > >
> > > Looking at the MPR file you sent, for the AJMC1 station, for
TMP/Z2, I
> > > extracted the lat, lon, level, and elevation columns, and here's
what I
> > > see:
> > >
> > > OBS_LAT OBS_LON OBS_LVL OBS_ELV
> > >
> > > 35.73420 -119.14890 NA 144.80000
> > >
> > > 35.73420 -119.14890 NA 151.44901
> > >
> > > 35.72780 -119.13250 NA 151.44901
> > > Stringing these values together to create the "single" key, you
can see
> > > that
> > > all the keys are unique.  So the current logic won't flag these
as
> being
> > > duplicates.
> > >
> > > Does that help clarify?  I suspect the note included in the
METv4.1
> > release
> > > note helps explain the question you had.
> > >
> > > Hope that helps and have a happy 4th.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > >        Queue: met_help
> > > >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I am using little_r formatted observation files from the MADIS
source
> > > > and post-processed WRF output in Point-Stat (METV4.1) to
generate MPR
> > > > as well as other line types.
> > > >
> > > > My config file specifies "SINGLE" duplicate flag.
> > > >
> > > > The MPR file contains Z2/TMP observations from AJMC1 (id)
which are
> > > > duplicate in the sense that one id has two different locations
(lat
> or
> > > > long or elev). I can trace the source of the duplicate obs
back to
> the
> > > > original little_r file.
> > > >
> > > > I also find in the MPR file a third instance (triple) of a
report
> > > > (Z2/TMP) from AJMC1 which contains location info in a
combination not
> > > > found in the duplicate reports for AJMC1 which are also
contained in
> > the
> > > > little_r file.
> > > > The lat/long/elev info for this station seems to be a
permutation of
> > > > the location data for the duplicate stations. See the
following
> > > > listings for three different model lead times:
> > > >
> > > > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > > > triple for Z2/TMP obs:
> > > >
> > > > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
>  290.92776
> > >  NA
> > > > (original dupe present in little_r file)
> > > > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
>  290.92776
> > > > NA(triple)
> > > > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
>  290.37222
> > >  NA
> > > > (original dupe present in little_r file)
> > > >
> > > > For 17 lead: same pattern is repeated
> > > >
> > > > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
> > 284.81668
> > > > NA
> > > > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
> > 284.81668
> > > > NA (triple)
> > > > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
> > 283.70557
> > > > NA
> > > >
> > > > For 21 lead: same pattern is repeated
> > > >
> > > > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
> > 282.03888
> > > > NA
> > > > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
> > 282.59445
> > > > NA
> > > > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
> > 282.03888
> > > > NA (triple)
> > > >
> > > > For 08 lead: exception  - missing one of the original dupes
> > > >
> > > > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
> > 292.03888
> > > > NA
> > > > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
> > 292.03888
> > > > NA (triple)
> > > >
> > > > It appears that Point-Stat is generating a fictitious ob
(triple)
> > > > whose combination of lat/long/elev cannot be traced to any of
the
> > > > input obs in the little_r files.
> > > >
> > > > I've attached one of the MPR text files (02 lead time) to this
email
> > > > for your inspection.
> > > >
> > > > Could you determine why this triple ob shows up in the MPR
file and
> > > > verify that this may or may not be a bug in the Point-Stat
software?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Wednesday, July 02, 2014 1:40 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US);
<jeffrey.a.smith at zianet.com>
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > > > to say the least.  In a quick scan of my duplicate table for
the
> > > > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > > > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.
For
> > > > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I
find
> > > > just doubles.  For that station ID, I find a total
> > > > of 33 records.   In each of the triples, two of the
OBS_VALID_END
> times
> > > are
> > > > the same and one is different.  It appears from this limited
sample,
> > > > the one element of the triple you excluded was one of the two
'SAME'
> > > > OBS_VALID_END times.  Furthermore, it appears that OBS should
have
> > > > been either part of the previous hour or the subsequent hour
forecast
> > > > (i.e., not within +/- 15 minutes of the top of the hour).
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Reen, Brian P CIV USARMY ARL (US)
> > > > Sent: Wednesday, July 02, 2014 8:03 AM
> > > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Jeffrey,
> > > > Your choices on which obs to include looks good to me.
> > > >
> > > > Following are some answers regarding the questions I could not
answer
> > > > yesterday on the phone about how WRF deals with potentially
duplicate
> > > obs.
> > > > It is most likely more detailed than you are looking for, but
I
> > > > thought it would be good to have at least for reference sake.
> > > >
> > > > The Obsgrid software deals with quality control and dealing
with
> > > > duplicate obs.  Obsgrid is used both to create the obs files I
gave
> to
> > > > John Raby for ingestion into MET and the obs files I use for
data
> > > > assimilation in WRF.
> > > > Therefore, the obs I provided to you should be the same as the
obs
> > > > that I used for data assimilation in WRF.  When I describe how
> Obsgrid
> > > > works, I'm referring to the version that I use that I have
modified
> > > > from the standard version.
> > > >
> > > > The observation files use a quality control flag that is
additive
> > > > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > > > another thing, so if the quality control flag is 288=256+32 we
know
> > > > that the condition represented by 32 is true as is the
condition
> > > represented
> > > > by 256 is true.
> > > > The most serious flags are the largest flags.  I have Obsgrid
set to
> > > > not output any obs with a QC flag of 32768 or greater, and WRF
is
> > > > hardcoded to ignore obs with a QC flag of 30000 or greater, so
any
> obs
> > > > in the obs file I gave should be being used by WRF (assuming
that
> they
> > > > fall within the WRF domain). In theory, obs with QC flags
between
> > > > 30000 and 32767 could be included in the obs file but rejected
by
> WRF,
> > > > but I'm doubtful that any obs would have QC flags in that
range.
> > > >
> > > > In Obsgrid it determines obs are duplicates if the following
is true:
> > > > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > > > 2) The station ID and name match exactly (note that for MADIS
data
> the
> > > > station ID is OBS_SID in your spreadsheet, but the name is
always
> > > > "MADIS")
> > > > 3) The times of the observations are within 30 minutes of one
another
> > > >
> > > > If the obs are duplicate then it tries to merge the two obs
and uses
> > > > the time of the ob closer to the time for which we are looking
for
> obs
> > > > (Obsgrid is being used to create hourly files that contain all
obs
> > > > closest to that hour so this means it will use the time of the
ob
> > > > closest to the current
> > > > hour) as the time of the ob.  For each variable in the ob it
tries to
> > > > pick the best from the two obs.  If does so using the
following:
> > > > 1) Use the non-missing value if one and only one of the obs is
> missing
> > > > 2) Use the ob with the lower QC flag
> > > > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > > > with more valid observations
> > > > 4) [I don't think the following flags are ever set so I
suspect these
> > > > never come into play] User the overall ob with fewer errors
noted,
> > > > fewer warnings, and higher sequence number
> > > > 5) All else being equal, use the ob that is closer to the
target time
> > > > for which we are looking for an observation
> > > >
> > > > Based on all this, it may appear that stations like AIDC1
should not
> > > > be in the obs file from both CA-Hydro and MesoWest since they
have
> > > > identical lat/lons.  It looks like obs from AIDC1 are not in
the ob
> > > > file from both sources at the same time.  At least for the
February
> > > > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and
at the
> > > > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > > > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > > > at that time to MADIS or perhaps the MesoWest version happened
to be
> > > > earlier in the file than the CA-Hydro version at that time,
and the
> > > > observations being identical Obsgrid chose the first ob.
> > > >
> > > > Brian
> > > >
> > > > -----Original Message-----
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Tuesday, July 01, 2014 2:45 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Brian.
> > > >
> > > > Once again, thank you for your time.  I appreciate your help.
> > > >
> > > > From your comments to the original spreadsheet, I added a
column
> > > > called "INCLUDE" that is Boolean -  true to include that data,
false
> if
> > > to
> > > > ignore.
> > > > Universally, if it was a choice between CA-Hydro and another
net, I
> > > > choose CA-Hydro because of the precision in the data.
> > > >
> > > > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > > > no idea which, if either, is the correct data so absent better
> > > > information, I think it's better to ignore both.
> > > >
> > > > WRK KIYK, I chose the NonFED one for the same reason as you
> > recommended.
> > > >
> > > > For the greyed out triple values, I'll ignore those and John
and I
> > > > will take a crack at figuring out what is going on there with
MET.
> > > >
> > > > Regards,
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Reen, Brian P CIV USARMY ARL (US)
> > > > Sent: Friday, June 27, 2014 11:47 AM
> > > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > > Jeffrey,
> > > > I've went through the spreadsheet and found that whenever
there are
> > > > three different lat/lon/elevation triplets for a given
station, I
> only
> > > > find two of the combinations in the observation files I'm
looking at.
> > > > The third combination, the one I do not have, has a
lat/lon/elevation
> > > > combination that can be constructed from taking 2 of the
fields from
> > > > one of the other triplets and the other field from the other
triplet.
> > > > For example, if we have three listings for a station called
YYY:
> > > > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > > > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > > > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > > > Then if lines 1 and 2 are the versions I have in my file while
I am
> > > > missing version 3, lat3 equals either lat1 or lat2, lon3
equals
> either
> > > > lon1 or lon2,
> > > > elev3 equals either elev1 or elev2 [and line 3 is not a
duplicate of
> > > > either line 1 or line 2]. This suggests that some processing
step may
> > > > be merging the station information in certain cases.  In light
of
> > > > this, in the spreadsheet I grayed out the lines that did not
seem to
> > > > occur in the observation files that I have.
> > > >
> > > > I've added columns showing the source of the observation,
according
> to
> > > > MADIS.  I also added columns showing the difference in
latitude,
> > > > longitude, and elevation between the two versions of each
observation
> > > > location.  I calculated an estimated distance between the two
> > > > locations as well.  I estimated the elevation at the station
location
> > > > according to the 1/3 arc-second NED dataset via
> > > http://ned.usgs.gov/epqs/
> > > > and compared this to
> > > > the elevation reported in the obs file to find an elevation
"error".
> > Of
> > > > course we do not know what the true elevation is exactly at
the
> > > > observation location so this is not really necessarily an
error.
> > > > Finally I put a summary column with some text regarding the
> comparison.
> > > >
> > > > Most of the differences are differences between how CA-Hydro
and
> > > > MesoWest report the lat/lon/elevation of stations. In general
it is
> > > > hard to know which is correct.  Often CA-Hydro matches better
with
> the
> > > > terrain database than MesoWest, but the number of times that
CA-Hydro
> > > > exactly matches the terrain database makes me suspicious that
they
> > > > might be directly pulling it out of the same terrain database.
>  Beside
> > > > the CA-Hydro / MesoWest discrepencies, the following were
found:
> > > >
> > > > For KIYK, the Inyokern Airport in California, OTHER-MTR and
> NonFedAWOS
> > > > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > > > NonFedAWOS reports the values with more precision, perhaps it
is more
> > > > likely to be right?; also, the location it reports is right
off one
> of
> > > > the runways of the airport.
> > > >
> > > > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m)
and
> > > > elevation
> > > > (~6 m).  The elevations reported by the two networks only
differ by
> > > > 8m, but the terrain database estimated elevation for the two
> locations
> > > > differs by
> > > > ~113 m, resulting in the CA-Hydro elevation matching the
terrain
> > > > database estimated elevation much better (~0.01m difference
compared
> to
> > > > 121.08 m).
> > > > This suggests that CA-Hydro is better.
> > > >
> > > > For WSPC1 and YRKC1, RAWS reports identical
latitude/longitude, and
> an
> > > > elevation that differs by 0.9 m.  Although the elevation
matches the
> > > > terrain database at this location pretty well, the location is
far
> > > > from those reported by HADS (by about 110 km for WSPC1 and 156
km for
> > > > YRKC1).  The HADS website lists descriptions of the two sites
> > > > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW")
that
> are
> > > > consistent with the HADS
> > > > locations but not the RAWS locations.    This suggests that
the HADS
> > > values
> > > > are correct.  However, note that the HADS elevation value for
WSPC1
> is
> > > > about
> > > > 200 m lower than expected by the terrain database.
> > > >
> > > >
> > > > Brian
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Thursday, June 26, 2014 3:51 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Brian.
> > > >
> > > > Here are the SID's I was speaking to you about.  They are all
SID's
> > > > that produced at least one multiple ob, in a least one case
hour of
> > > > the five case days.  After John reprocessed the LA Basin data
with
> the
> > > > new switch, I ran my scripts against the data to produce the
table.
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: FW: Bad SID Table (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jul 10 11:35:50 2014

VERSION MODEL        FCST_LEAD FCST_VALID_BEG  FCST_VALID_END
OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV  OBS_VAR
OBS_LEV   OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL INDEX OBS_SID OBS_LAT
OBS_LON    OBS_LVL   OBS_ELV   FCST      OBS       CLIMO
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 TMP      P1010-910 TMP     P1010-910
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     1     AJMC1   35.73420 -119.14890 993.17383
144.80000 290.55268 290.92776 NA
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 TMP      P1010-910 TMP     P1010-910
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     2     AJMC1   35.72780 -119.13250 992.55835
151.44901 290.64264 290.92776 NA
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 HGT      P1010-910 HGT     P1010-910
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     1     AJMC1   35.73420 -119.14890 993.17383
144.80000 129.03070 144.80000 NA
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 HGT      P1010-910 HGT     P1010-910
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     2     AJMC1   35.72780 -119.13250 992.55835
151.44901 132.80670 151.44901 NA
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 TMP      Z2        TMP     Z2
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     1     AJMC1   35.73420 -119.14890 NA
144.80000 288.06141 290.92776 NA
V4.1    Reen9kmWRE-N 020000    20120207_140000 20120207_140000 000000
20120207_134500 20120207_134500 TMP      Z2        TMP     Z2
ADPSFC AJMC1   DW_MEAN     4           NA          NA         NA
NA    MPR       2     2     AJMC1   35.72780 -119.13250 NA
151.44901 288.47229 290.92776 NA

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jul 10 12:14:49 2014

John -

Thanks for testing with my data.

I am copying the data I have for the same station from my MPR file
below to compare with yours:

For P1010-910 TMP I have two consecutive lines:
AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268  290.92776
NA (row 928)
AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215  290.92776
NA (row 929)

Then I skip down to row 2011 and find another line for the same
identifier as follows:
AJMC1   35.72780 -119.13250 992.55835  151.44901  290.64264  290.37222
NA

For P1010-910 HGT, the pattern repeats:
AJMC1   35.73420 -119.14890 993.17383  144.80000  129.03070  144.80000
NA (row 3407)
AJMC1   35.73420 -119.14890 992.55835  151.44901  134.25891  151.44901
NA (row 3408)
then skip down to row 4691:
AJMC1   35.72780 -119.13250 992.55835  151.44901  132.80670  151.44901
NA

For Z2 TMP, the pattern repeats:
AJMC1   35.73420 -119.14890 NA         144.80000  288.06141  290.92776
NA (row 9567)
AJMC1   35.73420 -119.14890 NA         151.44901  288.06141  290.92776
NA (row (9568)
then skip to row 11487:
AJMC1   35.72780 -119.13250 NA         151.44901  288.47229  290.37222
NA

The data you show below in your email for AJMC1 agrees with the data
(above) in my rows 928, 2011, 3407, 4691, 9567, 11487

So my point is why are you not showing in your MPR file the particular
lat/long/elev combination corresponding to that in my rows 929, 3408
and 9568?

I sent you my MPR file so you can look at the rows by number.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, July 10, 2014 11:35 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table (UNCLASSIFIED)

John,

I ran METv4.1 ascii2nc and point_stat on the data you sent me.  And I
looked at the AJMC1 station in the output MPR file.  I see 2 lines for
TMP/P1010-910, 2 lines for HGT/P1010-910, and 2 lines for TMP/Z2.

To simplify things, I defined a single mask using just the AJMC1
station.
The resulting *_mpr.txt file is attached.  It contains the 6 MPR lines
described above.  For each of the "duplicates" notice that the lat/lon
values differ:
   AJMC1   35.73420   -119.14890
   AJMC1   35.72780   -119.13250

In your Point-Stat config file, "duplicate_flag" is set to "SINGLE".
As I
described before, for this logic point_stat looks at the combination
of
lat/lon/level/elevation.  It keeps track of the observation values for
each
unique combination of those fields.

Since the combination of lat/lon/level/elevation differ between the
duplicates you're seeing in the output they are treated as different
observations.

And for TMP/Z2, since we're verifying at the surface, the level is set
to
bad data to make the vertical level matching logic simpler.

I don't see any "triples", as you described it, in that output.

>From my point of view, it's working as it should.  I don't understand
where
the problem is.  Can you please look at the attached MPR file and
restate
your question?

Thanks,
John





On Wed, Jul 9, 2014 at 1:40 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
>
> John -
>
> No problem.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, July 09, 2014 12:44 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
>
> John,
>
> Thanks for sending this.  Unfortunately, I don't have time to work
on this
> today, but will take a look tomorrow.
>
> Thanks,
> John
>
>
> On Tue, Jul 8, 2014 at 3:40 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> >
> > John  -
> >
> > I just dropped off the files using the ARL SAFE. You should
receive an
> > email notification soon.
> >
> > I searched for the string "AJMC1" in the little_r file and it
comes in
> two
> > flavors :>
> >
> > AJMC1   35.73420 -119.14890 144.80000
> > AJMC1   35.72780 -119.13250 151.44901
> >
> > The above is the duplicate which the MADIS data source is
providing and
> > represents a type of duplicate which isn't supposed to occur, that
is
> "one
> > station identifier having two different locations. I will be
taking this
> > problem up with the MADIS administrators soon.
> >
> > I found no instances of the following in the little_r file:
> > AJMC1   35.73420 -119.14890 992.55835  151.44901
> > This one seems to be a fictitious ob that may be generated by
Point-Stat.
> > I only see this in the MPR file.
> >
> > R/
> > John
> >
> > _______________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, July 08, 2014 2:13 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> >
> > John,
> >
> > Sure that would work, you can send it to "johnhg at ucar.edu".
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jul 8, 2014 at 1:51 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks. I will send you the data you request, but, from my last
attempt
> > to
> > > transfer large files (15mb), ftp is blocked locally (unless you
have a
> > sftp
> > > server you can point me to) and I will have to resort to using
the ARL
> > SAFE
> > > file exchange system. The problem is that it sends you the email
with
> > login
> > > info to provide you secure access for downloading the files, but
using
> > the
> > > MET_HELP address potentially publishes this info to the open
internet.
> Do
> > > you
> > > have an alternate email address I could use to facilitate this
> transfer?
> > I
> > > have your address from UCAR if that would work for you.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, July 04, 2014 11:53 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #68016] FW: Bad SID Table
(UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I'm having a difficult time exactly following the logic here,
but I'll
> do
> > > my
> > > best.  If you have more questions, please send me an input
little_r
> > file, a
> > > sample forecast file, and your Point-Stat config file.  Running
it here
> > > through the debugger is the easiest way for me to trace through
the
> > logic.
> > >
> > > The "single" logic in the code is handled by a map called
"map_single".
> >  A
> > > map
> > > is a data structure that takes an input "key" and maps it to a
"value".
> > >  For example, you might have a map that stores an baseball
team's home
> > > state.
> > > The "Rockies" key would map to a value of "Colorado", while the
> "Dodgers"
> > > key
> > > would map to a value of "California".
> > >
> > > Keys for the single map are constructed using the observation's
> latitude,
> > > longitude, level, and elevation.  The value is stored as the
station
> id,
> > > the
> > > valid time, and the observation value.  For each observation
Point-Stat
> > > processes, it constructs that key and checks to see if we've
already
> > > encountered it.  If not, we add a new entry.  If so, we check to
see if
> > > it's
> > > value should be updated.
> > >
> > > But there's a wrinkle when we're verifying surface observations,
such
> as
> > > 2-m
> > > temperature.  This is described in the METv4.1 release notes in
the 5th
> > > bullet
> > > from the bottom:
> > >
> > >
> > >
> >
>
http://www.dtcenter.org/met/users/support/release_notes/METv4.1_release_notes.php
> > >
> > > When verifying surface obs, we reset the level value to NA so
that it's
> > not
> > > used in the duplicate logic.
> > >
> > > Looking at the MPR file you sent, for the AJMC1 station, for
TMP/Z2, I
> > > extracted the lat, lon, level, and elevation columns, and here's
what I
> > > see:
> > >
> > > OBS_LAT OBS_LON OBS_LVL OBS_ELV
> > >
> > > 35.73420 -119.14890 NA 144.80000
> > >
> > > 35.73420 -119.14890 NA 151.44901
> > >
> > > 35.72780 -119.13250 NA 151.44901
> > > Stringing these values together to create the "single" key, you
can see
> > > that
> > > all the keys are unique.  So the current logic won't flag these
as
> being
> > > duplicates.
> > >
> > > Does that help clarify?  I suspect the note included in the
METv4.1
> > release
> > > note helps explain the question you had.
> > >
> > > Hope that helps and have a happy 4th.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Wed, Jul 2, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jul 02 16:05:13 2014: Request 68016 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > >        Queue: met_help
> > > >      Subject: FW: Bad SID Table (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68016
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I am using little_r formatted observation files from the MADIS
source
> > > > and post-processed WRF output in Point-Stat (METV4.1) to
generate MPR
> > > > as well as other line types.
> > > >
> > > > My config file specifies "SINGLE" duplicate flag.
> > > >
> > > > The MPR file contains Z2/TMP observations from AJMC1 (id)
which are
> > > > duplicate in the sense that one id has two different locations
(lat
> or
> > > > long or elev). I can trace the source of the duplicate obs
back to
> the
> > > > original little_r file.
> > > >
> > > > I also find in the MPR file a third instance (triple) of a
report
> > > > (Z2/TMP) from AJMC1 which contains location info in a
combination not
> > > > found in the duplicate reports for AJMC1 which are also
contained in
> > the
> > > > little_r file.
> > > > The lat/long/elev info for this station seems to be a
permutation of
> > > > the location data for the duplicate stations. See the
following
> > > > listings for three different model lead times:
> > > >
> > > > For 02 lead (fcst valid time = 1400Z, 07 FEB) I found the
following
> > > > triple for Z2/TMP obs:
> > > >
> > > > AJMC1   35.73420 -119.14890 993.17383  144.80000  290.55268
>  290.92776
> > >  NA
> > > > (original dupe present in little_r file)
> > > > AJMC1   35.73420 -119.14890 992.55835  151.44901  290.53215
>  290.92776
> > > > NA(triple)
> > > > AJMC1   35.72780 -119.13250 NA         151.44901  288.47229
>  290.37222
> > >  NA
> > > > (original dupe present in little_r file)
> > > >
> > > > For 17 lead: same pattern is repeated
> > > >
> > > > AJMC1    35.73420 -119.14890 999.93866  144.80000   287.72052
> > 284.81668
> > > > NA
> > > > AJMC1    35.73420 -119.14890 999.26752  151.44901   287.69231
> > 284.81668
> > > > NA (triple)
> > > > AJMC1    35.72780 -119.13250 NA         151.44901   285.73105
> > 283.70557
> > > > NA
> > > >
> > > > For 21 lead: same pattern is repeated
> > > >
> > > > AJMC1    35.73420 -119.14890 NA         144.80000   282.80484
> > 282.03888
> > > > NA
> > > > AJMC1    35.72780 -119.13250 NA         151.44901   282.74350
> > 282.59445
> > > > NA
> > > > AJMC1    35.73420 -119.14890 NA         151.44901   282.80484
> > 282.03888
> > > > NA (triple)
> > > >
> > > > For 08 lead: exception  - missing one of the original dupes
> > > >
> > > > AJMC1    35.73420 -119.14890 NA         144.80000   291.99427
> > 292.03888
> > > > NA
> > > > AJMC1    35.73420 -119.14890 NA         151.44901   291.99427
> > 292.03888
> > > > NA (triple)
> > > >
> > > > It appears that Point-Stat is generating a fictitious ob
(triple)
> > > > whose combination of lat/long/elev cannot be traced to any of
the
> > > > input obs in the little_r files.
> > > >
> > > > I've attached one of the MPR text files (02 lead time) to this
email
> > > > for your inspection.
> > > >
> > > > Could you determine why this triple ob shows up in the MPR
file and
> > > > verify that this may or may not be a bug in the Point-Stat
software?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Wednesday, July 02, 2014 1:40 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US);
<jeffrey.a.smith at zianet.com>
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Thanks Brian.  John and I are looking into this, but its..
intriguing
> > > > to say the least.  In a quick scan of my duplicate table for
the
> > > > 02072012 case day (attached), sheet Case_20120207_TMP_Dups for
> > > > OBS_SID=AJMC1, I find triples at FCS_LEAD = 02, 17 and 21.
For
> > > > FCST_LEAD= 01, 03, 04, 08, 11, 13, 15 18, 20, 22, 23 and 24 I
find
> > > > just doubles.  For that station ID, I find a total
> > > > of 33 records.   In each of the triples, two of the
OBS_VALID_END
> times
> > > are
> > > > the same and one is different.  It appears from this limited
sample,
> > > > the one element of the triple you excluded was one of the two
'SAME'
> > > > OBS_VALID_END times.  Furthermore, it appears that OBS should
have
> > > > been either part of the previous hour or the subsequent hour
forecast
> > > > (i.e., not within +/- 15 minutes of the top of the hour).
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Reen, Brian P CIV USARMY ARL (US)
> > > > Sent: Wednesday, July 02, 2014 8:03 AM
> > > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Jeffrey,
> > > > Your choices on which obs to include looks good to me.
> > > >
> > > > Following are some answers regarding the questions I could not
answer
> > > > yesterday on the phone about how WRF deals with potentially
duplicate
> > > obs.
> > > > It is most likely more detailed than you are looking for, but
I
> > > > thought it would be good to have at least for reference sake.
> > > >
> > > > The Obsgrid software deals with quality control and dealing
with
> > > > duplicate obs.  Obsgrid is used both to create the obs files I
gave
> to
> > > > John Raby for ingestion into MET and the obs files I use for
data
> > > > assimilation in WRF.
> > > > Therefore, the obs I provided to you should be the same as the
obs
> > > > that I used for data assimilation in WRF.  When I describe how
> Obsgrid
> > > > works, I'm referring to the version that I use that I have
modified
> > > > from the standard version.
> > > >
> > > > The observation files use a quality control flag that is
additive
> > > > powers of 2.  For example, 2^5=32 means one thing and 2^8=256
means
> > > > another thing, so if the quality control flag is 288=256+32 we
know
> > > > that the condition represented by 32 is true as is the
condition
> > > represented
> > > > by 256 is true.
> > > > The most serious flags are the largest flags.  I have Obsgrid
set to
> > > > not output any obs with a QC flag of 32768 or greater, and WRF
is
> > > > hardcoded to ignore obs with a QC flag of 30000 or greater, so
any
> obs
> > > > in the obs file I gave should be being used by WRF (assuming
that
> they
> > > > fall within the WRF domain). In theory, obs with QC flags
between
> > > > 30000 and 32767 could be included in the obs file but rejected
by
> WRF,
> > > > but I'm doubtful that any obs would have QC flags in that
range.
> > > >
> > > > In Obsgrid it determines obs are duplicates if the following
is true:
> > > > 1) The latitude and longitudes are both less than 0.01 degrees
apart
> > > > 2) The station ID and name match exactly (note that for MADIS
data
> the
> > > > station ID is OBS_SID in your spreadsheet, but the name is
always
> > > > "MADIS")
> > > > 3) The times of the observations are within 30 minutes of one
another
> > > >
> > > > If the obs are duplicate then it tries to merge the two obs
and uses
> > > > the time of the ob closer to the time for which we are looking
for
> obs
> > > > (Obsgrid is being used to create hourly files that contain all
obs
> > > > closest to that hour so this means it will use the time of the
ob
> > > > closest to the current
> > > > hour) as the time of the ob.  For each variable in the ob it
tries to
> > > > pick the best from the two obs.  If does so using the
following:
> > > > 1) Use the non-missing value if one and only one of the obs is
> missing
> > > > 2) Use the ob with the lower QC flag
> > > > 3) Use the overall ob (i.e., all variables and I believe all
levels)
> > > > with more valid observations
> > > > 4) [I don't think the following flags are ever set so I
suspect these
> > > > never come into play] User the overall ob with fewer errors
noted,
> > > > fewer warnings, and higher sequence number
> > > > 5) All else being equal, use the ob that is closer to the
target time
> > > > for which we are looking for an observation
> > > >
> > > > Based on all this, it may appear that stations like AIDC1
should not
> > > > be in the obs file from both CA-Hydro and MesoWest since they
have
> > > > identical lat/lons.  It looks like obs from AIDC1 are not in
the ob
> > > > file from both sources at the same time.  At least for the
February
> > > > 7th case, all but one of the AIDC1 obs is from CA-Hydro, and
at the
> > > > time of the MesoWest AIDC1 ob there is no matching ob from CA-
Hydro.
> > > > So perhaps, CA-Hydro for some reason did not report an ob from
AIDC1
> > > > at that time to MADIS or perhaps the MesoWest version happened
to be
> > > > earlier in the file than the CA-Hydro version at that time,
and the
> > > > observations being identical Obsgrid chose the first ob.
> > > >
> > > > Brian
> > > >
> > > > -----Original Message-----
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Tuesday, July 01, 2014 2:45 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Brian.
> > > >
> > > > Once again, thank you for your time.  I appreciate your help.
> > > >
> > > > From your comments to the original spreadsheet, I added a
column
> > > > called "INCLUDE" that is Boolean -  true to include that data,
false
> if
> > > to
> > > > ignore.
> > > > Universally, if it was a choice between CA-Hydro and another
net, I
> > > > choose CA-Hydro because of the precision in the data.
> > > >
> > > > WRT the WSPC1 and YRKC1 SID's, I will ignore both of them.  We
have
> > > > no idea which, if either, is the correct data so absent better
> > > > information, I think it's better to ignore both.
> > > >
> > > > WRK KIYK, I chose the NonFED one for the same reason as you
> > recommended.
> > > >
> > > > For the greyed out triple values, I'll ignore those and John
and I
> > > > will take a crack at figuring out what is going on there with
MET.
> > > >
> > > > Regards,
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Reen, Brian P CIV USARMY ARL (US)
> > > > Sent: Friday, June 27, 2014 11:47 AM
> > > > To: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: RE: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > > Jeffrey,
> > > > I've went through the spreadsheet and found that whenever
there are
> > > > three different lat/lon/elevation triplets for a given
station, I
> only
> > > > find two of the combinations in the observation files I'm
looking at.
> > > > The third combination, the one I do not have, has a
lat/lon/elevation
> > > > combination that can be constructed from taking 2 of the
fields from
> > > > one of the other triplets and the other field from the other
triplet.
> > > > For example, if we have three listings for a station called
YYY:
> > > > 1. YYY lat=lat1, lon=lon1, elev=elev1
> > > > 2. YYY lat=lat2, lon=lon2, elev=elev2
> > > > 3. YYY lat=lat3, lon=lon3, elev=elev3
> > > > Then if lines 1 and 2 are the versions I have in my file while
I am
> > > > missing version 3, lat3 equals either lat1 or lat2, lon3
equals
> either
> > > > lon1 or lon2,
> > > > elev3 equals either elev1 or elev2 [and line 3 is not a
duplicate of
> > > > either line 1 or line 2]. This suggests that some processing
step may
> > > > be merging the station information in certain cases.  In light
of
> > > > this, in the spreadsheet I grayed out the lines that did not
seem to
> > > > occur in the observation files that I have.
> > > >
> > > > I've added columns showing the source of the observation,
according
> to
> > > > MADIS.  I also added columns showing the difference in
latitude,
> > > > longitude, and elevation between the two versions of each
observation
> > > > location.  I calculated an estimated distance between the two
> > > > locations as well.  I estimated the elevation at the station
location
> > > > according to the 1/3 arc-second NED dataset via
> > > http://ned.usgs.gov/epqs/
> > > > and compared this to
> > > > the elevation reported in the obs file to find an elevation
"error".
> > Of
> > > > course we do not know what the true elevation is exactly at
the
> > > > observation location so this is not really necessarily an
error.
> > > > Finally I put a summary column with some text regarding the
> comparison.
> > > >
> > > > Most of the differences are differences between how CA-Hydro
and
> > > > MesoWest report the lat/lon/elevation of stations. In general
it is
> > > > hard to know which is correct.  Often CA-Hydro matches better
with
> the
> > > > terrain database than MesoWest, but the number of times that
CA-Hydro
> > > > exactly matches the terrain database makes me suspicious that
they
> > > > might be directly pulling it out of the same terrain database.
>  Beside
> > > > the CA-Hydro / MesoWest discrepencies, the following were
found:
> > > >
> > > > For KIYK, the Inyokern Airport in California, OTHER-MTR and
> NonFedAWOS
> > > > report different locations (~1.2km) and elevations (~0.7 m).
Since
> > > > NonFedAWOS reports the values with more precision, perhaps it
is more
> > > > likely to be right?; also, the location it reports is right
off one
> of
> > > > the runways of the airport.
> > > >
> > > > For SFOC1, HADS and CA-Hydro disagree on the location (~670 m)
and
> > > > elevation
> > > > (~6 m).  The elevations reported by the two networks only
differ by
> > > > 8m, but the terrain database estimated elevation for the two
> locations
> > > > differs by
> > > > ~113 m, resulting in the CA-Hydro elevation matching the
terrain
> > > > database estimated elevation much better (~0.01m difference
compared
> to
> > > > 121.08 m).
> > > > This suggests that CA-Hydro is better.
> > > >
> > > > For WSPC1 and YRKC1, RAWS reports identical
latitude/longitude, and
> an
> > > > elevation that differs by 0.9 m.  Although the elevation
matches the
> > > > terrain database at this location pretty well, the location is
far
> > > > from those reported by HADS (by about 110 km for WSPC1 and 156
km for
> > > > YRKC1).  The HADS website lists descriptions of the two sites
> > > > ("Whispering Pines near Middletown 6NW" and "Yorkville 1WNW")
that
> are
> > > > consistent with the HADS
> > > > locations but not the RAWS locations.    This suggests that
the HADS
> > > values
> > > > are correct.  However, note that the HADS elevation value for
WSPC1
> is
> > > > about
> > > > 200 m lower than expected by the terrain database.
> > > >
> > > >
> > > > Brian
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Smith, Jeffrey A CIV USARMY ARL (US)
> > > > Sent: Thursday, June 26, 2014 3:51 PM
> > > > To: Reen, Brian P CIV USARMY ARL (US)
> > > > Cc: Raby, John W CIV USARMY ARL (US)
> > > > Subject: Bad SID Table (UNCLASSIFIED)
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > Brian.
> > > >
> > > > Here are the SID's I was speaking to you about.  They are all
SID's
> > > > that produced at least one multiple ob, in a least one case
hour of
> > > > the five case days.  After John reprocessed the LA Basin data
with
> the
> > > > new switch, I ran my scripts against the data to produce the
table.
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>



------------------------------------------------