[Met_help] [rt.rap.ucar.edu #87676] History for HiRA ECNT and PSTD grid differences in point_stat
John Halley Gotway via RT
met_help at ucar.edu
Fri Nov 9 09:20:35 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello Met Help,
,
I've been playing with the new ECNT output type for HiRA in Metv8.0 (patched up to the start of November).
To test things I have been using a single observation point vs a single forecast at a set location using the hira settings
hira = {
flag = TRUE;
width = [ 1, 3, 5 ];
vld_thresh = 1.0;
cov_thresh = [ ==0.1 ];
shape = SQUARE;
}
The issue that is confusing me is most apparent at HiRA width 1.
I would expect the value of the Mean Error in the ECNT file (for Interp_pnts = 1)
To be the same as in the CNT file (for interp method NEAREST).
After digging into my data ,it looks like the forecast value being used in NBRHD(1) is one grid point to the west of the value being used by CNT
(taken from the value of FCST for the NEAREST line in the MPR output).
I retried my data setting the observation value coordinates to be at the bottom left of my domain to see whether the code would look for a forecast value which was outside the grid and it does seem to be the case, however the probability bit of HiRA in the same run didn't seem to have the same problem and happily found the forecast data.
I've attached some of the output below to indicate the issue,
For NEAREST (1) and for HiRA Probability NBRHD(1) the code successfully generates a pair, but for the HiRA Ensemble NBRHD(1) it does not.
(the larger neighbourhoods correctly reject all pairs because some of the neighbourhood is outside the model domain)
My expectation was that the HiRA probability and ensemble neighbourhoods would be using exactly the same raw forecast data, and so either both types of output would reject the matched pair, or both would accept it.
Am I wrong and the way the ECNT and PSTD output is generated means that the neighbourhood selection is slightly different? Or is there a grid shift between the two output types.
Cheers,
(and hoping this makes sense)
Ric
---------------------------------------------------------------------------------------
Processing votemper(0,0,*,*) versus WTMP/Z0, for observation type SFCSHP, over region FULL, for interpolation method NEAREST(1), using 1 pairs.
DEBUG 3: Number of matched pairs = 1
DEBUG 3: Observations processed = 1
DEBUG 3: Rejected: SID exclusion = 0
DEBUG 3: Rejected: obs type = 0
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 0
DEBUG 3: Rejected: level mismatch = 0
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 0
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2: Computing Categorical Statistics.
DEBUG 2: Computing Multi-Category Statistics.
DEBUG 2: Computing Continuous Statistics.
DEBUG 2: Computing Scalar Partial Sums.
DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for observation type SFCSHP, over region FULL, for interpolation method HiRA Ensemble NBRHD(1), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for observation type SFCSHP, over region FULL, for interpolation method HiRA Ensemble NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for observation type SFCSHP, over region FULL, for interpolation method HiRA Ensemble NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(1), using 1 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(9), using 0 pairs.
DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15, for observation type SFCSHP, over region FULL, for interpolation method HiRA Probability NBRHD(25), using 0 pairs.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Wed Nov 07 08:15:38 2018
Ric,
Thanks for bringing this apparent discrepancy to our attention. I'd
like
to do some detailed testing as you've done to confirm the behavior you
describe and track it down.
Once I clearly understand what is happening and why, we can discuss if
any
changes are required.
Thanks,
John Halley Gotway
On Tue, Nov 6, 2018 at 9:30 AM Crocker, Richard via RT
<met_help at ucar.edu>
wrote:
>
> Tue Nov 06 09:30:17 2018: Request 87676 was acted upon.
> Transaction: Ticket created by ric.crocker at metoffice.gov.uk
> Queue: met_help
> Subject: HiRA ECNT and PSTD grid differences in point_stat
> Owner: Nobody
> Requestors: ric.crocker at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>
>
> Hello Met Help,
>
> ,
> I've been playing with the new ECNT output type for HiRA in Metv8.0
> (patched up to the start of November).
>
> To test things I have been using a single observation point vs a
single
> forecast at a set location using the hira settings
>
> hira = {
> flag = TRUE;
> width = [ 1, 3, 5 ];
> vld_thresh = 1.0;
> cov_thresh = [ ==0.1 ];
> shape = SQUARE;
> }
>
> The issue that is confusing me is most apparent at HiRA width 1.
> I would expect the value of the Mean Error in the ECNT file (for
> Interp_pnts = 1)
> To be the same as in the CNT file (for interp method NEAREST).
>
> After digging into my data ,it looks like the forecast value being
used in
> NBRHD(1) is one grid point to the west of the value being used by
CNT
> (taken from the value of FCST for the NEAREST line in the MPR
output).
>
> I retried my data setting the observation value coordinates to be at
the
> bottom left of my domain to see whether the code would look for a
forecast
> value which was outside the grid and it does seem to be the case,
however
> the probability bit of HiRA in the same run didn't seem to have the
same
> problem and happily found the forecast data.
>
> I've attached some of the output below to indicate the issue,
> For NEAREST (1) and for HiRA Probability NBRHD(1) the code
successfully
> generates a pair, but for the HiRA Ensemble NBRHD(1) it does not.
> (the larger neighbourhoods correctly reject all pairs because some
of the
> neighbourhood is outside the model domain)
>
> My expectation was that the HiRA probability and ensemble
neighbourhoods
> would be using exactly the same raw forecast data, and so either
both types
> of output would reject the matched pair, or both would accept it.
>
> Am I wrong and the way the ECNT and PSTD output is generated means
that
> the neighbourhood selection is slightly different? Or is there a
grid shift
> between the two output types.
>
> Cheers,
> (and hoping this makes sense)
> Ric
>
>
>
>
>
---------------------------------------------------------------------------------------
> Processing votemper(0,0,*,*) versus WTMP/Z0, for observation type
SFCSHP,
> over region FULL, for interpolation method NEAREST(1), using 1
pairs.
> DEBUG 3: Number of matched pairs = 1
> DEBUG 3: Observations processed = 1
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 0
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 0
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2: Computing Categorical Statistics.
> DEBUG 2: Computing Multi-Category Statistics.
> DEBUG 2: Computing Continuous Statistics.
> DEBUG 2: Computing Scalar Partial Sums.
> DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for
observation type
> SFCSHP, over region FULL, for interpolation method HiRA Ensemble
NBRHD(1),
> using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for
observation type
> SFCSHP, over region FULL, for interpolation method HiRA Ensemble
NBRHD(9),
> using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*) versus WTMP/Z0, for
observation type
> SFCSHP, over region FULL, for interpolation method HiRA Ensemble
NBRHD(25),
> using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>279.15 versus WTMP/Z0>279.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>281.15 versus WTMP/Z0>281.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>283.15 versus WTMP/Z0>283.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>285.15 versus WTMP/Z0>285.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>287.15 versus WTMP/Z0>287.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>289.15 versus WTMP/Z0>289.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(1), using 1 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(9), using 0 pairs.
> DEBUG 2: Processing votemper(0,0,*,*)>291.15 versus WTMP/Z0>291.15,
for
> observation type SFCSHP, over region FULL, for interpolation method
HiRA
> Probability NBRHD(25), using 0 pairs.
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Wed Nov 07 10:50:44 2018
Ric,
OK, I did some detailed testing and have convinced myself that I do
not see
the problem you describe.
Here's how I tested...
(1) Write a Python script to create some fake data with integer values
ranging from 0 to 1499 on a 30x50 domain. This is to just ensure that
each
grid point has a different value so I can tell them apart.
(2) Define this fake data on a 1 degree lat/lon grid over the CONUS.
(3) Pick a single lat/lon grid box to test... I chose 41 lat and -92
lon.
(4) Define 4 lat/lon points inside that grid box which are closet to
each
of the 4 corners:
UL 41.75 -92.75
UR 41.75 -92.25
LR 41.25 -92.25
LL 41.25 -92.75
Assign them station ID's to indicate the corner closest to that point:
UL,
UR, LR, and LL
(5) Configure/run Point-Stat to verify against these 4 points,
treating
each as a separate region.
Choose 5 interpolation methods: NEAREST, UPPER_LEFT, UPPER_RIGHT,
LOWER_RIGHT, and LOWER_LEFT.
Note that the fake data values at these corner points happen to be
387,
388, 437, and 438.
Define categorical thresholds to be exactly equal to these values,
and
turn the HiRA method on using a width of 1.
(6) Look through the MPR output lines to make sure that...
- Point-Stat chooses the correct corner point for the NEAREST
interpolation method.
- HiRA chooses the same corner point for width = 1.
As far as I can tell, there are no inconsistencies in the MPR output.
The
choice of model point is consistent between the NEAREST interpolation
method and HiRA with a width of 1. And beyond being consistent, it's
correct.
I've attached the MPR output file I generated for you to inspect. If
it'd
be helpful, I could tar up my whole test directory.
But perhaps the testing I've done will suffice. Just let me know.
Thanks,
John
On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: HiRA ECNT and PSTD grid differences in point_stat
> Owner: johnhg
> Requestors: ric.crocker at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Wed Nov 07 10:50:44 2018
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL
OBS_ELV FCST OBS OBS_QC CLIMO_MEAN CLIMO_STDEV CLIMO_CDF
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL NEAREST 1 NA NA NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
387 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL UPPER_LEFT 1 NA NA NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
387 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL UPPER_RIGHT 1 NA NA NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
388 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL LOWER_RIGHT 1 NA NA NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
438 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL LOWER_LEFT 1 NA NA NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
437 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL NBRHD_SQUARE 1 ==387 ==387 NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
1 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL NBRHD_SQUARE 1 ==388 ==388 NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
0 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL NBRHD_SQUARE 1 ==437 ==437 NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
0 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UL NBRHD_SQUARE 1 ==438 ==438 NA
NA MPR 1 1 UL 41.75 -92.75 NA 1618
0 1 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR NEAREST 1 NA NA NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
388 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR UPPER_LEFT 1 NA NA NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
387 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR UPPER_RIGHT 1 NA NA NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
388 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR LOWER_RIGHT 1 NA NA NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
438 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR LOWER_LEFT 1 NA NA NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
437 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR NBRHD_SQUARE 1 ==387 ==387 NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
0 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR NBRHD_SQUARE 1 ==388 ==388 NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
1 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR NBRHD_SQUARE 1 ==437 ==437 NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
0 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC UR NBRHD_SQUARE 1 ==438 ==438 NA
NA MPR 1 1 UR 41.75 -92.25 NA 1618
0 2 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR NEAREST 1 NA NA NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
438 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR UPPER_LEFT 1 NA NA NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
387 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR UPPER_RIGHT 1 NA NA NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
388 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR LOWER_RIGHT 1 NA NA NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
438 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR LOWER_LEFT 1 NA NA NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
437 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR NBRHD_SQUARE 1 ==387 ==387 NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
0 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR NBRHD_SQUARE 1 ==388 ==388 NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
0 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR NBRHD_SQUARE 1 ==437 ==437 NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
0 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LR NBRHD_SQUARE 1 ==438 ==438 NA
NA MPR 1 1 LR 41.25 -92.25 NA 1618
1 3 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL NEAREST 1 NA NA NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
437 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL UPPER_LEFT 1 NA NA NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
387 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL UPPER_RIGHT 1 NA NA NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
388 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL LOWER_RIGHT 1 NA NA NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
438 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL LOWER_LEFT 1 NA NA NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
437 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL NBRHD_SQUARE 1 ==387 ==387 NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
0 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL NBRHD_SQUARE 1 ==388 ==388 NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
0 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL NBRHD_SQUARE 1 ==437 ==437 NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
1 4 NA NA NA NA
V8.0 WRF NA 000000 20070331_120000 20070331_120000 000000
20070331_120000 20070331_120000 TEST Surface TMP Z2
ADPSFC LL NBRHD_SQUARE 1 ==438 ==438 NA
NA MPR 1 1 LL 41.25 -92.75 NA 1618
0 4 NA NA NA NA
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid differences in point_stat
From: Crocker, Richard
Time: Thu Nov 08 05:29:59 2018
John,
Thanks for having a look at this,
It does look like HiRA and Nearest methods pick the same data, which
is reassuring.
I have tried replicating your method to check whether it was just user
error on my part and I do get the same outcome for the MPR lines
The question I now have comes when I look at the Mean Error output
values for the ECNT and CNT output with this test
Given that Nearest and NBOUR for a single point select the same grid
points,
Why do the values of ME in the CNT and ECNT files differ. (That is to
say they differ in my test, but do you get the same thing?)
The ME value in the CNT file gives what I would expect from the
NEAREST data, but the ME in the ECNT output is different (and I guess
I expect it to be the same when looking at a single grid point value).
I know there is an element for forecast mean in the ECNT output for
larger neighbourhoods, but for a single grid point this should be
irrelevant.
What I did was similar to you, a 50x50 grid with values increasing
from 0 to 2499 and a 1degree domain 0:50 latitude and 50:99 longitude.
Working on a centre of 25,52 I used 4 obs offset as you did, with all
obs values set to 0 (to remove the impact of the obs value from the
mean error calculation)
I get the following 4 values for nearest (and equivalent matches in
the nbour1 if I set thresholds to those forecast values) in the
matched pairs file,
TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV FCST OBS
4 1 00001 25.25 52.25 NA 0 1252 0
4 2 00002 25.25 52.75 NA 0 1253 0
4 3 00003 25.75 52.25 NA 0 1302 0
4 4 00004 25.75 52.75 NA 0 1303 0
In the ECNT file I get
TOTAL N_ENS ME
4 1 1252
But in the CNT files I get
ME
1277.5 which is the average of the 4 nearest forecast values above as
expected
If I rerun 4 times, just using each of the four observations on
their own (in the order above) I get the following mean error values
from the ECNT and CNT files:
ECNT CNT
1252 1252
1252 1253
1252 1302
1252 1303
So the ECNT forecast mean appears not to be changing to use a
different forecast value/point despite the CNT value matching the
NEAREST and NBOUR_1 forecast values.
Are you able to replicate this?
(and is it important, and am I fundamentally misunderstanding what the
mean error is in the ECNT file for point_stat/HiRA)
Many thanks,
Ric
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 07 November 2018 17:51
To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
differences in point_stat
Ric,
OK, I did some detailed testing and have convinced myself that I do
not see the problem you describe.
Here's how I tested...
(1) Write a Python script to create some fake data with integer values
ranging from 0 to 1499 on a 30x50 domain. This is to just ensure that
each grid point has a different value so I can tell them apart.
(2) Define this fake data on a 1 degree lat/lon grid over the CONUS.
(3) Pick a single lat/lon grid box to test... I chose 41 lat and -92
lon.
(4) Define 4 lat/lon points inside that grid box which are closet to
each of the 4 corners:
UL 41.75 -92.75
UR 41.75 -92.25
LR 41.25 -92.25
LL 41.25 -92.75
Assign them station ID's to indicate the corner closest to that point:
UL, UR, LR, and LL
(5) Configure/run Point-Stat to verify against these 4 points,
treating each as a separate region.
Choose 5 interpolation methods: NEAREST, UPPER_LEFT, UPPER_RIGHT,
LOWER_RIGHT, and LOWER_LEFT.
Note that the fake data values at these corner points happen to be
387, 388, 437, and 438.
Define categorical thresholds to be exactly equal to these values,
and turn the HiRA method on using a width of 1.
(6) Look through the MPR output lines to make sure that...
- Point-Stat chooses the correct corner point for the NEAREST
interpolation method.
- HiRA chooses the same corner point for width = 1.
As far as I can tell, there are no inconsistencies in the MPR output.
The choice of model point is consistent between the NEAREST
interpolation method and HiRA with a width of 1. And beyond being
consistent, it's correct.
I've attached the MPR output file I generated for you to inspect. If
it'd be helpful, I could tar up my whole test directory.
But perhaps the testing I've done will suffice. Just let me know.
Thanks,
John
On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: HiRA ECNT and PSTD grid differences in point_stat
> Owner: johnhg
> Requestors: ric.crocker at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
> >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Thu Nov 08 09:14:41 2018
Ric,
Ah OK, I see the behavior you describe in my test case. I see that
the ME
value from the ECNT line type appears to be computed using the
forecast
value from the lower-left corner in all cases.
I'll need to run this case through the debugger to determine why this
occurs and how to fix it. I'll let you know what I find.
Thanks,
John
On Thu, Nov 8, 2018 at 5:30 AM Crocker, Richard via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>
> John,
> Thanks for having a look at this,
> It does look like HiRA and Nearest methods pick the same data, which
is
> reassuring.
>
> I have tried replicating your method to check whether it was just
user
> error on my part and I do get the same outcome for the MPR lines
>
> The question I now have comes when I look at the Mean Error output
values
> for the ECNT and CNT output with this test
> Given that Nearest and NBOUR for a single point select the same grid
> points,
> Why do the values of ME in the CNT and ECNT files differ. (That is
to say
> they differ in my test, but do you get the same thing?)
> The ME value in the CNT file gives what I would expect from the
NEAREST
> data, but the ME in the ECNT output is different (and I guess I
expect it
> to be the same when looking at a single grid point value). I know
there is
> an element for forecast mean in the ECNT output for larger
neighbourhoods,
> but for a single grid point this should be irrelevant.
>
> What I did was similar to you, a 50x50 grid with values increasing
from 0
> to 2499 and a 1degree domain 0:50 latitude and 50:99 longitude.
>
> Working on a centre of 25,52 I used 4 obs offset as you did, with
all obs
> values set to 0 (to remove the impact of the obs value from the mean
error
> calculation)
> I get the following 4 values for nearest (and equivalent matches in
the
> nbour1 if I set thresholds to those forecast values) in the matched
pairs
> file,
> TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV FCST OBS
> 4 1 00001 25.25 52.25 NA 0 1252 0
> 4 2 00002 25.25 52.75 NA 0 1253 0
> 4 3 00003 25.75 52.25 NA 0 1302 0
> 4 4 00004 25.75 52.75 NA 0 1303 0
>
> In the ECNT file I get
> TOTAL N_ENS ME
> 4 1 1252
>
> But in the CNT files I get
> ME
> 1277.5 which is the average of the 4 nearest forecast values above
as
> expected
>
> If I rerun 4 times, just using each of the four observations on
their
> own (in the order above) I get the following mean error values from
the
> ECNT and CNT files:
> ECNT CNT
> 1252 1252
> 1252 1253
> 1252 1302
> 1252 1303
> So the ECNT forecast mean appears not to be changing to use a
different
> forecast value/point despite the CNT value matching the NEAREST and
NBOUR_1
> forecast values.
>
>
> Are you able to replicate this?
> (and is it important, and am I fundamentally misunderstanding what
the
> mean error is in the ECNT file for point_stat/HiRA)
>
> Many thanks,
> Ric
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 07 November 2018 17:51
> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
differences
> in point_stat
>
> Ric,
>
> OK, I did some detailed testing and have convinced myself that I do
not
> see the problem you describe.
>
> Here's how I tested...
>
> (1) Write a Python script to create some fake data with integer
values
> ranging from 0 to 1499 on a 30x50 domain. This is to just ensure
that each
> grid point has a different value so I can tell them apart.
> (2) Define this fake data on a 1 degree lat/lon grid over the CONUS.
> (3) Pick a single lat/lon grid box to test... I chose 41 lat and -92
lon.
> (4) Define 4 lat/lon points inside that grid box which are closet to
each
> of the 4 corners:
>
> UL 41.75 -92.75
>
> UR 41.75 -92.25
>
> LR 41.25 -92.25
>
> LL 41.25 -92.75
>
> Assign them station ID's to indicate the corner closest to that
point: UL,
> UR, LR, and LL
>
> (5) Configure/run Point-Stat to verify against these 4 points,
treating
> each as a separate region.
>
> Choose 5 interpolation methods: NEAREST, UPPER_LEFT, UPPER_RIGHT,
> LOWER_RIGHT, and LOWER_LEFT.
>
> Note that the fake data values at these corner points happen to be
387,
> 388, 437, and 438.
>
> Define categorical thresholds to be exactly equal to these values,
and
> turn the HiRA method on using a width of 1.
>
> (6) Look through the MPR output lines to make sure that...
>
> - Point-Stat chooses the correct corner point for the NEAREST
> interpolation method.
>
> - HiRA chooses the same corner point for width = 1.
>
>
> As far as I can tell, there are no inconsistencies in the MPR
output. The
> choice of model point is consistent between the NEAREST
interpolation
> method and HiRA with a width of 1. And beyond being consistent,
it's
> correct.
>
>
> I've attached the MPR output file I generated for you to inspect.
If it'd
> be helpful, I could tar up my whole test directory.
>
>
> But perhaps the testing I've done will suffice. Just let me know.
>
>
> Thanks,
> John
>
>
> On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > Queue: met_help
> > Subject: HiRA ECNT and PSTD grid differences in point_stat
> > Owner: johnhg
> > Requestors: ric.crocker at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
> > >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Thu Nov 08 09:57:05 2018
Ric,
I found it. At one point in the logic, we are inadvertently casting
floating point grid x/y values to integers... which truncates rather
than
rounding. I'm surprised I didn't get a compiler warning about this!
Adding a new signature for that function to first round those floating
points to the nearest integers solves the problem in my test case.
So this is definitely a bug which we should patch for met-8.0. Before
posting the bugfix, I want to re-run our regression test to quantify
what
impact it has on the output. But I'll write when the patch is
available.
Thanks for digging through the details and finding this!
John
*(1) From the MPR lines, select the NEAREST and look at FCST - OBS:*
VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE FCST OBS
UL NEAREST 1 MPR 387 1
UR NEAREST 1 MPR 388 2
LR NEAREST 1 MPR 438 3
LL NEAREST 1 MPR 437 4
*(2) From the CNT lines, select the ME column... and these values
match
MPR:FCST - MPR:OBS:*
VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
UL NEAREST 1 CNT 386
UR NEAREST 1 CNT 386
LR NEAREST 1 CNT 435
LL NEAREST 1 CNT 433
*(3) From the ECNT lines, select the ME column:*
**** BEFORE THE BUGFIX, ITS WRONG (always using the FCST LL corner
value of
437) ****
VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
UL NBRHD_SQUARE 1 ECNT *436*
UR NBRHD_SQUARE 1 ECNT *435*
LR NBRHD_SQUARE 1 ECNT *434*
LL NBRHD_SQUARE 1 ECNT *433*
****AFTER THE BUGFIX, ITS CORRECT (exactly matches the CNT:ME column)
****
VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
UL NBRHD_SQUARE 1 ECNT 386
UR NBRHD_SQUARE 1 ECNT 386
LR NBRHD_SQUARE 1 ECNT 435
LL NBRHD_SQUARE 1 ECNT 433
On Thu, Nov 8, 2018 at 9:14 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ric,
>
> Ah OK, I see the behavior you describe in my test case. I see that
the ME
> value from the ECNT line type appears to be computed using the
forecast
> value from the lower-left corner in all cases.
>
> I'll need to run this case through the debugger to determine why
this
> occurs and how to fix it. I'll let you know what I find.
>
> Thanks,
> John
>
> On Thu, Nov 8, 2018 at 5:30 AM Crocker, Richard via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>>
>> John,
>> Thanks for having a look at this,
>> It does look like HiRA and Nearest methods pick the same data,
which is
>> reassuring.
>>
>> I have tried replicating your method to check whether it was just
user
>> error on my part and I do get the same outcome for the MPR lines
>>
>> The question I now have comes when I look at the Mean Error output
values
>> for the ECNT and CNT output with this test
>> Given that Nearest and NBOUR for a single point select the same
grid
>> points,
>> Why do the values of ME in the CNT and ECNT files differ. (That is
to say
>> they differ in my test, but do you get the same thing?)
>> The ME value in the CNT file gives what I would expect from the
NEAREST
>> data, but the ME in the ECNT output is different (and I guess I
expect it
>> to be the same when looking at a single grid point value). I know
there is
>> an element for forecast mean in the ECNT output for larger
neighbourhoods,
>> but for a single grid point this should be irrelevant.
>>
>> What I did was similar to you, a 50x50 grid with values increasing
from 0
>> to 2499 and a 1degree domain 0:50 latitude and 50:99 longitude.
>>
>> Working on a centre of 25,52 I used 4 obs offset as you did, with
all obs
>> values set to 0 (to remove the impact of the obs value from the
mean error
>> calculation)
>> I get the following 4 values for nearest (and equivalent matches in
the
>> nbour1 if I set thresholds to those forecast values) in the matched
pairs
>> file,
>> TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV FCST OBS
>> 4 1 00001 25.25 52.25 NA 0 1252 0
>> 4 2 00002 25.25 52.75 NA 0 1253 0
>> 4 3 00003 25.75 52.25 NA 0 1302 0
>> 4 4 00004 25.75 52.75 NA 0 1303 0
>>
>> In the ECNT file I get
>> TOTAL N_ENS ME
>> 4 1 1252
>>
>> But in the CNT files I get
>> ME
>> 1277.5 which is the average of the 4 nearest forecast values above
as
>> expected
>>
>> If I rerun 4 times, just using each of the four observations on
their
>> own (in the order above) I get the following mean error values from
the
>> ECNT and CNT files:
>> ECNT CNT
>> 1252 1252
>> 1252 1253
>> 1252 1302
>> 1252 1303
>> So the ECNT forecast mean appears not to be changing to use a
different
>> forecast value/point despite the CNT value matching the NEAREST and
NBOUR_1
>> forecast values.
>>
>>
>> Are you able to replicate this?
>> (and is it important, and am I fundamentally misunderstanding what
the
>> mean error is in the ECNT file for point_stat/HiRA)
>>
>> Many thanks,
>> Ric
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: 07 November 2018 17:51
>> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
>> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
>> differences in point_stat
>>
>> Ric,
>>
>> OK, I did some detailed testing and have convinced myself that I do
not
>> see the problem you describe.
>>
>> Here's how I tested...
>>
>> (1) Write a Python script to create some fake data with integer
values
>> ranging from 0 to 1499 on a 30x50 domain. This is to just ensure
that each
>> grid point has a different value so I can tell them apart.
>> (2) Define this fake data on a 1 degree lat/lon grid over the
CONUS.
>> (3) Pick a single lat/lon grid box to test... I chose 41 lat and
-92 lon.
>> (4) Define 4 lat/lon points inside that grid box which are closet
to each
>> of the 4 corners:
>>
>> UL 41.75 -92.75
>>
>> UR 41.75 -92.25
>>
>> LR 41.25 -92.25
>>
>> LL 41.25 -92.75
>>
>> Assign them station ID's to indicate the corner closest to that
point:
>> UL, UR, LR, and LL
>>
>> (5) Configure/run Point-Stat to verify against these 4 points,
treating
>> each as a separate region.
>>
>> Choose 5 interpolation methods: NEAREST, UPPER_LEFT, UPPER_RIGHT,
>> LOWER_RIGHT, and LOWER_LEFT.
>>
>> Note that the fake data values at these corner points happen to
be 387,
>> 388, 437, and 438.
>>
>> Define categorical thresholds to be exactly equal to these
values, and
>> turn the HiRA method on using a width of 1.
>>
>> (6) Look through the MPR output lines to make sure that...
>>
>> - Point-Stat chooses the correct corner point for the NEAREST
>> interpolation method.
>>
>> - HiRA chooses the same corner point for width = 1.
>>
>>
>> As far as I can tell, there are no inconsistencies in the MPR
output.
>> The choice of model point is consistent between the NEAREST
interpolation
>> method and HiRA with a width of 1. And beyond being consistent,
it's
>> correct.
>>
>>
>> I've attached the MPR output file I generated for you to inspect.
If
>> it'd be helpful, I could tar up my whole test directory.
>>
>>
>> But perhaps the testing I've done will suffice. Just let me know.
>>
>>
>> Thanks,
>> John
>>
>>
>> On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
>> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> > Queue: met_help
>> > Subject: HiRA ECNT and PSTD grid differences in point_stat
>> > Owner: johnhg
>> > Requestors: ric.crocker at metoffice.gov.uk
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
>> > >
>> >
>> >
>> > This transaction appears to have no content
>> >
>>
>>
>>
>>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Thu Nov 08 16:53:52 2018
Ric,
OK, I just posted the bugfix for this issue to the MET website:
https://dtcenter.org/met/users/support/known_issues/METv8.0/index.php
Rerunning the regression test revealed differences in the ECNT line
type.
But the differences only show up in the ECNT output from Point-Stat
for
HiRA. So MET is only calling the problematic library utility in that
context.
I expanded my testing to include widths of 1 and 2. And I made sure
that
with 1, we're picking the *correct* grid point. And with 2, the mean
value
is computed using the 4 closest grid points.
Please let me know if you see inconsistent or questionable results
after
the patch.
Thanks,
John
On Thu, Nov 8, 2018 at 9:56 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ric,
>
> I found it. At one point in the logic, we are inadvertently casting
> floating point grid x/y values to integers... which truncates rather
than
> rounding. I'm surprised I didn't get a compiler warning about this!
>
> Adding a new signature for that function to first round those
floating
> points to the nearest integers solves the problem in my test case.
>
> So this is definitely a bug which we should patch for met-8.0.
Before
> posting the bugfix, I want to re-run our regression test to quantify
what
> impact it has on the output. But I'll write when the patch is
available.
>
> Thanks for digging through the details and finding this!
>
> John
>
> *(1) From the MPR lines, select the NEAREST and look at FCST - OBS:*
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE FCST OBS
>
> UL NEAREST 1 MPR 387 1
>
> UR NEAREST 1 MPR 388 2
>
> LR NEAREST 1 MPR 438 3
>
> LL NEAREST 1 MPR 437 4
>
> *(2) From the CNT lines, select the ME column... and these values
match
> MPR:FCST - MPR:OBS:*
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NEAREST 1 CNT 386
>
> UR NEAREST 1 CNT 386
>
> LR NEAREST 1 CNT 435
>
> LL NEAREST 1 CNT 433
>
>
> *(3) From the ECNT lines, select the ME column:*
>
> **** BEFORE THE BUGFIX, ITS WRONG (always using the FCST LL corner
value
> of 437) ****
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NBRHD_SQUARE 1 ECNT *436*
>
> UR NBRHD_SQUARE 1 ECNT *435*
>
> LR NBRHD_SQUARE 1 ECNT *434*
>
> LL NBRHD_SQUARE 1 ECNT *433*
>
>
> ****AFTER THE BUGFIX, ITS CORRECT (exactly matches the CNT:ME
column) ****
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NBRHD_SQUARE 1 ECNT 386
>
> UR NBRHD_SQUARE 1 ECNT 386
>
> LR NBRHD_SQUARE 1 ECNT 435
>
> LL NBRHD_SQUARE 1 ECNT 433
>
>
> On Thu, Nov 8, 2018 at 9:14 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Ric,
>>
>> Ah OK, I see the behavior you describe in my test case. I see that
the
>> ME value from the ECNT line type appears to be computed using the
forecast
>> value from the lower-left corner in all cases.
>>
>> I'll need to run this case through the debugger to determine why
this
>> occurs and how to fix it. I'll let you know what I find.
>>
>> Thanks,
>> John
>>
>> On Thu, Nov 8, 2018 at 5:30 AM Crocker, Richard via RT
<met_help at ucar.edu>
>> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>>>
>>> John,
>>> Thanks for having a look at this,
>>> It does look like HiRA and Nearest methods pick the same data,
which is
>>> reassuring.
>>>
>>> I have tried replicating your method to check whether it was just
user
>>> error on my part and I do get the same outcome for the MPR lines
>>>
>>> The question I now have comes when I look at the Mean Error output
>>> values for the ECNT and CNT output with this test
>>> Given that Nearest and NBOUR for a single point select the same
grid
>>> points,
>>> Why do the values of ME in the CNT and ECNT files differ. (That is
to
>>> say they differ in my test, but do you get the same thing?)
>>> The ME value in the CNT file gives what I would expect from the
NEAREST
>>> data, but the ME in the ECNT output is different (and I guess I
expect it
>>> to be the same when looking at a single grid point value). I know
there is
>>> an element for forecast mean in the ECNT output for larger
neighbourhoods,
>>> but for a single grid point this should be irrelevant.
>>>
>>> What I did was similar to you, a 50x50 grid with values increasing
from
>>> 0 to 2499 and a 1degree domain 0:50 latitude and 50:99 longitude.
>>>
>>> Working on a centre of 25,52 I used 4 obs offset as you did, with
all
>>> obs values set to 0 (to remove the impact of the obs value from
the mean
>>> error calculation)
>>> I get the following 4 values for nearest (and equivalent matches
in the
>>> nbour1 if I set thresholds to those forecast values) in the
matched pairs
>>> file,
>>> TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV FCST OBS
>>> 4 1 00001 25.25 52.25 NA 0 1252 0
>>> 4 2 00002 25.25 52.75 NA 0 1253 0
>>> 4 3 00003 25.75 52.25 NA 0 1302 0
>>> 4 4 00004 25.75 52.75 NA 0 1303 0
>>>
>>> In the ECNT file I get
>>> TOTAL N_ENS ME
>>> 4 1 1252
>>>
>>> But in the CNT files I get
>>> ME
>>> 1277.5 which is the average of the 4 nearest forecast values
above as
>>> expected
>>>
>>> If I rerun 4 times, just using each of the four observations on
their
>>> own (in the order above) I get the following mean error values
from the
>>> ECNT and CNT files:
>>> ECNT CNT
>>> 1252 1252
>>> 1252 1253
>>> 1252 1302
>>> 1252 1303
>>> So the ECNT forecast mean appears not to be changing to use a
different
>>> forecast value/point despite the CNT value matching the NEAREST
and NBOUR_1
>>> forecast values.
>>>
>>>
>>> Are you able to replicate this?
>>> (and is it important, and am I fundamentally misunderstanding what
the
>>> mean error is in the ECNT file for point_stat/HiRA)
>>>
>>> Many thanks,
>>> Ric
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway via RT <met_help at ucar.edu>
>>> Sent: 07 November 2018 17:51
>>> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
>>> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
>>> differences in point_stat
>>>
>>> Ric,
>>>
>>> OK, I did some detailed testing and have convinced myself that I
do not
>>> see the problem you describe.
>>>
>>> Here's how I tested...
>>>
>>> (1) Write a Python script to create some fake data with integer
values
>>> ranging from 0 to 1499 on a 30x50 domain. This is to just ensure
that each
>>> grid point has a different value so I can tell them apart.
>>> (2) Define this fake data on a 1 degree lat/lon grid over the
CONUS.
>>> (3) Pick a single lat/lon grid box to test... I chose 41 lat and
-92 lon.
>>> (4) Define 4 lat/lon points inside that grid box which are closet
to
>>> each of the 4 corners:
>>>
>>> UL 41.75 -92.75
>>>
>>> UR 41.75 -92.25
>>>
>>> LR 41.25 -92.25
>>>
>>> LL 41.25 -92.75
>>>
>>> Assign them station ID's to indicate the corner closest to that
point:
>>> UL, UR, LR, and LL
>>>
>>> (5) Configure/run Point-Stat to verify against these 4 points,
treating
>>> each as a separate region.
>>>
>>> Choose 5 interpolation methods: NEAREST, UPPER_LEFT,
UPPER_RIGHT,
>>> LOWER_RIGHT, and LOWER_LEFT.
>>>
>>> Note that the fake data values at these corner points happen to
be
>>> 387, 388, 437, and 438.
>>>
>>> Define categorical thresholds to be exactly equal to these
values, and
>>> turn the HiRA method on using a width of 1.
>>>
>>> (6) Look through the MPR output lines to make sure that...
>>>
>>> - Point-Stat chooses the correct corner point for the NEAREST
>>> interpolation method.
>>>
>>> - HiRA chooses the same corner point for width = 1.
>>>
>>>
>>> As far as I can tell, there are no inconsistencies in the MPR
output.
>>> The choice of model point is consistent between the NEAREST
interpolation
>>> method and HiRA with a width of 1. And beyond being consistent,
it's
>>> correct.
>>>
>>>
>>> I've attached the MPR output file I generated for you to inspect.
If
>>> it'd be helpful, I could tar up my whole test directory.
>>>
>>>
>>> But perhaps the testing I've done will suffice. Just let me know.
>>>
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
>>> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
>>> > Queue: met_help
>>> > Subject: HiRA ECNT and PSTD grid differences in point_stat
>>> > Owner: johnhg
>>> > Requestors: ric.crocker at metoffice.gov.uk
>>> > Status: new
>>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
>>> > >
>>> >
>>> >
>>> > This transaction appears to have no content
>>> >
>>>
>>>
>>>
>>>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid differences in point_stat
From: Crocker, Richard
Time: Fri Nov 09 02:55:16 2018
John,
Thanks for the super-fast fix for this.
I'll patch at our end and do a few tests,
Thanks,
Ric
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 08 November 2018 23:54
To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
differences in point_stat
Ric,
OK, I just posted the bugfix for this issue to the MET website:
https://dtcenter.org/met/users/support/known_issues/METv8.0/index.php
Rerunning the regression test revealed differences in the ECNT line
type.
But the differences only show up in the ECNT output from Point-Stat
for HiRA. So MET is only calling the problematic library utility in
that context.
I expanded my testing to include widths of 1 and 2. And I made sure
that with 1, we're picking the *correct* grid point. And with 2, the
mean value is computed using the 4 closest grid points.
Please let me know if you see inconsistent or questionable results
after the patch.
Thanks,
John
On Thu, Nov 8, 2018 at 9:56 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ric,
>
> I found it. At one point in the logic, we are inadvertently casting
> floating point grid x/y values to integers... which truncates rather
> than rounding. I'm surprised I didn't get a compiler warning about
this!
>
> Adding a new signature for that function to first round those
floating
> points to the nearest integers solves the problem in my test case.
>
> So this is definitely a bug which we should patch for met-8.0.
Before
> posting the bugfix, I want to re-run our regression test to quantify
> what impact it has on the output. But I'll write when the patch is
available.
>
> Thanks for digging through the details and finding this!
>
> John
>
> *(1) From the MPR lines, select the NEAREST and look at FCST - OBS:*
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE FCST OBS
>
> UL NEAREST 1 MPR 387 1
>
> UR NEAREST 1 MPR 388 2
>
> LR NEAREST 1 MPR 438 3
>
> LL NEAREST 1 MPR 437 4
>
> *(2) From the CNT lines, select the ME column... and these values
> match MPR:FCST - MPR:OBS:*
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NEAREST 1 CNT 386
>
> UR NEAREST 1 CNT 386
>
> LR NEAREST 1 CNT 435
>
> LL NEAREST 1 CNT 433
>
>
> *(3) From the ECNT lines, select the ME column:*
>
> **** BEFORE THE BUGFIX, ITS WRONG (always using the FCST LL corner
> value of 437) ****
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NBRHD_SQUARE 1 ECNT *436*
>
> UR NBRHD_SQUARE 1 ECNT *435*
>
> LR NBRHD_SQUARE 1 ECNT *434*
>
> LL NBRHD_SQUARE 1 ECNT *433*
>
>
> ****AFTER THE BUGFIX, ITS CORRECT (exactly matches the CNT:ME
column)
> ****
>
> VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
>
> UL NBRHD_SQUARE 1 ECNT 386
>
> UR NBRHD_SQUARE 1 ECNT 386
>
> LR NBRHD_SQUARE 1 ECNT 435
>
> LL NBRHD_SQUARE 1 ECNT 433
>
>
> On Thu, Nov 8, 2018 at 9:14 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Ric,
>>
>> Ah OK, I see the behavior you describe in my test case. I see that
>> the ME value from the ECNT line type appears to be computed using
the
>> forecast value from the lower-left corner in all cases.
>>
>> I'll need to run this case through the debugger to determine why
this
>> occurs and how to fix it. I'll let you know what I find.
>>
>> Thanks,
>> John
>>
>> On Thu, Nov 8, 2018 at 5:30 AM Crocker, Richard via RT
>> <met_help at ucar.edu>
>> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>>>
>>> John,
>>> Thanks for having a look at this,
>>> It does look like HiRA and Nearest methods pick the same data,
which
>>> is reassuring.
>>>
>>> I have tried replicating your method to check whether it was just
>>> user error on my part and I do get the same outcome for the MPR
>>> lines
>>>
>>> The question I now have comes when I look at the Mean Error output
>>> values for the ECNT and CNT output with this test Given that
>>> Nearest and NBOUR for a single point select the same grid points,
>>> Why do the values of ME in the CNT and ECNT files differ. (That is
>>> to say they differ in my test, but do you get the same thing?) The
>>> ME value in the CNT file gives what I would expect from the
NEAREST
>>> data, but the ME in the ECNT output is different (and I guess I
>>> expect it to be the same when looking at a single grid point
value).
>>> I know there is an element for forecast mean in the ECNT output
for
>>> larger neighbourhoods, but for a single grid point this should be
irrelevant.
>>>
>>> What I did was similar to you, a 50x50 grid with values increasing
>>> from
>>> 0 to 2499 and a 1degree domain 0:50 latitude and 50:99 longitude.
>>>
>>> Working on a centre of 25,52 I used 4 obs offset as you did, with
>>> all obs values set to 0 (to remove the impact of the obs value
from
>>> the mean error calculation) I get the following 4 values for
nearest
>>> (and equivalent matches in the
>>> nbour1 if I set thresholds to those forecast values) in the
matched
>>> pairs file, TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV
FCST
>>> OBS
>>> 4 1 00001 25.25 52.25 NA 0 1252 0
>>> 4 2 00002 25.25 52.75 NA 0 1253 0
>>> 4 3 00003 25.75 52.25 NA 0 1302 0
>>> 4 4 00004 25.75 52.75 NA 0 1303 0
>>>
>>> In the ECNT file I get
>>> TOTAL N_ENS ME
>>> 4 1 1252
>>>
>>> But in the CNT files I get
>>> ME
>>> 1277.5 which is the average of the 4 nearest forecast values
above
>>> as expected
>>>
>>> If I rerun 4 times, just using each of the four observations on
>>> their own (in the order above) I get the following mean error
values
>>> from the ECNT and CNT files:
>>> ECNT CNT
>>> 1252 1252
>>> 1252 1253
>>> 1252 1302
>>> 1252 1303
>>> So the ECNT forecast mean appears not to be changing to use a
>>> different forecast value/point despite the CNT value matching the
>>> NEAREST and NBOUR_1 forecast values.
>>>
>>>
>>> Are you able to replicate this?
>>> (and is it important, and am I fundamentally misunderstanding what
>>> the mean error is in the ECNT file for point_stat/HiRA)
>>>
>>> Many thanks,
>>> Ric
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway via RT <met_help at ucar.edu>
>>> Sent: 07 November 2018 17:51
>>> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
>>> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
>>> differences in point_stat
>>>
>>> Ric,
>>>
>>> OK, I did some detailed testing and have convinced myself that I
do
>>> not see the problem you describe.
>>>
>>> Here's how I tested...
>>>
>>> (1) Write a Python script to create some fake data with integer
>>> values ranging from 0 to 1499 on a 30x50 domain. This is to just
>>> ensure that each grid point has a different value so I can tell
them apart.
>>> (2) Define this fake data on a 1 degree lat/lon grid over the
CONUS.
>>> (3) Pick a single lat/lon grid box to test... I chose 41 lat and
-92 lon.
>>> (4) Define 4 lat/lon points inside that grid box which are closet
to
>>> each of the 4 corners:
>>>
>>> UL 41.75 -92.75
>>>
>>> UR 41.75 -92.25
>>>
>>> LR 41.25 -92.25
>>>
>>> LL 41.25 -92.75
>>>
>>> Assign them station ID's to indicate the corner closest to that
point:
>>> UL, UR, LR, and LL
>>>
>>> (5) Configure/run Point-Stat to verify against these 4 points,
>>> treating each as a separate region.
>>>
>>> Choose 5 interpolation methods: NEAREST, UPPER_LEFT,
UPPER_RIGHT,
>>> LOWER_RIGHT, and LOWER_LEFT.
>>>
>>> Note that the fake data values at these corner points happen to
be
>>> 387, 388, 437, and 438.
>>>
>>> Define categorical thresholds to be exactly equal to these
values,
>>> and turn the HiRA method on using a width of 1.
>>>
>>> (6) Look through the MPR output lines to make sure that...
>>>
>>> - Point-Stat chooses the correct corner point for the NEAREST
>>> interpolation method.
>>>
>>> - HiRA chooses the same corner point for width = 1.
>>>
>>>
>>> As far as I can tell, there are no inconsistencies in the MPR
output.
>>> The choice of model point is consistent between the NEAREST
>>> interpolation method and HiRA with a width of 1. And beyond being
>>> consistent, it's correct.
>>>
>>>
>>> I've attached the MPR output file I generated for you to inspect.
>>> If it'd be helpful, I could tar up my whole test directory.
>>>
>>>
>>> But perhaps the testing I've done will suffice. Just let me know.
>>>
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
>>> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
>>> > Queue: met_help
>>> > Subject: HiRA ECNT and PSTD grid differences in point_stat
>>> > Owner: johnhg
>>> > Requestors: ric.crocker at metoffice.gov.uk
>>> > Status: new
>>> > Ticket <URL:
>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
>>> > >
>>> >
>>> >
>>> > This transaction appears to have no content
>>> >
>>>
>>>
>>>
>>>
------------------------------------------------
Subject: HiRA ECNT and PSTD grid differences in point_stat
From: John Halley Gotway
Time: Fri Nov 09 09:20:10 2018
Ric,
Great, thanks. I really appreciate having users pay close attention
to the
details.
I'll go ahead and resolve this support ticket now, but please let me
know
if your testing uncovers any additional issues.
Thanks,
John
On Fri, Nov 9, 2018 at 2:55 AM Crocker, Richard via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
>
> John,
> Thanks for the super-fast fix for this.
> I'll patch at our end and do a few tests,
> Thanks,
> Ric
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 08 November 2018 23:54
> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
differences
> in point_stat
>
> Ric,
>
> OK, I just posted the bugfix for this issue to the MET website:
>
https://dtcenter.org/met/users/support/known_issues/METv8.0/index.php
>
> Rerunning the regression test revealed differences in the ECNT line
type.
> But the differences only show up in the ECNT output from Point-Stat
for
> HiRA. So MET is only calling the problematic library utility in
that
> context.
>
> I expanded my testing to include widths of 1 and 2. And I made sure
that
> with 1, we're picking the *correct* grid point. And with 2, the
mean value
> is computed using the 4 closest grid points.
>
> Please let me know if you see inconsistent or questionable results
after
> the patch.
>
> Thanks,
> John
>
>
>
> On Thu, Nov 8, 2018 at 9:56 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Ric,
> >
> > I found it. At one point in the logic, we are inadvertently
casting
> > floating point grid x/y values to integers... which truncates
rather
> > than rounding. I'm surprised I didn't get a compiler warning
about this!
> >
> > Adding a new signature for that function to first round those
floating
> > points to the nearest integers solves the problem in my test case.
> >
> > So this is definitely a bug which we should patch for met-8.0.
Before
> > posting the bugfix, I want to re-run our regression test to
quantify
> > what impact it has on the output. But I'll write when the patch
is
> available.
> >
> > Thanks for digging through the details and finding this!
> >
> > John
> >
> > *(1) From the MPR lines, select the NEAREST and look at FCST -
OBS:*
> >
> > VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE FCST OBS
> >
> > UL NEAREST 1 MPR 387 1
> >
> > UR NEAREST 1 MPR 388 2
> >
> > LR NEAREST 1 MPR 438 3
> >
> > LL NEAREST 1 MPR 437 4
> >
> > *(2) From the CNT lines, select the ME column... and these values
> > match MPR:FCST - MPR:OBS:*
> >
> > VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
> >
> > UL NEAREST 1 CNT 386
> >
> > UR NEAREST 1 CNT 386
> >
> > LR NEAREST 1 CNT 435
> >
> > LL NEAREST 1 CNT 433
> >
> >
> > *(3) From the ECNT lines, select the ME column:*
> >
> > **** BEFORE THE BUGFIX, ITS WRONG (always using the FCST LL corner
> > value of 437) ****
> >
> > VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
> >
> > UL NBRHD_SQUARE 1 ECNT *436*
> >
> > UR NBRHD_SQUARE 1 ECNT *435*
> >
> > LR NBRHD_SQUARE 1 ECNT *434*
> >
> > LL NBRHD_SQUARE 1 ECNT *433*
> >
> >
> > ****AFTER THE BUGFIX, ITS CORRECT (exactly matches the CNT:ME
column)
> > ****
> >
> > VX_MASK INTERP_MTHD INTERP_PNTS LINE_TYPE ME
> >
> > UL NBRHD_SQUARE 1 ECNT 386
> >
> > UR NBRHD_SQUARE 1 ECNT 386
> >
> > LR NBRHD_SQUARE 1 ECNT 435
> >
> > LL NBRHD_SQUARE 1 ECNT 433
> >
> >
> > On Thu, Nov 8, 2018 at 9:14 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> >> Ric,
> >>
> >> Ah OK, I see the behavior you describe in my test case. I see
that
> >> the ME value from the ECNT line type appears to be computed using
the
> >> forecast value from the lower-left corner in all cases.
> >>
> >> I'll need to run this case through the debugger to determine why
this
> >> occurs and how to fix it. I'll let you know what I find.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Nov 8, 2018 at 5:30 AM Crocker, Richard via RT
> >> <met_help at ucar.edu>
> >> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676 >
> >>>
> >>> John,
> >>> Thanks for having a look at this,
> >>> It does look like HiRA and Nearest methods pick the same data,
which
> >>> is reassuring.
> >>>
> >>> I have tried replicating your method to check whether it was
just
> >>> user error on my part and I do get the same outcome for the MPR
> >>> lines
> >>>
> >>> The question I now have comes when I look at the Mean Error
output
> >>> values for the ECNT and CNT output with this test Given that
> >>> Nearest and NBOUR for a single point select the same grid
points,
> >>> Why do the values of ME in the CNT and ECNT files differ. (That
is
> >>> to say they differ in my test, but do you get the same thing?)
The
> >>> ME value in the CNT file gives what I would expect from the
NEAREST
> >>> data, but the ME in the ECNT output is different (and I guess I
> >>> expect it to be the same when looking at a single grid point
value).
> >>> I know there is an element for forecast mean in the ECNT output
for
> >>> larger neighbourhoods, but for a single grid point this should
be
> irrelevant.
> >>>
> >>> What I did was similar to you, a 50x50 grid with values
increasing
> >>> from
> >>> 0 to 2499 and a 1degree domain 0:50 latitude and 50:99
longitude.
> >>>
> >>> Working on a centre of 25,52 I used 4 obs offset as you did,
with
> >>> all obs values set to 0 (to remove the impact of the obs value
from
> >>> the mean error calculation) I get the following 4 values for
nearest
> >>> (and equivalent matches in the
> >>> nbour1 if I set thresholds to those forecast values) in the
matched
> >>> pairs file, TOTAL INDEX OBS_SID OBS_LAT OBS_LON OBS_LVL OBS_ELV
FCST
> >>> OBS
> >>> 4 1 00001 25.25 52.25 NA 0 1252 0
> >>> 4 2 00002 25.25 52.75 NA 0 1253 0
> >>> 4 3 00003 25.75 52.25 NA 0 1302 0
> >>> 4 4 00004 25.75 52.75 NA 0 1303 0
> >>>
> >>> In the ECNT file I get
> >>> TOTAL N_ENS ME
> >>> 4 1 1252
> >>>
> >>> But in the CNT files I get
> >>> ME
> >>> 1277.5 which is the average of the 4 nearest forecast values
above
> >>> as expected
> >>>
> >>> If I rerun 4 times, just using each of the four observations
on
> >>> their own (in the order above) I get the following mean error
values
> >>> from the ECNT and CNT files:
> >>> ECNT CNT
> >>> 1252 1252
> >>> 1252 1253
> >>> 1252 1302
> >>> 1252 1303
> >>> So the ECNT forecast mean appears not to be changing to use a
> >>> different forecast value/point despite the CNT value matching
the
> >>> NEAREST and NBOUR_1 forecast values.
> >>>
> >>>
> >>> Are you able to replicate this?
> >>> (and is it important, and am I fundamentally misunderstanding
what
> >>> the mean error is in the ECNT file for point_stat/HiRA)
> >>>
> >>> Many thanks,
> >>> Ric
> >>>
> >>> -----Original Message-----
> >>> From: John Halley Gotway via RT <met_help at ucar.edu>
> >>> Sent: 07 November 2018 17:51
> >>> To: Crocker, Richard <ric.crocker at metoffice.gov.uk>
> >>> Subject: Re: [rt.rap.ucar.edu #87676] HiRA ECNT and PSTD grid
> >>> differences in point_stat
> >>>
> >>> Ric,
> >>>
> >>> OK, I did some detailed testing and have convinced myself that I
do
> >>> not see the problem you describe.
> >>>
> >>> Here's how I tested...
> >>>
> >>> (1) Write a Python script to create some fake data with integer
> >>> values ranging from 0 to 1499 on a 30x50 domain. This is to
just
> >>> ensure that each grid point has a different value so I can tell
them
> apart.
> >>> (2) Define this fake data on a 1 degree lat/lon grid over the
CONUS.
> >>> (3) Pick a single lat/lon grid box to test... I chose 41 lat and
-92
> lon.
> >>> (4) Define 4 lat/lon points inside that grid box which are
closet to
> >>> each of the 4 corners:
> >>>
> >>> UL 41.75 -92.75
> >>>
> >>> UR 41.75 -92.25
> >>>
> >>> LR 41.25 -92.25
> >>>
> >>> LL 41.25 -92.75
> >>>
> >>> Assign them station ID's to indicate the corner closest to that
point:
> >>> UL, UR, LR, and LL
> >>>
> >>> (5) Configure/run Point-Stat to verify against these 4 points,
> >>> treating each as a separate region.
> >>>
> >>> Choose 5 interpolation methods: NEAREST, UPPER_LEFT,
UPPER_RIGHT,
> >>> LOWER_RIGHT, and LOWER_LEFT.
> >>>
> >>> Note that the fake data values at these corner points happen
to be
> >>> 387, 388, 437, and 438.
> >>>
> >>> Define categorical thresholds to be exactly equal to these
values,
> >>> and turn the HiRA method on using a width of 1.
> >>>
> >>> (6) Look through the MPR output lines to make sure that...
> >>>
> >>> - Point-Stat chooses the correct corner point for the NEAREST
> >>> interpolation method.
> >>>
> >>> - HiRA chooses the same corner point for width = 1.
> >>>
> >>>
> >>> As far as I can tell, there are no inconsistencies in the MPR
output.
> >>> The choice of model point is consistent between the NEAREST
> >>> interpolation method and HiRA with a width of 1. And beyond
being
> >>> consistent, it's correct.
> >>>
> >>>
> >>> I've attached the MPR output file I generated for you to
inspect.
> >>> If it'd be helpful, I could tar up my whole test directory.
> >>>
> >>>
> >>> But perhaps the testing I've done will suffice. Just let me
know.
> >>>
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>>
> >>> On Wed, Nov 7, 2018 at 8:15 AM The RT System itself via RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>> >
> >>> > Wed Nov 07 08:15:38 2018: Request 87676 was acted upon.
> >>> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >>> > Queue: met_help
> >>> > Subject: HiRA ECNT and PSTD grid differences in
point_stat
> >>> > Owner: johnhg
> >>> > Requestors: ric.crocker at metoffice.gov.uk
> >>> > Status: new
> >>> > Ticket <URL:
> >>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87676
> >>> > >
> >>> >
> >>> >
> >>> > This transaction appears to have no content
> >>> >
> >>>
> >>>
> >>>
> >>>
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list