[Met_help] [rt.rap.ucar.edu #80988] History for Grid Stat Missing Value
John Halley Gotway via RT
met_help at ucar.edu
Tue Jun 27 13:26:05 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I have been running into an error when I use NCEP 2.5 deg climo for the climo reference. Below is the command line:
/h/WXQC/met-6.0/bin/grid_stat /h/data/global/WXQC/data/geps_verify/qcsave/2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
When I run this I get the following error:
DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
DEBUG 3: Reading observation data for PWS15(*,*).
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcMet".
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll = 90
DEBUG 4: lon_ll = -0
DEBUG 4: delta_lat = 2.5
DEBUG 4: delta_lon = 2.5
DEBUG 4: Nlat = 73
DEBUG 4: Nlon = 144
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcMet".
DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
DEBUG 3: For forecast valid at 20170619_000000, found 1 climatology field(s) valid at 20170619_000000.
DEBUG 3: Found 1 climatology fields.
DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
DEBUG 2: Processing masking regions.
DEBUG 3: Processing grid mask: FULL
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate interpolation method.
DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate interpolation method.
DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation method NEAREST(1), over region FULL, using 65160 pairs.
DEBUG 2: Computing Probabilistic Statistics.
ERROR :
ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ... -9999
ERROR :
I have sent you all the files involved. When I run the code using a netCDF file which is the same resolution as the FCST which contains the average frequency of the event I don't get this error. If I switch to using the ncep 2.5 deg climo and remap it to the FCST, I get the error. Looking in all the files I don't see any values of -9999. So I don't know where this is coming from.
Thanks
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Thu Jun 22 14:03:48 2017
Bob,
I grabbed the files you posted to our ftp site, but I noticed that the
timestamp of this file is older:
GridStatConfig_geps_updated
Can you please make sure that file on the ftp site exactly matches the
one
you're using:
/h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
I was able to get it to run using met-6.0 but unable to replicate your
error.
So I want to make sure I'm testing with the same configuration.
Thanks,
John
On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Grid Stat Missing Value
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
>
> John, I have been running into an error when I use NCEP 2.5 deg
climo for
> the climo reference. Below is the command line:
>
> /h/WXQC/met-6.0/bin/grid_stat
/h/data/global/WXQC/data/geps_verify/qcsave/
> 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb /h/WXQC/met-
6.0/data/config/GridStatConfig_geps_updated
> -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>
> When I run this I get the following error:
>
> DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> DEBUG 3: Reading observation data for PWS15(*,*).
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = 90
> DEBUG 4: lon_ll = -0
> DEBUG 4: delta_lat = 2.5
> DEBUG 4: delta_lon = 2.5
> DEBUG 4: Nlat = 73
> DEBUG 4: Nlon = 144
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcMet".
> DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
> DEBUG 3: For forecast valid at 20170619_000000, found 1 climatology
> field(s) valid at 20170619_000000.
> DEBUG 3: Found 1 climatology fields.
> DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2:
> DEBUG 2:
------------------------------------------------------------
> --------------------
> DEBUG 2:
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
method
> NEAREST(1), over region FULL, using 65160 pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> ERROR :
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
-9999
> ERROR :
>
> I have sent you all the files involved. When I run the code using
a
> netCDF file which is the same resolution as the FCST which contains
the
> average frequency of the event I don't get this error. If I switch
to
> using the ncep 2.5 deg climo and remap it to the FCST, I get the
error.
> Looking in all the files I don't see any values of -9999. So I
don't know
> where this is coming from.
>
> Thanks
> Bob Craig
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Thu Jun 22 14:13:42 2017
Bob,
I think you may have a couple of problems. I plotted your
observations and
climo fields...
met-6.0/bin/plot_data_plane \
a1262a1d-4201-48b4-bc5b-5cbf84af32bb \
obs.ps 'name="PWS15"; level="Z10";'
met-6.0/bin/plot_data_plane \
c5d93206-eb26-4512-bf50-be39ed9635ed_climo climo.ps \
'name="PWS15"; level="Z10";'
The results are attached.
Looking at obs.png, I suspect that there's a problem in how you've
encoded
the data. Doesn't look like much structure there.
And looking at climo.png, notice that there's no map data... meaning
that
MET doesn't know where on earth this data lives. I'm guessing its the
same
problem with lat_ll = 90.0 north which should be -90.
Thanks,
John
On Thu, Jun 22, 2017 at 2:03 PM, The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 22 14:03:48 2017: Request 80988 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Grid Stat Missing Value
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 06:44:44 2017
Thanks for the image of the obs file, I should use data_plane more to
check those things. I degrib Grib 2 u and v wind components,
calculate the wind speed, then convert that array to a netCDF file. I
looks like the x and y columns might be reversed. I will dig into
that and see if I can fix the problem.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 22, 2017 3:14 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I think you may have a couple of problems. I plotted your
observations and climo fields...
met-6.0/bin/plot_data_plane \
a1262a1d-4201-48b4-bc5b-5cbf84af32bb \
obs.ps 'name="PWS15"; level="Z10";'
met-6.0/bin/plot_data_plane \
c5d93206-eb26-4512-bf50-be39ed9635ed_climo climo.ps \ 'name="PWS15";
level="Z10";'
The results are attached.
Looking at obs.png, I suspect that there's a problem in how you've
encoded the data. Doesn't look like much structure there.
And looking at climo.png, notice that there's no map data... meaning
that MET doesn't know where on earth this data lives. I'm guessing
its the same problem with lat_ll = 90.0 north which should be -90.
Thanks,
John
On Thu, Jun 22, 2017 at 2:03 PM, The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 22 14:03:48 2017: Request 80988 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Grid Stat Missing Value
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 09:14:09 2017
John, I fixed the problem you found with my observation file. After I
converted it to a netCDF file, I displayed it (on the ftp server).
Note the map does not fit right. I placed the netCDF file on the
server. The header for the file seems correct:
netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
dimensions:
lat = 1152 ;
lon = 1536 ;
variables:
float PWS15(lat, lon) ;
PWS15:name = "PWS15" ;
PWS15:init_time_ut = "1497916800" ;
PWS15:valid_time = "20170620_000000" ;
PWS15:valid_time_ut = "1497916800" ;
PWS15:init_time = "20170620_000000" ;
PWS15:units = "%" ;
PWS15:level = "Z10" ;
// global attributes:
:FileOrigins = "Model Data" ;
:MET_version = "V6.0" ;
:Projection = "LatLon" ;
:lat_ll = "-89.921875 degrees_north" ;
:lon_ll = "0.117187 degrees_east" ;
:delta_lat = "0.234375 degrees" ;
:delta_lon = "0.15625 degrees" ;
:Nlat = "1152 grid_points" ;
:Nlon = "1536 grid_points" ;
Any idea why the map is wrong - should I worry?
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 22, 2017 3:04 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I grabbed the files you posted to our ftp site, but I noticed that the
timestamp of this file is older:
GridStatConfig_geps_updated
Can you please make sure that file on the ftp site exactly matches the
one you're using:
/h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
I was able to get it to run using met-6.0 but unable to replicate your
error.
So I want to make sure I'm testing with the same configuration.
Thanks,
John
On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Grid Stat Missing Value
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> >
>
>
> John, I have been running into an error when I use NCEP 2.5 deg
climo
> for the climo reference. Below is the command line:
>
> /h/WXQC/met-6.0/bin/grid_stat
> /h/data/global/WXQC/data/geps_verify/qcsave/
> 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>
> When I run this I get the following error:
>
> DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> DEBUG 3: Reading observation data for PWS15(*,*).
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = 90
> DEBUG 4: lon_ll = -0
> DEBUG 4: delta_lat = 2.5
> DEBUG 4: delta_lon = 2.5
> DEBUG 4: Nlat = 73
> DEBUG 4: Nlon = 144
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcMet".
> DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
> DEBUG 3: For forecast valid at 20170619_000000, found 1 climatology
> field(s) valid at 20170619_000000.
> DEBUG 3: Found 1 climatology fields.
> DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2:
> DEBUG 2:
------------------------------------------------------------
> --------------------
> DEBUG 2:
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> method NEAREST(1), over region FULL, using 65160 pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> ERROR :
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
> -9999 ERROR :
>
> I have sent you all the files involved. When I run the code using
a
> netCDF file which is the same resolution as the FCST which contains
> the average frequency of the event I don't get this error. If I
> switch to using the ncep 2.5 deg climo and remap it to the FCST, I
get the error.
> Looking in all the files I don't see any values of -9999. So I
don't
> know where this is coming from.
>
> Thanks
> Bob Craig
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 10:10:03 2017
Bob,
Thanks for sending the sample observation data file. I ran it through
plot_data_plane and see what you mean. The earth doesn't fit the data
well. I suspect there's something wrong in your grid definition.
Let's think about this. The lower left corner is defined as (lat,
lon) =
(-89.92, 0.117). Let's calculate the upper-right corner:
UR Lat = -89.92 + (0.234375 * 1152) = 180.51
UR Lon = 0.117 + (.15625 * 1536) = 240.117
You have an upper-right (lat, lon) = (180.51, 240.117), but you want
it to
be about (90, 360).
Swapping the delta_lat and delta_lon values results in an upper-right
corner of (90.078, 360.117). I suspect that's the problem.
I swapped those values using ncatted and replotted using
plot_data_plane.
The resulting plot (attached) looks a lot better:
ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
-a delta_lon,global,o,c,"0.234375 degrees" \
-o obs_new.nc \
1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
met-6.0/bin/plot_data_plane \
obs_new.nc obs_new.ps \
'name="PWS15"; level="(*,*)";'
Thanks,
John
On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I fixed the problem you found with my observation file. After
I
> converted it to a netCDF file, I displayed it (on the ftp server).
Note
> the map does not fit right. I placed the netCDF file on the server.
The
> header for the file seems correct:
>
> netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> dimensions:
> lat = 1152 ;
> lon = 1536 ;
> variables:
> float PWS15(lat, lon) ;
> PWS15:name = "PWS15" ;
> PWS15:init_time_ut = "1497916800" ;
> PWS15:valid_time = "20170620_000000" ;
> PWS15:valid_time_ut = "1497916800" ;
> PWS15:init_time = "20170620_000000" ;
> PWS15:units = "%" ;
> PWS15:level = "Z10" ;
>
> // global attributes:
> :FileOrigins = "Model Data" ;
> :MET_version = "V6.0" ;
> :Projection = "LatLon" ;
> :lat_ll = "-89.921875 degrees_north" ;
> :lon_ll = "0.117187 degrees_east" ;
> :delta_lat = "0.234375 degrees" ;
> :delta_lon = "0.15625 degrees" ;
> :Nlat = "1152 grid_points" ;
> :Nlon = "1536 grid_points" ;
>
> Any idea why the map is wrong - should I worry?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 22, 2017 3:04 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing
> Value
>
> Bob,
>
> I grabbed the files you posted to our ftp site, but I noticed that
the
> timestamp of this file is older:
> GridStatConfig_geps_updated
>
> Can you please make sure that file on the ftp site exactly matches
the one
> you're using:
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>
> I was able to get it to run using met-6.0 but unable to replicate
your
> error.
>
> So I want to make sure I'm testing with the same configuration.
>
> Thanks,
> John
>
>
> On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Missing Value
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > >
> >
> >
> > John, I have been running into an error when I use NCEP 2.5 deg
climo
> > for the climo reference. Below is the command line:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > When I run this I get the following error:
> >
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = 90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170619_000000, found 1
climatology
> > field(s) valid at 20170619_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 65160 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > I have sent you all the files involved. When I run the code
using a
> > netCDF file which is the same resolution as the FCST which
contains
> > the average frequency of the event I don't get this error. If I
> > switch to using the ncep 2.5 deg climo and remap it to the FCST, I
get
> the error.
> > Looking in all the files I don't see any values of -9999. So I
don't
> > know where this is coming from.
> >
> > Thanks
> > Bob Craig
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 12:37:57 2017
John, With your help (thanks for that) I fixed the observation file
and climo file issues and they now look right (the files are on the
server). I also placed the most recent config file there. When I run
the following command:
/h/WXQC/met-6.0/bin/grid_stat
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900.0024
/h/data/global/WXQC/data/met/tmp/68bf9465-1763-4eb3-a42f-413092931b60
/h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated -outdir
/h/data/global/WXQC/data/met/gdstat/geps -v 4
I get the same error:
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet".
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll = -89.9219
DEBUG 4: lon_ll = -0.117187
DEBUG 4: delta_lat = 0.15625
DEBUG 4: delta_lon = 0.234375
DEBUG 4: Nlat = 1152
DEBUG 4: Nlon = 1536
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_Gb2".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcMet".
DEBUG 3: Use the forecast grid.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181 lat_ll:
-90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072977206617
DEBUG 1: Forecast File:
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900.0024
DEBUG 1: Observation File: /h/data/global/WXQC/data/met/tmp/68bf9465-
1763-4eb3-a42f-413092931b60
DEBUG 3: Reading forecast data for PWS15/Z10.
DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
'/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900.0024'
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 100
DEBUG 4: scan mode: 0
DEBUG 4: valid time: 20170620_000000
DEBUG 4: lead time: 240000
DEBUG 4: init time: 20170619_000000
DEBUG 4: bitmap flag: 255
DEBUG 4:
DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
DEBUG 3: Reading observation data for PWS15(*,*).
DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet".
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll = -90
DEBUG 4: lon_ll = -0
DEBUG 4: delta_lat = 2.5
DEBUG 4: delta_lon = 2.5
DEBUG 4: Nlat = 73
DEBUG 4: Nlon = 144
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcMet".
DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
DEBUG 3: For forecast valid at 20170620_000000, found 1 climatology
field(s) valid at 20170620_000000.
DEBUG 3: Found 1 climatology fields.
DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
DEBUG 2: Processing masking regions.
DEBUG 3: Processing grid mask: FULL
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
interpolation method.
DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
interpolation method.
DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
method NEAREST(1), over region FULL, using 64620 pairs.
DEBUG 2: Computing Probabilistic Statistics.
ERROR :
ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
-9999
ERROR :
Let me know if you get the same error?
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 11:10 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
Thanks for sending the sample observation data file. I ran it through
plot_data_plane and see what you mean. The earth doesn't fit the data
well. I suspect there's something wrong in your grid definition.
Let's think about this. The lower left corner is defined as (lat,
lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 + (.15625
* 1536) = 240.117
You have an upper-right (lat, lon) = (180.51, 240.117), but you want
it to be about (90, 360).
Swapping the delta_lat and delta_lon values results in an upper-right
corner of (90.078, 360.117). I suspect that's the problem.
I swapped those values using ncatted and replotted using
plot_data_plane.
The resulting plot (attached) looks a lot better:
ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
-a delta_lon,global,o,c,"0.234375 degrees" \
-o obs_new.nc \
1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
met-6.0/bin/plot_data_plane \
obs_new.nc obs_new.ps \
'name="PWS15"; level="(*,*)";'
Thanks,
John
On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I fixed the problem you found with my observation file. After
I
> converted it to a netCDF file, I displayed it (on the ftp server).
> Note the map does not fit right. I placed the netCDF file on the
> server. The header for the file seems correct:
>
> netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> dimensions:
> lat = 1152 ;
> lon = 1536 ;
> variables:
> float PWS15(lat, lon) ;
> PWS15:name = "PWS15" ;
> PWS15:init_time_ut = "1497916800" ;
> PWS15:valid_time = "20170620_000000" ;
> PWS15:valid_time_ut = "1497916800" ;
> PWS15:init_time = "20170620_000000" ;
> PWS15:units = "%" ;
> PWS15:level = "Z10" ;
>
> // global attributes:
> :FileOrigins = "Model Data" ;
> :MET_version = "V6.0" ;
> :Projection = "LatLon" ;
> :lat_ll = "-89.921875 degrees_north" ;
> :lon_ll = "0.117187 degrees_east" ;
> :delta_lat = "0.234375 degrees" ;
> :delta_lon = "0.15625 degrees" ;
> :Nlat = "1152 grid_points" ;
> :Nlon = "1536 grid_points" ;
>
> Any idea why the map is wrong - should I worry?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 22, 2017 3:04 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I grabbed the files you posted to our ftp site, but I noticed that
the
> timestamp of this file is older:
> GridStatConfig_geps_updated
>
> Can you please make sure that file on the ftp site exactly matches
the
> one you're using:
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>
> I was able to get it to run using met-6.0 but unable to replicate
your
> error.
>
> So I want to make sure I'm testing with the same configuration.
>
> Thanks,
> John
>
>
> On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Missing Value
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > >
> >
> >
> > John, I have been running into an error when I use NCEP 2.5 deg
> > climo for the climo reference. Below is the command line:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > When I run this I get the following error:
> >
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = 90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170619_000000, found 1
climatology
> > field(s) valid at 20170619_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
> > ------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 65160 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > I have sent you all the files involved. When I run the code
using a
> > netCDF file which is the same resolution as the FCST which
contains
> > the average frequency of the event I don't get this error. If I
> > switch to using the ncep 2.5 deg climo and remap it to the FCST, I
> > get
> the error.
> > Looking in all the files I don't see any values of -9999. So I
> > don't know where this is coming from.
> >
> > Thanks
> > Bob Craig
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 12:51:48 2017
Bob,
I don't see your input files on the ftp site. Looking in:
incoming/irap/met_help
The following files do not exist:
grib2.2017061900.0024
GridStatConfig_geps_updated
Could you please post them to make sure I'm running exactly the same
case
as you?
Thanks,
John
On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, With your help (thanks for that) I fixed the observation file
and
> climo file issues and they now look right (the files are on the
server). I
> also placed the most recent config file there. When I run the
following
> command:
>
> /h/WXQC/met-6.0/bin/grid_stat
/h/data/global/WXQC/data/geps_verify/qcsave/
> 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60 /h/WXQC/met-
6.0/data/config/GridStatConfig_geps_updated
> -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>
> I get the same error:
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -89.9219
> DEBUG 4: lon_ll = -0.117187
> DEBUG 4: delta_lat = 0.15625
> DEBUG 4: delta_lon = 0.234375
> DEBUG 4: Nlat = 1152
> DEBUG 4: Nlon = 1536
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Gb2".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcMet".
> DEBUG 3: Use the forecast grid.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072977206617
> DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> verify/qcsave/2017061900/grib2.2017061900.0024
> DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60
> DEBUG 3: Reading forecast data for PWS15/Z10.
> DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> grib2.2017061900.0024'
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 100
> DEBUG 4: scan mode: 0
> DEBUG 4: valid time: 20170620_000000
> DEBUG 4: lead time: 240000
> DEBUG 4: init time: 20170619_000000
> DEBUG 4: bitmap flag: 255
> DEBUG 4:
> DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> DEBUG 3: Reading observation data for PWS15(*,*).
> DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -90
> DEBUG 4: lon_ll = -0
> DEBUG 4: delta_lat = 2.5
> DEBUG 4: delta_lon = 2.5
> DEBUG 4: Nlat = 73
> DEBUG 4: Nlon = 144
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcMet".
> DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
> DEBUG 3: For forecast valid at 20170620_000000, found 1 climatology
> field(s) valid at 20170620_000000.
> DEBUG 3: Found 1 climatology fields.
> DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2:
> DEBUG 2:
------------------------------------------------------------
> --------------------
> DEBUG 2:
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
method
> NEAREST(1), over region FULL, using 64620 pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> ERROR :
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
-9999
> ERROR :
>
> Let me know if you get the same error?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 11:10 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> Thanks for sending the sample observation data file. I ran it
through
> plot_data_plane and see what you mean. The earth doesn't fit the
data
> well. I suspect there's something wrong in your grid definition.
>
> Let's think about this. The lower left corner is defined as (lat,
lon) =
> (-89.92, 0.117). Let's calculate the upper-right corner:
>
> UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
(.15625 *
> 1536) = 240.117
>
> You have an upper-right (lat, lon) = (180.51, 240.117), but you want
it to
> be about (90, 360).
>
> Swapping the delta_lat and delta_lon values results in an upper-
right
> corner of (90.078, 360.117). I suspect that's the problem.
>
> I swapped those values using ncatted and replotted using
plot_data_plane.
> The resulting plot (attached) looks a lot better:
>
> ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> -a delta_lon,global,o,c,"0.234375 degrees" \
> -o obs_new.nc \
> 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
>
> met-6.0/bin/plot_data_plane \
> obs_new.nc obs_new.ps \
> 'name="PWS15"; level="(*,*)";'
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, I fixed the problem you found with my observation file.
After I
> > converted it to a netCDF file, I displayed it (on the ftp server).
> > Note the map does not fit right. I placed the netCDF file on the
> > server. The header for the file seems correct:
> >
> > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > dimensions:
> > lat = 1152 ;
> > lon = 1536 ;
> > variables:
> > float PWS15(lat, lon) ;
> > PWS15:name = "PWS15" ;
> > PWS15:init_time_ut = "1497916800" ;
> > PWS15:valid_time = "20170620_000000" ;
> > PWS15:valid_time_ut = "1497916800" ;
> > PWS15:init_time = "20170620_000000" ;
> > PWS15:units = "%" ;
> > PWS15:level = "Z10" ;
> >
> > // global attributes:
> > :FileOrigins = "Model Data" ;
> > :MET_version = "V6.0" ;
> > :Projection = "LatLon" ;
> > :lat_ll = "-89.921875 degrees_north" ;
> > :lon_ll = "0.117187 degrees_east" ;
> > :delta_lat = "0.234375 degrees" ;
> > :delta_lon = "0.15625 degrees" ;
> > :Nlat = "1152 grid_points" ;
> > :Nlon = "1536 grid_points" ;
> >
> > Any idea why the map is wrong - should I worry?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 22, 2017 3:04 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > Missing Value
> >
> > Bob,
> >
> > I grabbed the files you posted to our ftp site, but I noticed that
the
> > timestamp of this file is older:
> > GridStatConfig_geps_updated
> >
> > Can you please make sure that file on the ftp site exactly matches
the
> > one you're using:
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> >
> > I was able to get it to run using met-6.0 but unable to replicate
your
> > error.
> >
> > So I want to make sure I'm testing with the same configuration.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Missing Value
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > >
> > >
> > >
> > > John, I have been running into an error when I use NCEP 2.5 deg
> > > climo for the climo reference. Below is the command line:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > When I run this I get the following error:
> > >
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = 90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170619_000000, found 1
climatology
> > > field(s) valid at 20170619_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL
> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 65160 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > I have sent you all the files involved. When I run the code
using a
> > > netCDF file which is the same resolution as the FCST which
contains
> > > the average frequency of the event I don't get this error. If I
> > > switch to using the ncep 2.5 deg climo and remap it to the FCST,
I
> > > get
> > the error.
> > > Looking in all the files I don't see any values of -9999. So I
> > > don't know where this is coming from.
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value - update
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 12:53:09 2017
John, I did notice that my lower left longitude of my ob file is
negative in the MET output but it is positive in the netCDF file where
it was read from???
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 11:10 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
Thanks for sending the sample observation data file. I ran it through
plot_data_plane and see what you mean. The earth doesn't fit the data
well. I suspect there's something wrong in your grid definition.
Let's think about this. The lower left corner is defined as (lat,
lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 + (.15625
* 1536) = 240.117
You have an upper-right (lat, lon) = (180.51, 240.117), but you want
it to be about (90, 360).
Swapping the delta_lat and delta_lon values results in an upper-right
corner of (90.078, 360.117). I suspect that's the problem.
I swapped those values using ncatted and replotted using
plot_data_plane.
The resulting plot (attached) looks a lot better:
ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
-a delta_lon,global,o,c,"0.234375 degrees" \
-o obs_new.nc \
1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
met-6.0/bin/plot_data_plane \
obs_new.nc obs_new.ps \
'name="PWS15"; level="(*,*)";'
Thanks,
John
On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I fixed the problem you found with my observation file. After
I
> converted it to a netCDF file, I displayed it (on the ftp server).
> Note the map does not fit right. I placed the netCDF file on the
> server. The header for the file seems correct:
>
> netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> dimensions:
> lat = 1152 ;
> lon = 1536 ;
> variables:
> float PWS15(lat, lon) ;
> PWS15:name = "PWS15" ;
> PWS15:init_time_ut = "1497916800" ;
> PWS15:valid_time = "20170620_000000" ;
> PWS15:valid_time_ut = "1497916800" ;
> PWS15:init_time = "20170620_000000" ;
> PWS15:units = "%" ;
> PWS15:level = "Z10" ;
>
> // global attributes:
> :FileOrigins = "Model Data" ;
> :MET_version = "V6.0" ;
> :Projection = "LatLon" ;
> :lat_ll = "-89.921875 degrees_north" ;
> :lon_ll = "0.117187 degrees_east" ;
> :delta_lat = "0.234375 degrees" ;
> :delta_lon = "0.15625 degrees" ;
> :Nlat = "1152 grid_points" ;
> :Nlon = "1536 grid_points" ;
>
> Any idea why the map is wrong - should I worry?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 22, 2017 3:04 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I grabbed the files you posted to our ftp site, but I noticed that
the
> timestamp of this file is older:
> GridStatConfig_geps_updated
>
> Can you please make sure that file on the ftp site exactly matches
the
> one you're using:
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>
> I was able to get it to run using met-6.0 but unable to replicate
your
> error.
>
> So I want to make sure I'm testing with the same configuration.
>
> Thanks,
> John
>
>
> On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Missing Value
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > >
> >
> >
> > John, I have been running into an error when I use NCEP 2.5 deg
> > climo for the climo reference. Below is the command line:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > When I run this I get the following error:
> >
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = 90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170619_000000, found 1
climatology
> > field(s) valid at 20170619_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
> > ------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 65160 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > I have sent you all the files involved. When I run the code
using a
> > netCDF file which is the same resolution as the FCST which
contains
> > the average frequency of the event I don't get this error. If I
> > switch to using the ncep 2.5 deg climo and remap it to the FCST, I
> > get
> the error.
> > Looking in all the files I don't see any values of -9999. So I
> > don't know where this is coming from.
> >
> > Thanks
> > Bob Craig
> >
> >
> >
>
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 12:56:56 2017
Bob,
People use different conventions for specifying longitudes. Some use
a
range of [0, 360) while other use (-180, 180].
MET uses the latter. When it reads longitudes in the input, it
rescales
them to the range of (-180, 180] by adding/subtracting 360 as many
times as
necessary.
Nothing to worry about.
John
On Fri, Jun 23, 2017 at 12:53 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I did notice that my lower left longitude of my ob file is
negative
> in the MET output but it is positive in the netCDF file where it was
read
> from???
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 11:10 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> Thanks for sending the sample observation data file. I ran it
through
> plot_data_plane and see what you mean. The earth doesn't fit the
data
> well. I suspect there's something wrong in your grid definition.
>
> Let's think about this. The lower left corner is defined as (lat,
lon) =
> (-89.92, 0.117). Let's calculate the upper-right corner:
>
> UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
(.15625 *
> 1536) = 240.117
>
> You have an upper-right (lat, lon) = (180.51, 240.117), but you want
it to
> be about (90, 360).
>
> Swapping the delta_lat and delta_lon values results in an upper-
right
> corner of (90.078, 360.117). I suspect that's the problem.
>
> I swapped those values using ncatted and replotted using
plot_data_plane.
> The resulting plot (attached) looks a lot better:
>
> ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> -a delta_lon,global,o,c,"0.234375 degrees" \
> -o obs_new.nc \
> 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
>
> met-6.0/bin/plot_data_plane \
> obs_new.nc obs_new.ps \
> 'name="PWS15"; level="(*,*)";'
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, I fixed the problem you found with my observation file.
After I
> > converted it to a netCDF file, I displayed it (on the ftp server).
> > Note the map does not fit right. I placed the netCDF file on the
> > server. The header for the file seems correct:
> >
> > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > dimensions:
> > lat = 1152 ;
> > lon = 1536 ;
> > variables:
> > float PWS15(lat, lon) ;
> > PWS15:name = "PWS15" ;
> > PWS15:init_time_ut = "1497916800" ;
> > PWS15:valid_time = "20170620_000000" ;
> > PWS15:valid_time_ut = "1497916800" ;
> > PWS15:init_time = "20170620_000000" ;
> > PWS15:units = "%" ;
> > PWS15:level = "Z10" ;
> >
> > // global attributes:
> > :FileOrigins = "Model Data" ;
> > :MET_version = "V6.0" ;
> > :Projection = "LatLon" ;
> > :lat_ll = "-89.921875 degrees_north" ;
> > :lon_ll = "0.117187 degrees_east" ;
> > :delta_lat = "0.234375 degrees" ;
> > :delta_lon = "0.15625 degrees" ;
> > :Nlat = "1152 grid_points" ;
> > :Nlon = "1536 grid_points" ;
> >
> > Any idea why the map is wrong - should I worry?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 22, 2017 3:04 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > Missing Value
> >
> > Bob,
> >
> > I grabbed the files you posted to our ftp site, but I noticed that
the
> > timestamp of this file is older:
> > GridStatConfig_geps_updated
> >
> > Can you please make sure that file on the ftp site exactly matches
the
> > one you're using:
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> >
> > I was able to get it to run using met-6.0 but unable to replicate
your
> > error.
> >
> > So I want to make sure I'm testing with the same configuration.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Missing Value
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > >
> > >
> > >
> > > John, I have been running into an error when I use NCEP 2.5 deg
> > > climo for the climo reference. Below is the command line:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > When I run this I get the following error:
> > >
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = 90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170619_000000, found 1
climatology
> > > field(s) valid at 20170619_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL
> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 65160 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > I have sent you all the files involved. When I run the code
using a
> > > netCDF file which is the same resolution as the FCST which
contains
> > > the average frequency of the event I don't get this error. If I
> > > switch to using the ncep 2.5 deg climo and remap it to the FCST,
I
> > > get
> > the error.
> > > Looking in all the files I don't see any values of -9999. So I
> > > don't know where this is coming from.
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 13:18:48 2017
I pushed them to incoming/irap/met_help/craigr_data again.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 1:52 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I don't see your input files on the ftp site. Looking in:
incoming/irap/met_help
The following files do not exist:
grib2.2017061900.0024
GridStatConfig_geps_updated
Could you please post them to make sure I'm running exactly the same
case as you?
Thanks,
John
On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, With your help (thanks for that) I fixed the observation file
> and climo file issues and they now look right (the files are on the
> server). I also placed the most recent config file there. When I
run
> the following
> command:
>
> /h/WXQC/met-6.0/bin/grid_stat
> /h/data/global/WXQC/data/geps_verify/qcsave/
> 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>
> I get the same error:
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -89.9219
> DEBUG 4: lon_ll = -0.117187
> DEBUG 4: delta_lat = 0.15625
> DEBUG 4: delta_lon = 0.234375
> DEBUG 4: Nlat = 1152
> DEBUG 4: Nlon = 1536
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_Gb2".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcMet".
> DEBUG 3: Use the forecast grid.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072977206617
> DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> verify/qcsave/2017061900/grib2.2017061900.0024
> DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60
> DEBUG 3: Reading forecast data for PWS15/Z10.
> DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> grib2.2017061900.0024'
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 100
> DEBUG 4: scan mode: 0
> DEBUG 4: valid time: 20170620_000000
> DEBUG 4: lead time: 240000
> DEBUG 4: init time: 20170619_000000
> DEBUG 4: bitmap flag: 255
> DEBUG 4:
> DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> DEBUG 3: Reading observation data for PWS15(*,*).
> DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -90
> DEBUG 4: lon_ll = -0
> DEBUG 4: delta_lat = 2.5
> DEBUG 4: delta_lon = 2.5
> DEBUG 4: Nlat = 73
> DEBUG 4: Nlon = 144
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcMet".
> DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
> DEBUG 3: For forecast valid at 20170620_000000, found 1 climatology
> field(s) valid at 20170620_000000.
> DEBUG 3: Found 1 climatology fields.
> DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2:
> DEBUG 2:
------------------------------------------------------------
> --------------------
> DEBUG 2:
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> method NEAREST(1), over region FULL, using 64620 pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> ERROR :
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
> -9999 ERROR :
>
> Let me know if you get the same error?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 11:10 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> Thanks for sending the sample observation data file. I ran it
through
> plot_data_plane and see what you mean. The earth doesn't fit the
data
> well. I suspect there's something wrong in your grid definition.
>
> Let's think about this. The lower left corner is defined as (lat,
> lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
>
> UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
(.15625
> *
> 1536) = 240.117
>
> You have an upper-right (lat, lon) = (180.51, 240.117), but you want
> it to be about (90, 360).
>
> Swapping the delta_lat and delta_lon values results in an upper-
right
> corner of (90.078, 360.117). I suspect that's the problem.
>
> I swapped those values using ncatted and replotted using
plot_data_plane.
> The resulting plot (attached) looks a lot better:
>
> ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> -a delta_lon,global,o,c,"0.234375 degrees" \
> -o obs_new.nc \
> 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
>
> met-6.0/bin/plot_data_plane \
> obs_new.nc obs_new.ps \
> 'name="PWS15"; level="(*,*)";'
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, I fixed the problem you found with my observation file.
After
> > I converted it to a netCDF file, I displayed it (on the ftp
server).
> > Note the map does not fit right. I placed the netCDF file on the
> > server. The header for the file seems correct:
> >
> > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > dimensions:
> > lat = 1152 ;
> > lon = 1536 ;
> > variables:
> > float PWS15(lat, lon) ;
> > PWS15:name = "PWS15" ;
> > PWS15:init_time_ut = "1497916800" ;
> > PWS15:valid_time = "20170620_000000" ;
> > PWS15:valid_time_ut = "1497916800" ;
> > PWS15:init_time = "20170620_000000" ;
> > PWS15:units = "%" ;
> > PWS15:level = "Z10" ;
> >
> > // global attributes:
> > :FileOrigins = "Model Data" ;
> > :MET_version = "V6.0" ;
> > :Projection = "LatLon" ;
> > :lat_ll = "-89.921875 degrees_north" ;
> > :lon_ll = "0.117187 degrees_east" ;
> > :delta_lat = "0.234375 degrees" ;
> > :delta_lon = "0.15625 degrees" ;
> > :Nlat = "1152 grid_points" ;
> > :Nlon = "1536 grid_points" ;
> >
> > Any idea why the map is wrong - should I worry?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 22, 2017 3:04 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > Missing Value
> >
> > Bob,
> >
> > I grabbed the files you posted to our ftp site, but I noticed that
> > the timestamp of this file is older:
> > GridStatConfig_geps_updated
> >
> > Can you please make sure that file on the ftp site exactly matches
> > the one you're using:
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> >
> > I was able to get it to run using met-6.0 but unable to replicate
> > your error.
> >
> > So I want to make sure I'm testing with the same configuration.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Missing Value
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > >
> > >
> > >
> > > John, I have been running into an error when I use NCEP 2.5 deg
> > > climo for the climo reference. Below is the command line:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > When I run this I get the following error:
> > >
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = 90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > climatology
> > > field(s) valid at 20170619_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
->
> > > parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 65160 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > I have sent you all the files involved. When I run the code
using a
> > > netCDF file which is the same resolution as the FCST which
> > > contains the average frequency of the event I don't get this
> > > error. If I switch to using the ncep 2.5 deg climo and remap it
> > > to the FCST, I get
> > the error.
> > > Looking in all the files I don't see any values of -9999. So I
> > > don't know where this is coming from.
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 13:25:20 2017
Just set config file - it didn't seem to go the first time I sent.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 1:52 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I don't see your input files on the ftp site. Looking in:
incoming/irap/met_help
The following files do not exist:
grib2.2017061900.0024
GridStatConfig_geps_updated
Could you please post them to make sure I'm running exactly the same
case as you?
Thanks,
John
On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, With your help (thanks for that) I fixed the observation file
> and climo file issues and they now look right (the files are on the
> server). I also placed the most recent config file there. When I
run
> the following
> command:
>
> /h/WXQC/met-6.0/bin/grid_stat
> /h/data/global/WXQC/data/geps_verify/qcsave/
> 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60
> /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>
> I get the same error:
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -89.9219
> DEBUG 4: lon_ll = -0.117187
> DEBUG 4: delta_lat = 0.15625
> DEBUG 4: delta_lon = 0.234375
> DEBUG 4: Nlat = 1152
> DEBUG 4: Nlon = 1536
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_Gb2".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcMet".
> DEBUG 3: Use the forecast grid.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072977206617
> DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> verify/qcsave/2017061900/grib2.2017061900.0024
> DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> tmp/68bf9465-1763-4eb3-a42f-413092931b60
> DEBUG 3: Reading forecast data for PWS15/Z10.
> DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> grib2.2017061900.0024'
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 100
> DEBUG 4: scan mode: 0
> DEBUG 4: valid time: 20170620_000000
> DEBUG 4: lead time: 240000
> DEBUG 4: init time: 20170619_000000
> DEBUG 4: bitmap flag: 255
> DEBUG 4:
> DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> DEBUG 3: Reading observation data for PWS15(*,*).
> DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll = -90
> DEBUG 4: lon_ll = -0
> DEBUG 4: delta_lat = 2.5
> DEBUG 4: delta_lon = 2.5
> DEBUG 4: Nlat = 73
> DEBUG 4: Nlon = 144
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcMet".
> DEBUG 1: Regridding climotology PWS15(*,*) to the verification grid.
> DEBUG 3: For forecast valid at 20170620_000000, found 1 climatology
> field(s) valid at 20170620_000000.
> DEBUG 3: Found 1 climatology fields.
> DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2:
> DEBUG 2:
------------------------------------------------------------
> --------------------
> DEBUG 2:
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> interpolation method.
> DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> method NEAREST(1), over region FULL, using 64620 pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> ERROR :
> ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value ...
> -9999 ERROR :
>
> Let me know if you get the same error?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 11:10 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> Thanks for sending the sample observation data file. I ran it
through
> plot_data_plane and see what you mean. The earth doesn't fit the
data
> well. I suspect there's something wrong in your grid definition.
>
> Let's think about this. The lower left corner is defined as (lat,
> lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
>
> UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
(.15625
> *
> 1536) = 240.117
>
> You have an upper-right (lat, lon) = (180.51, 240.117), but you want
> it to be about (90, 360).
>
> Swapping the delta_lat and delta_lon values results in an upper-
right
> corner of (90.078, 360.117). I suspect that's the problem.
>
> I swapped those values using ncatted and replotted using
plot_data_plane.
> The resulting plot (attached) looks a lot better:
>
> ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> -a delta_lon,global,o,c,"0.234375 degrees" \
> -o obs_new.nc \
> 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
>
> met-6.0/bin/plot_data_plane \
> obs_new.nc obs_new.ps \
> 'name="PWS15"; level="(*,*)";'
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, I fixed the problem you found with my observation file.
After
> > I converted it to a netCDF file, I displayed it (on the ftp
server).
> > Note the map does not fit right. I placed the netCDF file on the
> > server. The header for the file seems correct:
> >
> > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > dimensions:
> > lat = 1152 ;
> > lon = 1536 ;
> > variables:
> > float PWS15(lat, lon) ;
> > PWS15:name = "PWS15" ;
> > PWS15:init_time_ut = "1497916800" ;
> > PWS15:valid_time = "20170620_000000" ;
> > PWS15:valid_time_ut = "1497916800" ;
> > PWS15:init_time = "20170620_000000" ;
> > PWS15:units = "%" ;
> > PWS15:level = "Z10" ;
> >
> > // global attributes:
> > :FileOrigins = "Model Data" ;
> > :MET_version = "V6.0" ;
> > :Projection = "LatLon" ;
> > :lat_ll = "-89.921875 degrees_north" ;
> > :lon_ll = "0.117187 degrees_east" ;
> > :delta_lat = "0.234375 degrees" ;
> > :delta_lon = "0.15625 degrees" ;
> > :Nlat = "1152 grid_points" ;
> > :Nlon = "1536 grid_points" ;
> >
> > Any idea why the map is wrong - should I worry?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 22, 2017 3:04 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > Missing Value
> >
> > Bob,
> >
> > I grabbed the files you posted to our ftp site, but I noticed that
> > the timestamp of this file is older:
> > GridStatConfig_geps_updated
> >
> > Can you please make sure that file on the ftp site exactly matches
> > the one you're using:
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> >
> > I was able to get it to run using met-6.0 but unable to replicate
> > your error.
> >
> > So I want to make sure I'm testing with the same configuration.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Missing Value
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > >
> > >
> > >
> > > John, I have been running into an error when I use NCEP 2.5 deg
> > > climo for the climo reference. Below is the command line:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > When I run this I get the following error:
> > >
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = 90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > climatology
> > > field(s) valid at 20170619_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
->
> > > parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 65160 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > I have sent you all the files involved. When I run the code
using a
> > > netCDF file which is the same resolution as the FCST which
> > > contains the average frequency of the event I don't get this
> > > error. If I switch to using the ncep 2.5 deg climo and remap it
> > > to the FCST, I get
> > the error.
> > > Looking in all the files I don't see any values of -9999. So I
> > > don't know where this is coming from.
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 14:18:24 2017
Bob,
I still don't have the forecast file you're referencing:
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900
.0024
The one of the ftp site is for the previous day:
grib2.2017061800.0024
I tried using the previous day's file in Grid-Stat, but it runs fine
(other
than a warning about the times not matching). So I'm not able to
replicate
your error. I really need to do that before I can debug it.
My guess is that this has to do with the grid definition issues you've
been
working through. But it really would be nice to replicate this so
that I
can make the code more robust in handling missing data values.
John
On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> Just set config file - it didn't seem to go the first time I sent.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 1:52 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I don't see your input files on the ftp site. Looking in:
> incoming/irap/met_help
>
> The following files do not exist:
> grib2.2017061900.0024
> GridStatConfig_geps_updated
>
> Could you please post them to make sure I'm running exactly the same
case
> as you?
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, With your help (thanks for that) I fixed the observation
file
> > and climo file issues and they now look right (the files are on
the
> > server). I also placed the most recent config file there. When I
run
> > the following
> > command:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > I get the same error:
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -89.9219
> > DEBUG 4: lon_ll = -0.117187
> > DEBUG 4: delta_lat = 0.15625
> > DEBUG 4: delta_lon = 0.234375
> > DEBUG 4: Nlat = 1152
> > DEBUG 4: Nlon = 1536
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_Gb2".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_NcMet".
> > DEBUG 3: Use the forecast grid.
> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744072977206617
> > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > verify/qcsave/2017061900/grib2.2017061900.0024
> > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > DEBUG 3: Reading forecast data for PWS15/Z10.
> > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> > grib2.2017061900.0024'
> > DEBUG 4:
> > DEBUG 4: Data plane information:
> > DEBUG 4: plane min: 0
> > DEBUG 4: plane max: 100
> > DEBUG 4: scan mode: 0
> > DEBUG 4: valid time: 20170620_000000
> > DEBUG 4: lead time: 240000
> > DEBUG 4: init time: 20170619_000000
> > DEBUG 4: bitmap flag: 255
> > DEBUG 4:
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170620_000000, found 1
climatology
> > field(s) valid at 20170620_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 64620 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > Let me know if you get the same error?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 11:10 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > Thanks for sending the sample observation data file. I ran it
through
> > plot_data_plane and see what you mean. The earth doesn't fit the
data
> > well. I suspect there's something wrong in your grid definition.
> >
> > Let's think about this. The lower left corner is defined as (lat,
> > lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
> >
> > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
(.15625
> > *
> > 1536) = 240.117
> >
> > You have an upper-right (lat, lon) = (180.51, 240.117), but you
want
> > it to be about (90, 360).
> >
> > Swapping the delta_lat and delta_lon values results in an upper-
right
> > corner of (90.078, 360.117). I suspect that's the problem.
> >
> > I swapped those values using ncatted and replotted using
plot_data_plane.
> > The resulting plot (attached) looks a lot better:
> >
> > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > -a delta_lon,global,o,c,"0.234375 degrees" \
> > -o obs_new.nc \
> > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> >
> > met-6.0/bin/plot_data_plane \
> > obs_new.nc obs_new.ps \
> > 'name="PWS15"; level="(*,*)";'
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > John, I fixed the problem you found with my observation file.
After
> > > I converted it to a netCDF file, I displayed it (on the ftp
server).
> > > Note the map does not fit right. I placed the netCDF file on
the
> > > server. The header for the file seems correct:
> > >
> > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > dimensions:
> > > lat = 1152 ;
> > > lon = 1536 ;
> > > variables:
> > > float PWS15(lat, lon) ;
> > > PWS15:name = "PWS15" ;
> > > PWS15:init_time_ut = "1497916800" ;
> > > PWS15:valid_time = "20170620_000000" ;
> > > PWS15:valid_time_ut = "1497916800" ;
> > > PWS15:init_time = "20170620_000000" ;
> > > PWS15:units = "%" ;
> > > PWS15:level = "Z10" ;
> > >
> > > // global attributes:
> > > :FileOrigins = "Model Data" ;
> > > :MET_version = "V6.0" ;
> > > :Projection = "LatLon" ;
> > > :lat_ll = "-89.921875 degrees_north" ;
> > > :lon_ll = "0.117187 degrees_east" ;
> > > :delta_lat = "0.234375 degrees" ;
> > > :delta_lon = "0.15625 degrees" ;
> > > :Nlat = "1152 grid_points" ;
> > > :Nlon = "1536 grid_points" ;
> > >
> > > Any idea why the map is wrong - should I worry?
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, June 22, 2017 3:04 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > I grabbed the files you posted to our ftp site, but I noticed
that
> > > the timestamp of this file is older:
> > > GridStatConfig_geps_updated
> > >
> > > Can you please make sure that file on the ftp site exactly
matches
> > > the one you're using:
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > >
> > > I was able to get it to run using met-6.0 but unable to
replicate
> > > your error.
> > >
> > > So I want to make sure I'm testing with the same configuration.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Grid Stat Missing Value
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > >
> > > >
> > > >
> > > > John, I have been running into an error when I use NCEP 2.5
deg
> > > > climo for the climo reference. Below is the command line:
> > > >
> > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > >
> > > > When I run this I get the following error:
> > > >
> > > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = 90
> > > > DEBUG 4: lon_ll = -0
> > > > DEBUG 4: delta_lat = 2.5
> > > > DEBUG 4: delta_lon = 2.5
> > > > DEBUG 4: Nlat = 73
> > > > DEBUG 4: Nlon = 144
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > climatology
> > > > field(s) valid at 20170619_000000.
> > > > DEBUG 3: Found 1 climatology fields.
> > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > DEBUG 2: Processing masking regions.
> > > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
->
> > > > parsing grid mask "FULL"
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > DEBUG 2:
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > > interpolation method.
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > > interpolation method.
> > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > > method NEAREST(1), over region FULL, using 65160 pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > ERROR :
> > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value ...
> > > > -9999 ERROR :
> > > >
> > > > I have sent you all the files involved. When I run the code
using a
> > > > netCDF file which is the same resolution as the FCST which
> > > > contains the average frequency of the event I don't get this
> > > > error. If I switch to using the ncep 2.5 deg climo and remap
it
> > > > to the FCST, I get
> > > the error.
> > > > Looking in all the files I don't see any values of -9999. So
I
> > > > don't know where this is coming from.
> > > >
> > > > Thanks
> > > > Bob Craig
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 14:21:53 2017
Okay, I forgot about the forecast file - you will have it shortly.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 3:18 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I still don't have the forecast file you're referencing:
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900
.0024
The one of the ftp site is for the previous day:
grib2.2017061800.0024
I tried using the previous day's file in Grid-Stat, but it runs fine
(other than a warning about the times not matching). So I'm not able
to replicate your error. I really need to do that before I can debug
it.
My guess is that this has to do with the grid definition issues you've
been working through. But it really would be nice to replicate this
so that I can make the code more robust in handling missing data
values.
John
On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> Just set config file - it didn't seem to go the first time I sent.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 1:52 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I don't see your input files on the ftp site. Looking in:
> incoming/irap/met_help
>
> The following files do not exist:
> grib2.2017061900.0024
> GridStatConfig_geps_updated
>
> Could you please post them to make sure I'm running exactly the same
> case as you?
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, With your help (thanks for that) I fixed the observation
file
> > and climo file issues and they now look right (the files are on
the
> > server). I also placed the most recent config file there. When I
> > run the following
> > command:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > I get the same error:
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -89.9219
> > DEBUG 4: lon_ll = -0.117187
> > DEBUG 4: delta_lat = 0.15625
> > DEBUG 4: delta_lon = 0.234375
> > DEBUG 4: Nlat = 1152
> > DEBUG 4: Nlon = 1536
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_Gb2".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 3: Use the forecast grid.
> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744072977206617
> > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > verify/qcsave/2017061900/grib2.2017061900.0024
> > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > DEBUG 3: Reading forecast data for PWS15/Z10.
> > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> > grib2.2017061900.0024'
> > DEBUG 4:
> > DEBUG 4: Data plane information:
> > DEBUG 4: plane min: 0
> > DEBUG 4: plane max: 100
> > DEBUG 4: scan mode: 0
> > DEBUG 4: valid time: 20170620_000000
> > DEBUG 4: lead time: 240000
> > DEBUG 4: init time: 20170619_000000
> > DEBUG 4: bitmap flag: 255
> > DEBUG 4:
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170620_000000, found 1
climatology
> > field(s) valid at 20170620_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
> > ------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 64620 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > Let me know if you get the same error?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 11:10 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > Thanks for sending the sample observation data file. I ran it
> > through plot_data_plane and see what you mean. The earth doesn't
> > fit the data well. I suspect there's something wrong in your grid
definition.
> >
> > Let's think about this. The lower left corner is defined as (lat,
> > lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
> >
> > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > (.15625
> > *
> > 1536) = 240.117
> >
> > You have an upper-right (lat, lon) = (180.51, 240.117), but you
want
> > it to be about (90, 360).
> >
> > Swapping the delta_lat and delta_lon values results in an
> > upper-right corner of (90.078, 360.117). I suspect that's the
problem.
> >
> > I swapped those values using ncatted and replotted using
plot_data_plane.
> > The resulting plot (attached) looks a lot better:
> >
> > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > -a delta_lon,global,o,c,"0.234375 degrees" \
> > -o obs_new.nc \
> > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> >
> > met-6.0/bin/plot_data_plane \
> > obs_new.nc obs_new.ps \
> > 'name="PWS15"; level="(*,*)";'
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > John, I fixed the problem you found with my observation file.
> > > After I converted it to a netCDF file, I displayed it (on the
ftp server).
> > > Note the map does not fit right. I placed the netCDF file on
the
> > > server. The header for the file seems correct:
> > >
> > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > dimensions:
> > > lat = 1152 ;
> > > lon = 1536 ;
> > > variables:
> > > float PWS15(lat, lon) ;
> > > PWS15:name = "PWS15" ;
> > > PWS15:init_time_ut = "1497916800" ;
> > > PWS15:valid_time = "20170620_000000" ;
> > > PWS15:valid_time_ut = "1497916800" ;
> > > PWS15:init_time = "20170620_000000" ;
> > > PWS15:units = "%" ;
> > > PWS15:level = "Z10" ;
> > >
> > > // global attributes:
> > > :FileOrigins = "Model Data" ;
> > > :MET_version = "V6.0" ;
> > > :Projection = "LatLon" ;
> > > :lat_ll = "-89.921875 degrees_north" ;
> > > :lon_ll = "0.117187 degrees_east" ;
> > > :delta_lat = "0.234375 degrees" ;
> > > :delta_lon = "0.15625 degrees" ;
> > > :Nlat = "1152 grid_points" ;
> > > :Nlon = "1536 grid_points" ;
> > >
> > > Any idea why the map is wrong - should I worry?
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, June 22, 2017 3:04 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > I grabbed the files you posted to our ftp site, but I noticed
that
> > > the timestamp of this file is older:
> > > GridStatConfig_geps_updated
> > >
> > > Can you please make sure that file on the ftp site exactly
matches
> > > the one you're using:
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > >
> > > I was able to get it to run using met-6.0 but unable to
replicate
> > > your error.
> > >
> > > So I want to make sure I'm testing with the same configuration.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Grid Stat Missing Value
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > >
> > > >
> > > >
> > > > John, I have been running into an error when I use NCEP 2.5
deg
> > > > climo for the climo reference. Below is the command line:
> > > >
> > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > >
> > > > When I run this I get the following error:
> > > >
> > > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = 90
> > > > DEBUG 4: lon_ll = -0
> > > > DEBUG 4: delta_lat = 2.5
> > > > DEBUG 4: delta_lon = 2.5
> > > > DEBUG 4: Nlat = 73
> > > > DEBUG 4: Nlon = 144
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > climatology
> > > > field(s) valid at 20170619_000000.
> > > > DEBUG 3: Found 1 climatology fields.
> > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > DEBUG 2: Processing masking regions.
> > > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
> > > > -> parsing grid mask "FULL"
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > DEBUG 2:
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > interpolation method NEAREST(1), over region FULL, using 65160
pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > ERROR :
> > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value ...
> > > > -9999 ERROR :
> > > >
> > > > I have sent you all the files involved. When I run the code
using a
> > > > netCDF file which is the same resolution as the FCST which
> > > > contains the average frequency of the event I don't get this
> > > > error. If I switch to using the ncep 2.5 deg climo and remap
it
> > > > to the FCST, I get
> > > the error.
> > > > Looking in all the files I don't see any values of -9999. So
I
> > > > don't know where this is coming from.
> > > >
> > > > Thanks
> > > > Bob Craig
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Fri Jun 23 14:29:27 2017
You should have it now. By the way, the forecast file has 65160 data
points but in the MET output listing it says it is using 64620 for the
full region. Could the missing points here be the -9999 values?
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 3:18 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I still don't have the forecast file you're referencing:
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900
.0024
The one of the ftp site is for the previous day:
grib2.2017061800.0024
I tried using the previous day's file in Grid-Stat, but it runs fine
(other than a warning about the times not matching). So I'm not able
to replicate your error. I really need to do that before I can debug
it.
My guess is that this has to do with the grid definition issues you've
been working through. But it really would be nice to replicate this
so that I can make the code more robust in handling missing data
values.
John
On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> Just set config file - it didn't seem to go the first time I sent.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 1:52 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I don't see your input files on the ftp site. Looking in:
> incoming/irap/met_help
>
> The following files do not exist:
> grib2.2017061900.0024
> GridStatConfig_geps_updated
>
> Could you please post them to make sure I'm running exactly the same
> case as you?
>
> Thanks,
> John
>
>
> On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, With your help (thanks for that) I fixed the observation
file
> > and climo file issues and they now look right (the files are on
the
> > server). I also placed the most recent config file there. When I
> > run the following
> > command:
> >
> > /h/WXQC/met-6.0/bin/grid_stat
> > /h/data/global/WXQC/data/geps_verify/qcsave/
> > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> >
> > I get the same error:
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -89.9219
> > DEBUG 4: lon_ll = -0.117187
> > DEBUG 4: delta_lat = 0.15625
> > DEBUG 4: delta_lon = 0.234375
> > DEBUG 4: Nlat = 1152
> > DEBUG 4: Nlon = 1536
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_Gb2".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 3: Use the forecast grid.
> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744072977206617
> > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > verify/qcsave/2017061900/grib2.2017061900.0024
> > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > DEBUG 3: Reading forecast data for PWS15/Z10.
> > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/
> > grib2.2017061900.0024'
> > DEBUG 4:
> > DEBUG 4: Data plane information:
> > DEBUG 4: plane min: 0
> > DEBUG 4: plane max: 100
> > DEBUG 4: scan mode: 0
> > DEBUG 4: valid time: 20170620_000000
> > DEBUG 4: lead time: 240000
> > DEBUG 4: init time: 20170619_000000
> > DEBUG 4: bitmap flag: 255
> > DEBUG 4:
> > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > DEBUG 3: Reading observation data for PWS15(*,*).
> > DEBUG 1: Regridding forecast PWS15(*,*) to the verification grid.
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll = -90
> > DEBUG 4: lon_ll = -0
> > DEBUG 4: delta_lat = 2.5
> > DEBUG 4: delta_lon = 2.5
> > DEBUG 4: Nlat = 73
> > DEBUG 4: Nlon = 144
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_NcMet".
> > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > DEBUG 3: For forecast valid at 20170620_000000, found 1
climatology
> > field(s) valid at 20170620_000000.
> > DEBUG 3: Found 1 climatology fields.
> > DEBUG 3: Found 1 climatology mean field(s) for forecast PWS15/Z10.
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2:
> > DEBUG 2:
> > ------------------------------------------------------------
> > --------------------
> > DEBUG 2:
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 3: Smoothing field using the NEAREST(1) RectangularTemplate
> > interpolation method.
> > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for interpolation
> > method NEAREST(1), over region FULL, using 64620 pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > ERROR :
> > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > -9999 ERROR :
> >
> > Let me know if you get the same error?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 11:10 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > Thanks for sending the sample observation data file. I ran it
> > through plot_data_plane and see what you mean. The earth doesn't
> > fit the data well. I suspect there's something wrong in your grid
definition.
> >
> > Let's think about this. The lower left corner is defined as (lat,
> > lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
> >
> > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > (.15625
> > *
> > 1536) = 240.117
> >
> > You have an upper-right (lat, lon) = (180.51, 240.117), but you
want
> > it to be about (90, 360).
> >
> > Swapping the delta_lat and delta_lon values results in an
> > upper-right corner of (90.078, 360.117). I suspect that's the
problem.
> >
> > I swapped those values using ncatted and replotted using
plot_data_plane.
> > The resulting plot (attached) looks a lot better:
> >
> > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > -a delta_lon,global,o,c,"0.234375 degrees" \
> > -o obs_new.nc \
> > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> >
> > met-6.0/bin/plot_data_plane \
> > obs_new.nc obs_new.ps \
> > 'name="PWS15"; level="(*,*)";'
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > John, I fixed the problem you found with my observation file.
> > > After I converted it to a netCDF file, I displayed it (on the
ftp server).
> > > Note the map does not fit right. I placed the netCDF file on
the
> > > server. The header for the file seems correct:
> > >
> > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > dimensions:
> > > lat = 1152 ;
> > > lon = 1536 ;
> > > variables:
> > > float PWS15(lat, lon) ;
> > > PWS15:name = "PWS15" ;
> > > PWS15:init_time_ut = "1497916800" ;
> > > PWS15:valid_time = "20170620_000000" ;
> > > PWS15:valid_time_ut = "1497916800" ;
> > > PWS15:init_time = "20170620_000000" ;
> > > PWS15:units = "%" ;
> > > PWS15:level = "Z10" ;
> > >
> > > // global attributes:
> > > :FileOrigins = "Model Data" ;
> > > :MET_version = "V6.0" ;
> > > :Projection = "LatLon" ;
> > > :lat_ll = "-89.921875 degrees_north" ;
> > > :lon_ll = "0.117187 degrees_east" ;
> > > :delta_lat = "0.234375 degrees" ;
> > > :delta_lon = "0.15625 degrees" ;
> > > :Nlat = "1152 grid_points" ;
> > > :Nlon = "1536 grid_points" ;
> > >
> > > Any idea why the map is wrong - should I worry?
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, June 22, 2017 3:04 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > I grabbed the files you posted to our ftp site, but I noticed
that
> > > the timestamp of this file is older:
> > > GridStatConfig_geps_updated
> > >
> > > Can you please make sure that file on the ftp site exactly
matches
> > > the one you're using:
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > >
> > > I was able to get it to run using met-6.0 but unable to
replicate
> > > your error.
> > >
> > > So I want to make sure I'm testing with the same configuration.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Grid Stat Missing Value
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > >
> > > >
> > > >
> > > > John, I have been running into an error when I use NCEP 2.5
deg
> > > > climo for the climo reference. Below is the command line:
> > > >
> > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > 2017061800/grib2.2017061800.0024 /h/data/global/WXQC/data/met/
> > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > >
> > > > When I run this I get the following error:
> > > >
> > > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = 90
> > > > DEBUG 4: lon_ll = -0
> > > > DEBUG 4: delta_lat = 2.5
> > > > DEBUG 4: delta_lon = 2.5
> > > > DEBUG 4: Nlat = 73
> > > > DEBUG 4: Nlon = 144
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > climatology
> > > > field(s) valid at 20170619_000000.
> > > > DEBUG 3: Found 1 climatology fields.
> > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > DEBUG 2: Processing masking regions.
> > > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
> > > > -> parsing grid mask "FULL"
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > DEBUG 2:
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > interpolation method NEAREST(1), over region FULL, using 65160
pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > ERROR :
> > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value ...
> > > > -9999 ERROR :
> > > >
> > > > I have sent you all the files involved. When I run the code
using a
> > > > netCDF file which is the same resolution as the FCST which
> > > > contains the average frequency of the event I don't get this
> > > > error. If I switch to using the ncep 2.5 deg climo and remap
it
> > > > to the FCST, I get
> > > the error.
> > > > Looking in all the files I don't see any values of -9999. So
I
> > > > don't know where this is coming from.
> > > >
> > > > Thanks
> > > > Bob Craig
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 16:03:59 2017
Bob,
I was finally able to replicate your error. Turns out that while
Grid-Stat
was running without error, it wasn't actually producing any output.
So I
wasn't actually exercising the problematic piece of code.
Yes, I can confirm that the problem is caused by missing data values
in the
climatology field.
To make the testing even easier, I regridded your climo data to a very
small grid so that almost all the climo values would be bad data.
I'm working on updating Grid-Stat to make sure that when the user
specifies
a climatology field, we only compute stats at grid points where the
forecast, observation, and climatology all contain valid data.
Unfortunately, this will require a patch and recompiling.
John
On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> You should have it now. By the way, the forecast file has 65160
data
> points but in the MET output listing it says it is using 64620 for
the full
> region. Could the missing points here be the -9999 values?
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 3:18 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I still don't have the forecast file you're referencing:
>
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061900
> .0024
>
> The one of the ftp site is for the previous day:
> grib2.2017061800.0024
>
> I tried using the previous day's file in Grid-Stat, but it runs fine
> (other than a warning about the times not matching). So I'm not
able to
> replicate your error. I really need to do that before I can debug
it.
>
> My guess is that this has to do with the grid definition issues
you've
> been working through. But it really would be nice to replicate this
so
> that I can make the code more robust in handling missing data
values.
>
> John
>
> On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > Just set config file - it didn't seem to go the first time I sent.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 1:52 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > I don't see your input files on the ftp site. Looking in:
> > incoming/irap/met_help
> >
> > The following files do not exist:
> > grib2.2017061900.0024
> > GridStatConfig_geps_updated
> >
> > Could you please post them to make sure I'm running exactly the
same
> > case as you?
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > John, With your help (thanks for that) I fixed the observation
file
> > > and climo file issues and they now look right (the files are on
the
> > > server). I also placed the most recent config file there. When
I
> > > run the following
> > > command:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > I get the same error:
> > >
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = -89.9219
> > > DEBUG 4: lon_ll = -0.117187
> > > DEBUG 4: delta_lat = 0.15625
> > > DEBUG 4: delta_lon = 0.234375
> > > DEBUG 4: Nlat = 1152
> > > DEBUG 4: Nlon = 1536
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_Gb2".
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 3: Use the forecast grid.
> > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > > GSL_RNG_TYPE=mt19937
> > > GSL_RNG_SEED=18446744072977206617
> > > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
> <(201)%20706-1900>.0024
> > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > DEBUG 3: Reading forecast data for PWS15/Z10.
> > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
> <(201)%20706-1900>/
> > > grib2.2017061900 <(201)%20706-1900>.0024'
> > > DEBUG 4:
> > > DEBUG 4: Data plane information:
> > > DEBUG 4: plane min: 0
> > > DEBUG 4: plane max: 100
> > > DEBUG 4: scan mode: 0
> > > DEBUG 4: valid time: 20170620_000000
> > > DEBUG 4: lead time: 240000
> > > DEBUG 4: init time: 20170619_000000
> > > DEBUG 4: bitmap flag: 255
> > > DEBUG 4:
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = -90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170620_000000, found 1
climatology
> > > field(s) valid at 20170620_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL
> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 64620 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > Let me know if you get the same error?
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, June 23, 2017 11:10 AM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > Thanks for sending the sample observation data file. I ran it
> > > through plot_data_plane and see what you mean. The earth
doesn't
> > > fit the data well. I suspect there's something wrong in your
grid
> definition.
> > >
> > > Let's think about this. The lower left corner is defined as
(lat,
> > > lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
> > >
> > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > > (.15625
> > > *
> > > 1536) = 240.117
> > >
> > > You have an upper-right (lat, lon) = (180.51, 240.117), but you
want
> > > it to be about (90, 360).
> > >
> > > Swapping the delta_lat and delta_lon values results in an
> > > upper-right corner of (90.078, 360.117). I suspect that's the
problem.
> > >
> > > I swapped those values using ncatted and replotted using
> plot_data_plane.
> > > The resulting plot (attached) looks a lot better:
> > >
> > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > > -a delta_lon,global,o,c,"0.234375 degrees" \
> > > -o obs_new.nc \
> > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> > >
> > > met-6.0/bin/plot_data_plane \
> > > obs_new.nc obs_new.ps \
> > > 'name="PWS15"; level="(*,*)";'
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
> > > >
> > > > John, I fixed the problem you found with my observation file.
> > > > After I converted it to a netCDF file, I displayed it (on the
ftp
> server).
> > > > Note the map does not fit right. I placed the netCDF file on
the
> > > > server. The header for the file seems correct:
> > > >
> > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > > dimensions:
> > > > lat = 1152 ;
> > > > lon = 1536 ;
> > > > variables:
> > > > float PWS15(lat, lon) ;
> > > > PWS15:name = "PWS15" ;
> > > > PWS15:init_time_ut = "1497916800" ;
> > > > PWS15:valid_time = "20170620_000000" ;
> > > > PWS15:valid_time_ut = "1497916800" ;
> > > > PWS15:init_time = "20170620_000000" ;
> > > > PWS15:units = "%" ;
> > > > PWS15:level = "Z10" ;
> > > >
> > > > // global attributes:
> > > > :FileOrigins = "Model Data" ;
> > > > :MET_version = "V6.0" ;
> > > > :Projection = "LatLon" ;
> > > > :lat_ll = "-89.921875 degrees_north" ;
> > > > :lon_ll = "0.117187 degrees_east" ;
> > > > :delta_lat = "0.234375 degrees" ;
> > > > :delta_lon = "0.15625 degrees" ;
> > > > :Nlat = "1152 grid_points" ;
> > > > :Nlon = "1536 grid_points" ;
> > > >
> > > > Any idea why the map is wrong - should I worry?
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, June 22, 2017 3:04 PM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > > Missing Value
> > > >
> > > > Bob,
> > > >
> > > > I grabbed the files you posted to our ftp site, but I noticed
that
> > > > the timestamp of this file is older:
> > > > GridStatConfig_geps_updated
> > > >
> > > > Can you please make sure that file on the ftp site exactly
matches
> > > > the one you're using:
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > >
> > > > I was able to get it to run using met-6.0 but unable to
replicate
> > > > your error.
> > > >
> > > > So I want to make sure I'm testing with the same
configuration.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: Grid Stat Missing Value
> > > > > Owner: Nobody
> > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > >
> > > > >
> > > > >
> > > > > John, I have been running into an error when I use NCEP 2.5
deg
> > > > > climo for the climo reference. Below is the command line:
> > > > >
> > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > 2017061800/grib2.2017061800.0024
/h/data/global/WXQC/data/met/
> > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > >
> > > > > When I run this I get the following error:
> > > > >
> > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > DEBUG 4: lat_ll = 90
> > > > > DEBUG 4: lon_ll = -0
> > > > > DEBUG 4: delta_lat = 2.5
> > > > > DEBUG 4: delta_lon = 2.5
> > > > > DEBUG 4: Nlat = 73
> > > > > DEBUG 4: Nlon = 144
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > object of type "FileType_NcMet".
> > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
verification
> grid.
> > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > > climatology
> > > > > field(s) valid at 20170619_000000.
> > > > > DEBUG 3: Found 1 climatology fields.
> > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > > DEBUG 2: Processing masking regions.
> > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
parse_grid_mask()
> > > > > -> parsing grid mask "FULL"
> > > > > DEBUG 2:
> > > > > DEBUG 2:
> > > > > ------------------------------------------------------------
> > > > > --------------------
> > > > > DEBUG 2:
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > interpolation method NEAREST(1), over region FULL, using
65160
> pairs.
> > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > ERROR :
> > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value
> ...
> > > > > -9999 ERROR :
> > > > >
> > > > > I have sent you all the files involved. When I run the
code
> using a
> > > > > netCDF file which is the same resolution as the FCST which
> > > > > contains the average frequency of the event I don't get this
> > > > > error. If I switch to using the ncep 2.5 deg climo and
remap it
> > > > > to the FCST, I get
> > > > the error.
> > > > > Looking in all the files I don't see any values of -9999.
So I
> > > > > don't know where this is coming from.
> > > > >
> > > > > Thanks
> > > > > Bob Craig
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Fri Jun 23 16:37:27 2017
Bob,
OK, I found a pretty simple fix that only affects grid_stat.cc and
posted
it as a bugfix on the MET website:
http://www.dtcenter.org/met/users/support/known_issues/METv6.0/index.php
Recompile with the fix and verify that it solves the problem you
found.
Thanks for uncovering this issue!
John
On Fri, Jun 23, 2017 at 4:03 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Bob,
>
> I was finally able to replicate your error. Turns out that while
> Grid-Stat was running without error, it wasn't actually producing
any
> output. So I wasn't actually exercising the problematic piece of
code.
>
> Yes, I can confirm that the problem is caused by missing data values
in
> the climatology field.
>
> To make the testing even easier, I regridded your climo data to a
very
> small grid so that almost all the climo values would be bad data.
>
> I'm working on updating Grid-Stat to make sure that when the user
> specifies a climatology field, we only compute stats at grid points
where
> the forecast, observation, and climatology all contain valid data.
>
> Unfortunately, this will require a patch and recompiling.
>
> John
>
>
> On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>>
>> You should have it now. By the way, the forecast file has 65160
data
>> points but in the MET output listing it says it is using 64620 for
the full
>> region. Could the missing points here be the -9999 values?
>>
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Friday, June 23, 2017 3:18 PM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
>> Missing Value
>>
>> Bob,
>>
>> I still don't have the forecast file you're referencing:
>> /h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.
>> 2017061900
>> .0024
>>
>> The one of the ftp site is for the previous day:
>> grib2.2017061800.0024
>>
>> I tried using the previous day's file in Grid-Stat, but it runs
fine
>> (other than a warning about the times not matching). So I'm not
able to
>> replicate your error. I really need to do that before I can debug
it.
>>
>> My guess is that this has to do with the grid definition issues
you've
>> been working through. But it really would be nice to replicate
this so
>> that I can make the code more robust in handling missing data
values.
>>
>> John
>>
>> On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>> >
>> > Just set config file - it didn't seem to go the first time I
sent.
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Friday, June 23, 2017 1:52 PM
>> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > <robert.craig.2 at us.af.mil>
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
>> > Missing Value
>> >
>> > Bob,
>> >
>> > I don't see your input files on the ftp site. Looking in:
>> > incoming/irap/met_help
>> >
>> > The following files do not exist:
>> > grib2.2017061900.0024
>> > GridStatConfig_geps_updated
>> >
>> > Could you please post them to make sure I'm running exactly the
same
>> > case as you?
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT
<
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>> > >
>> > > John, With your help (thanks for that) I fixed the observation
file
>> > > and climo file issues and they now look right (the files are on
the
>> > > server). I also placed the most recent config file there.
When I
>> > > run the following
>> > > command:
>> > >
>> > > /h/WXQC/met-6.0/bin/grid_stat
>> > > /h/data/global/WXQC/data/geps_verify/qcsave/
>> > > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
>> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
>> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>> > >
>> > > I get the same error:
>> > >
>> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
>> > > Met2dDataFile object of type "FileType_NcMet".
>> > > DEBUG 4: Latitude/Longitude Grid Data:
>> > > DEBUG 4: lat_ll = -89.9219
>> > > DEBUG 4: lon_ll = -0.117187
>> > > DEBUG 4: delta_lat = 0.15625
>> > > DEBUG 4: delta_lon = 0.234375
>> > > DEBUG 4: Nlat = 1152
>> > > DEBUG 4: Nlon = 1536
>> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> > > object of type "FileType_Gb2".
>> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> > > object of type "FileType_NcMet".
>> > > DEBUG 3: Use the forecast grid.
>> > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
>> > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
>> > > GSL_RNG_TYPE=mt19937
>> > > GSL_RNG_SEED=18446744072977206617
>> > > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
>> > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
>> <(201)%20706-1900>.0024
>> > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
>> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
>> > > DEBUG 3: Reading forecast data for PWS15/Z10.
>> > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
>> > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
>> > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
>> <(201)%20706-1900>/
>> > > grib2.2017061900 <(201)%20706-1900>.0024'
>> > > DEBUG 4:
>> > > DEBUG 4: Data plane information:
>> > > DEBUG 4: plane min: 0
>> > > DEBUG 4: plane max: 100
>> > > DEBUG 4: scan mode: 0
>> > > DEBUG 4: valid time: 20170620_000000
>> > > DEBUG 4: lead time: 240000
>> > > DEBUG 4: init time: 20170619_000000
>> > > DEBUG 4: bitmap flag: 255
>> > > DEBUG 4:
>> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
>> > > DEBUG 3: Reading observation data for PWS15(*,*).
>> > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
>> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
>> > > Met2dDataFile object of type "FileType_NcMet".
>> > > DEBUG 4: Latitude/Longitude Grid Data:
>> > > DEBUG 4: lat_ll = -90
>> > > DEBUG 4: lon_ll = -0
>> > > DEBUG 4: delta_lat = 2.5
>> > > DEBUG 4: delta_lon = 2.5
>> > > DEBUG 4: Nlat = 73
>> > > DEBUG 4: Nlon = 144
>> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> > > object of type "FileType_NcMet".
>> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
>> > > DEBUG 3: For forecast valid at 20170620_000000, found 1
climatology
>> > > field(s) valid at 20170620_000000.
>> > > DEBUG 3: Found 1 climatology fields.
>> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
>> > > DEBUG 2: Processing masking regions.
>> > > DEBUG 3: Processing grid mask: FULL
>> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > > ------------------------------------------------------------
>> > > --------------------
>> > > DEBUG 2:
>> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
>> > > interpolation method.
>> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
>> > > interpolation method.
>> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
>> > > method NEAREST(1), over region FULL, using 64620 pairs.
>> > > DEBUG 2: Computing Probabilistic Statistics.
>> > > ERROR :
>> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
>> > > -9999 ERROR :
>> > >
>> > > Let me know if you get the same error?
>> > >
>> > > Thanks
>> > > Bob
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > > Sent: Friday, June 23, 2017 11:10 AM
>> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > <robert.craig.2 at us.af.mil>
>> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
>> > > Missing Value
>> > >
>> > > Bob,
>> > >
>> > > Thanks for sending the sample observation data file. I ran it
>> > > through plot_data_plane and see what you mean. The earth
doesn't
>> > > fit the data well. I suspect there's something wrong in your
grid
>> definition.
>> > >
>> > > Let's think about this. The lower left corner is defined as
(lat,
>> > > lon) = (-89.92, 0.117). Let's calculate the upper-right
corner:
>> > >
>> > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
>> > > (.15625
>> > > *
>> > > 1536) = 240.117
>> > >
>> > > You have an upper-right (lat, lon) = (180.51, 240.117), but you
want
>> > > it to be about (90, 360).
>> > >
>> > > Swapping the delta_lat and delta_lon values results in an
>> > > upper-right corner of (90.078, 360.117). I suspect that's the
>> problem.
>> > >
>> > > I swapped those values using ncatted and replotted using
>> plot_data_plane.
>> > > The resulting plot (attached) looks a lot better:
>> > >
>> > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
>> > > -a delta_lon,global,o,c,"0.234375 degrees" \
>> > > -o obs_new.nc \
>> > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
>> > >
>> > > met-6.0/bin/plot_data_plane \
>> > > obs_new.nc obs_new.ps \
>> > > 'name="PWS15"; level="(*,*)";'
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > >
>> > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via
RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
>> > > >
>> > > > John, I fixed the problem you found with my observation file.
>> > > > After I converted it to a netCDF file, I displayed it (on the
ftp
>> server).
>> > > > Note the map does not fit right. I placed the netCDF file on
the
>> > > > server. The header for the file seems correct:
>> > > >
>> > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
>> > > > dimensions:
>> > > > lat = 1152 ;
>> > > > lon = 1536 ;
>> > > > variables:
>> > > > float PWS15(lat, lon) ;
>> > > > PWS15:name = "PWS15" ;
>> > > > PWS15:init_time_ut = "1497916800" ;
>> > > > PWS15:valid_time = "20170620_000000" ;
>> > > > PWS15:valid_time_ut = "1497916800" ;
>> > > > PWS15:init_time = "20170620_000000" ;
>> > > > PWS15:units = "%" ;
>> > > > PWS15:level = "Z10" ;
>> > > >
>> > > > // global attributes:
>> > > > :FileOrigins = "Model Data" ;
>> > > > :MET_version = "V6.0" ;
>> > > > :Projection = "LatLon" ;
>> > > > :lat_ll = "-89.921875 degrees_north" ;
>> > > > :lon_ll = "0.117187 degrees_east" ;
>> > > > :delta_lat = "0.234375 degrees" ;
>> > > > :delta_lon = "0.15625 degrees" ;
>> > > > :Nlat = "1152 grid_points" ;
>> > > > :Nlon = "1536 grid_points" ;
>> > > >
>> > > > Any idea why the map is wrong - should I worry?
>> > > >
>> > > > Thanks
>> > > > Bob
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > > > Sent: Thursday, June 22, 2017 3:04 PM
>> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > > <robert.craig.2 at us.af.mil>
>> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
>> > > > Missing Value
>> > > >
>> > > > Bob,
>> > > >
>> > > > I grabbed the files you posted to our ftp site, but I noticed
that
>> > > > the timestamp of this file is older:
>> > > > GridStatConfig_geps_updated
>> > > >
>> > > > Can you please make sure that file on the ftp site exactly
matches
>> > > > the one you're using:
>> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>> > > >
>> > > > I was able to get it to run using met-6.0 but unable to
replicate
>> > > > your error.
>> > > >
>> > > > So I want to make sure I'm testing with the same
configuration.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > >
>> > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via
RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
>> > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> > > > > Queue: met_help
>> > > > > Subject: Grid Stat Missing Value
>> > > > > Owner: Nobody
>> > > > > Requestors: robert.craig.2 at us.af.mil
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>> > > > > >
>> > > > >
>> > > > >
>> > > > > John, I have been running into an error when I use NCEP 2.5
deg
>> > > > > climo for the climo reference. Below is the command line:
>> > > > >
>> > > > > /h/WXQC/met-6.0/bin/grid_stat
>> > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
>> > > > > 2017061800/grib2.2017061800.0024
/h/data/global/WXQC/data/met/
>> > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
>> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
>> > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
>> > > > >
>> > > > > When I run this I get the following error:
>> > > > >
>> > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
>> > > > > DEBUG 3: Reading observation data for PWS15(*,*).
>> > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> > > > > new Met2dDataFile object of type "FileType_NcMet".
>> > > > > DEBUG 4: Latitude/Longitude Grid Data:
>> > > > > DEBUG 4: lat_ll = 90
>> > > > > DEBUG 4: lon_ll = -0
>> > > > > DEBUG 4: delta_lat = 2.5
>> > > > > DEBUG 4: delta_lon = 2.5
>> > > > > DEBUG 4: Nlat = 73
>> > > > > DEBUG 4: Nlon = 144
>> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>> > > > > object of type "FileType_NcMet".
>> > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
verification
>> grid.
>> > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
>> > > > > climatology
>> > > > > field(s) valid at 20170619_000000.
>> > > > > DEBUG 3: Found 1 climatology fields.
>> > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
>> > > > > DEBUG 2: Processing masking regions.
>> > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
parse_grid_mask()
>> > > > > -> parsing grid mask "FULL"
>> > > > > DEBUG 2:
>> > > > > DEBUG 2:
>> > > > >
------------------------------------------------------------
>> > > > > --------------------
>> > > > > DEBUG 2:
>> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
>> > > > > RectangularTemplate interpolation method.
>> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
>> > > > > RectangularTemplate interpolation method.
>> > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
>> > > > > interpolation method NEAREST(1), over region FULL, using
65160
>> pairs.
>> > > > > DEBUG 2: Computing Probabilistic Statistics.
>> > > > > ERROR :
>> > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value
>> ...
>> > > > > -9999 ERROR :
>> > > > >
>> > > > > I have sent you all the files involved. When I run the
code
>> using a
>> > > > > netCDF file which is the same resolution as the FCST which
>> > > > > contains the average frequency of the event I don't get
this
>> > > > > error. If I switch to using the ncep 2.5 deg climo and
remap it
>> > > > > to the FCST, I get
>> > > > the error.
>> > > > > Looking in all the files I don't see any values of -9999.
So I
>> > > > > don't know where this is coming from.
>> > > > >
>> > > > > Thanks
>> > > > > Bob Craig
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Mon Jun 26 07:47:08 2017
John, I am glad you found that issue. Please place the fix on the ftp
site in my directory since the download site is blocked here.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, June 23, 2017 5:04 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
I was finally able to replicate your error. Turns out that while
Grid-Stat was running without error, it wasn't actually producing any
output. So I wasn't actually exercising the problematic piece of
code.
Yes, I can confirm that the problem is caused by missing data values
in the climatology field.
To make the testing even easier, I regridded your climo data to a very
small grid so that almost all the climo values would be bad data.
I'm working on updating Grid-Stat to make sure that when the user
specifies a climatology field, we only compute stats at grid points
where the forecast, observation, and climatology all contain valid
data.
Unfortunately, this will require a patch and recompiling.
John
On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> You should have it now. By the way, the forecast file has 65160
data
> points but in the MET output listing it says it is using 64620 for
the
> full region. Could the missing points here be the -9999 values?
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 3:18 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I still don't have the forecast file you're referencing:
>
>
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.201706190
> 0
> .0024
>
> The one of the ftp site is for the previous day:
> grib2.2017061800.0024
>
> I tried using the previous day's file in Grid-Stat, but it runs fine
> (other than a warning about the times not matching). So I'm not
able
> to replicate your error. I really need to do that before I can
debug it.
>
> My guess is that this has to do with the grid definition issues
you've
> been working through. But it really would be nice to replicate this
> so that I can make the code more robust in handling missing data
values.
>
> John
>
> On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > Just set config file - it didn't seem to go the first time I sent.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 1:52 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > I don't see your input files on the ftp site. Looking in:
> > incoming/irap/met_help
> >
> > The following files do not exist:
> > grib2.2017061900.0024
> > GridStatConfig_geps_updated
> >
> > Could you please post them to make sure I'm running exactly the
same
> > case as you?
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > John, With your help (thanks for that) I fixed the observation
> > > file and climo file issues and they now look right (the files
are
> > > on the server). I also placed the most recent config file
there.
> > > When I run the following
> > > command:
> > >
> > > /h/WXQC/met-6.0/bin/grid_stat
> > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > 2017061900/grib2.2017061900.0024 /h/data/global/WXQC/data/met/
> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > >
> > > I get the same error:
> > >
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = -89.9219
> > > DEBUG 4: lon_ll = -0.117187
> > > DEBUG 4: delta_lat = 0.15625
> > > DEBUG 4: delta_lon = 0.234375
> > > DEBUG 4: Nlat = 1152
> > > DEBUG 4: Nlon = 1536
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_Gb2".
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 3: Use the forecast grid.
> > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > > GSL_RNG_TYPE=mt19937
> > > GSL_RNG_SEED=18446744072977206617
> > > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
> <(201)%20706-1900>.0024
> > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > DEBUG 3: Reading forecast data for PWS15/Z10.
> > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
> <(201)%20706-1900>/
> > > grib2.2017061900 <(201)%20706-1900>.0024'
> > > DEBUG 4:
> > > DEBUG 4: Data plane information:
> > > DEBUG 4: plane min: 0
> > > DEBUG 4: plane max: 100
> > > DEBUG 4: scan mode: 0
> > > DEBUG 4: valid time: 20170620_000000
> > > DEBUG 4: lead time: 240000
> > > DEBUG 4: init time: 20170619_000000
> > > DEBUG 4: bitmap flag: 255
> > > DEBUG 4:
> > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll = -90
> > > DEBUG 4: lon_ll = -0
> > > DEBUG 4: delta_lat = 2.5
> > > DEBUG 4: delta_lon = 2.5
> > > DEBUG 4: Nlat = 73
> > > DEBUG 4: Nlon = 144
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_NcMet".
> > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > DEBUG 3: For forecast valid at 20170620_000000, found 1
> > > climatology
> > > field(s) valid at 20170620_000000.
> > > DEBUG 3: Found 1 climatology fields.
> > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
->
> > > parsing grid mask "FULL"
> > > DEBUG 2:
> > > DEBUG 2:
> > > ------------------------------------------------------------
> > > --------------------
> > > DEBUG 2:
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > interpolation method.
> > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > method NEAREST(1), over region FULL, using 64620 pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > ERROR :
> > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad value
...
> > > -9999 ERROR :
> > >
> > > Let me know if you get the same error?
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, June 23, 2017 11:10 AM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
> > > Stat Missing Value
> > >
> > > Bob,
> > >
> > > Thanks for sending the sample observation data file. I ran it
> > > through plot_data_plane and see what you mean. The earth
doesn't
> > > fit the data well. I suspect there's something wrong in your
grid
> definition.
> > >
> > > Let's think about this. The lower left corner is defined as
(lat,
> > > lon) = (-89.92, 0.117). Let's calculate the upper-right corner:
> > >
> > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > > (.15625
> > > *
> > > 1536) = 240.117
> > >
> > > You have an upper-right (lat, lon) = (180.51, 240.117), but you
> > > want it to be about (90, 360).
> > >
> > > Swapping the delta_lat and delta_lon values results in an
> > > upper-right corner of (90.078, 360.117). I suspect that's the
problem.
> > >
> > > I swapped those values using ncatted and replotted using
> plot_data_plane.
> > > The resulting plot (attached) looks a lot better:
> > >
> > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > > -a delta_lon,global,o,c,"0.234375 degrees" \
> > > -o obs_new.nc \
> > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> > >
> > > met-6.0/bin/plot_data_plane \
> > > obs_new.nc obs_new.ps \
> > > 'name="PWS15"; level="(*,*)";'
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
> > > >
> > > > John, I fixed the problem you found with my observation file.
> > > > After I converted it to a netCDF file, I displayed it (on the
> > > > ftp
> server).
> > > > Note the map does not fit right. I placed the netCDF file on
> > > > the server. The header for the file seems correct:
> > > >
> > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > > dimensions:
> > > > lat = 1152 ;
> > > > lon = 1536 ;
> > > > variables:
> > > > float PWS15(lat, lon) ;
> > > > PWS15:name = "PWS15" ;
> > > > PWS15:init_time_ut = "1497916800" ;
> > > > PWS15:valid_time = "20170620_000000" ;
> > > > PWS15:valid_time_ut = "1497916800" ;
> > > > PWS15:init_time = "20170620_000000" ;
> > > > PWS15:units = "%" ;
> > > > PWS15:level = "Z10" ;
> > > >
> > > > // global attributes:
> > > > :FileOrigins = "Model Data" ;
> > > > :MET_version = "V6.0" ;
> > > > :Projection = "LatLon" ;
> > > > :lat_ll = "-89.921875 degrees_north" ;
> > > > :lon_ll = "0.117187 degrees_east" ;
> > > > :delta_lat = "0.234375 degrees" ;
> > > > :delta_lon = "0.15625 degrees" ;
> > > > :Nlat = "1152 grid_points" ;
> > > > :Nlon = "1536 grid_points" ;
> > > >
> > > > Any idea why the map is wrong - should I worry?
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, June 22, 2017 3:04 PM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > > Missing Value
> > > >
> > > > Bob,
> > > >
> > > > I grabbed the files you posted to our ftp site, but I noticed
> > > > that the timestamp of this file is older:
> > > > GridStatConfig_geps_updated
> > > >
> > > > Can you please make sure that file on the ftp site exactly
> > > > matches the one you're using:
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > >
> > > > I was able to get it to run using met-6.0 but unable to
> > > > replicate your error.
> > > >
> > > > So I want to make sure I'm testing with the same
configuration.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via
RT
> > > > < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: Grid Stat Missing Value
> > > > > Owner: Nobody
> > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > >
> > > > >
> > > > >
> > > > > John, I have been running into an error when I use NCEP 2.5
> > > > > deg climo for the climo reference. Below is the command
line:
> > > > >
> > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > 2017061800/grib2.2017061800.0024
/h/data/global/WXQC/data/met/
> > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > >
> > > > > When I run this I get the following error:
> > > > >
> > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > > created new Met2dDataFile object of type "FileType_NcMet".
> > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > DEBUG 4: lat_ll = 90
> > > > > DEBUG 4: lon_ll = -0
> > > > > DEBUG 4: delta_lat = 2.5
> > > > > DEBUG 4: delta_lon = 2.5
> > > > > DEBUG 4: Nlat = 73
> > > > > DEBUG 4: Nlon = 144
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > object of type "FileType_NcMet".
> > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
verification
> grid.
> > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > > climatology
> > > > > field(s) valid at 20170619_000000.
> > > > > DEBUG 3: Found 1 climatology fields.
> > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > > DEBUG 2: Processing masking regions.
> > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
parse_grid_mask()
> > > > > -> parsing grid mask "FULL"
> > > > > DEBUG 2:
> > > > > DEBUG 2:
> > > > > ------------------------------------------------------------
> > > > > --------------------
> > > > > DEBUG 2:
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > interpolation method NEAREST(1), over region FULL, using
65160
> pairs.
> > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > ERROR :
> > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
> > > > > value
> ...
> > > > > -9999 ERROR :
> > > > >
> > > > > I have sent you all the files involved. When I run the
code
> using a
> > > > > netCDF file which is the same resolution as the FCST which
> > > > > contains the average frequency of the event I don't get this
> > > > > error. If I switch to using the ncep 2.5 deg climo and
remap
> > > > > it to the FCST, I get
> > > > the error.
> > > > > Looking in all the files I don't see any values of -9999.
So
> > > > > I don't know where this is coming from.
> > > > >
> > > > > Thanks
> > > > > Bob Craig
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Mon Jun 26 09:38:47 2017
Bob,
OK, you can find the patch file named "met-
6.0_patches_20170623.tar.gz" in
your directory on the ftp site.
Thanks,
John
On Mon, Jun 26, 2017 at 7:47 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I am glad you found that issue. Please place the fix on the
ftp
> site in my directory since the download site is blocked here.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 5:04 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I was finally able to replicate your error. Turns out that while
> Grid-Stat was running without error, it wasn't actually producing
any
> output. So I wasn't actually exercising the problematic piece of
code.
>
> Yes, I can confirm that the problem is caused by missing data values
in
> the climatology field.
>
> To make the testing even easier, I regridded your climo data to a
very
> small grid so that almost all the climo values would be bad data.
>
> I'm working on updating Grid-Stat to make sure that when the user
> specifies a climatology field, we only compute stats at grid points
where
> the forecast, observation, and climatology all contain valid data.
>
> Unfortunately, this will require a patch and recompiling.
>
> John
>
>
> On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > You should have it now. By the way, the forecast file has 65160
data
> > points but in the MET output listing it says it is using 64620 for
the
> > full region. Could the missing points here be the -9999 values?
> >
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 3:18 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > I still don't have the forecast file you're referencing:
> >
> >
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.201706190
> > 0
> > .0024
> >
> > The one of the ftp site is for the previous day:
> > grib2.2017061800.0024
> >
> > I tried using the previous day's file in Grid-Stat, but it runs
fine
> > (other than a warning about the times not matching). So I'm not
able
> > to replicate your error. I really need to do that before I can
debug it.
> >
> > My guess is that this has to do with the grid definition issues
you've
> > been working through. But it really would be nice to replicate
this
> > so that I can make the code more robust in handling missing data
values.
> >
> > John
> >
> > On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > Just set config file - it didn't seem to go the first time I
sent.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, June 23, 2017 1:52 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > I don't see your input files on the ftp site. Looking in:
> > > incoming/irap/met_help
> > >
> > > The following files do not exist:
> > > grib2.2017061900.0024
> > > GridStatConfig_geps_updated
> > >
> > > Could you please post them to make sure I'm running exactly the
same
> > > case as you?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
> > > >
> > > > John, With your help (thanks for that) I fixed the observation
> > > > file and climo file issues and they now look right (the files
are
> > > > on the server). I also placed the most recent config file
there.
> > > > When I run the following
> > > > command:
> > > >
> > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > 2017061900/grib2.2017061900.0024
/h/data/global/WXQC/data/met/
> > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > >
> > > > I get the same error:
> > > >
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = -89.9219
> > > > DEBUG 4: lon_ll = -0.117187
> > > > DEBUG 4: delta_lat = 0.15625
> > > > DEBUG 4: delta_lon = 0.234375
> > > > DEBUG 4: Nlat = 1152
> > > > DEBUG 4: Nlon = 1536
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_Gb2".
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 3: Use the forecast grid.
> > > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > > > GSL_RNG_TYPE=mt19937
> > > > GSL_RNG_SEED=18446744072977206617
> > > > DEBUG 1: Forecast File: /h/data/global/WXQC/data/geps_
> > > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
> > <(201)%20706-1900>.0024
> > > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > DEBUG 3: Reading forecast data for PWS15/Z10.
> > > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match
for
> > > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
> > <(201)%20706-1900>/
> > > > grib2.2017061900 <(201)%20706-1900>.0024'
> > > > DEBUG 4:
> > > > DEBUG 4: Data plane information:
> > > > DEBUG 4: plane min: 0
> > > > DEBUG 4: plane max: 100
> > > > DEBUG 4: scan mode: 0
> > > > DEBUG 4: valid time: 20170620_000000
> > > > DEBUG 4: lead time: 240000
> > > > DEBUG 4: init time: 20170619_000000
> > > > DEBUG 4: bitmap flag: 255
> > > > DEBUG 4:
> > > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = -90
> > > > DEBUG 4: lon_ll = -0
> > > > DEBUG 4: delta_lat = 2.5
> > > > DEBUG 4: delta_lon = 2.5
> > > > DEBUG 4: Nlat = 73
> > > > DEBUG 4: Nlon = 144
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > > DEBUG 3: For forecast valid at 20170620_000000, found 1
> > > > climatology
> > > > field(s) valid at 20170620_000000.
> > > > DEBUG 3: Found 1 climatology fields.
> > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > DEBUG 2: Processing masking regions.
> > > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
->
> > > > parsing grid mask "FULL"
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > DEBUG 2:
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > > interpolation method.
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
RectangularTemplate
> > > > interpolation method.
> > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
interpolation
> > > > method NEAREST(1), over region FULL, using 64620 pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > ERROR :
> > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value ...
> > > > -9999 ERROR :
> > > >
> > > > Let me know if you get the same error?
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, June 23, 2017 11:10 AM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988]
Grid
> > > > Stat Missing Value
> > > >
> > > > Bob,
> > > >
> > > > Thanks for sending the sample observation data file. I ran it
> > > > through plot_data_plane and see what you mean. The earth
doesn't
> > > > fit the data well. I suspect there's something wrong in your
grid
> > definition.
> > > >
> > > > Let's think about this. The lower left corner is defined as
(lat,
> > > > lon) = (-89.92, 0.117). Let's calculate the upper-right
corner:
> > > >
> > > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > > > (.15625
> > > > *
> > > > 1536) = 240.117
> > > >
> > > > You have an upper-right (lat, lon) = (180.51, 240.117), but
you
> > > > want it to be about (90, 360).
> > > >
> > > > Swapping the delta_lat and delta_lon values results in an
> > > > upper-right corner of (90.078, 360.117). I suspect that's the
> problem.
> > > >
> > > > I swapped those values using ncatted and replotted using
> > plot_data_plane.
> > > > The resulting plot (attached) looks a lot better:
> > > >
> > > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > > > -a delta_lon,global,o,c,"0.234375 degrees" \
> > > > -o obs_new.nc \
> > > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> > > >
> > > > met-6.0/bin/plot_data_plane \
> > > > obs_new.nc obs_new.ps \
> > > > 'name="PWS15"; level="(*,*)";'
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > > > >
> > > > > John, I fixed the problem you found with my observation
file.
> > > > > After I converted it to a netCDF file, I displayed it (on
the
> > > > > ftp
> > server).
> > > > > Note the map does not fit right. I placed the netCDF file
on
> > > > > the server. The header for the file seems correct:
> > > > >
> > > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > > > dimensions:
> > > > > lat = 1152 ;
> > > > > lon = 1536 ;
> > > > > variables:
> > > > > float PWS15(lat, lon) ;
> > > > > PWS15:name = "PWS15" ;
> > > > > PWS15:init_time_ut = "1497916800" ;
> > > > > PWS15:valid_time = "20170620_000000" ;
> > > > > PWS15:valid_time_ut = "1497916800" ;
> > > > > PWS15:init_time = "20170620_000000" ;
> > > > > PWS15:units = "%" ;
> > > > > PWS15:level = "Z10" ;
> > > > >
> > > > > // global attributes:
> > > > > :FileOrigins = "Model Data" ;
> > > > > :MET_version = "V6.0" ;
> > > > > :Projection = "LatLon" ;
> > > > > :lat_ll = "-89.921875 degrees_north" ;
> > > > > :lon_ll = "0.117187 degrees_east" ;
> > > > > :delta_lat = "0.234375 degrees" ;
> > > > > :delta_lon = "0.15625 degrees" ;
> > > > > :Nlat = "1152 grid_points" ;
> > > > > :Nlon = "1536 grid_points" ;
> > > > >
> > > > > Any idea why the map is wrong - should I worry?
> > > > >
> > > > > Thanks
> > > > > Bob
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Thursday, June 22, 2017 3:04 PM
> > > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > > <robert.craig.2 at us.af.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > > > Missing Value
> > > > >
> > > > > Bob,
> > > > >
> > > > > I grabbed the files you posted to our ftp site, but I
noticed
> > > > > that the timestamp of this file is older:
> > > > > GridStatConfig_geps_updated
> > > > >
> > > > > Can you please make sure that file on the ftp site exactly
> > > > > matches the one you're using:
> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > >
> > > > > I was able to get it to run using met-6.0 but unable to
> > > > > replicate your error.
> > > > >
> > > > > So I want to make sure I'm testing with the same
configuration.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil
via RT
> > > > > < met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > > Queue: met_help
> > > > > > Subject: Grid Stat Missing Value
> > > > > > Owner: Nobody
> > > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > > >
> > > > > >
> > > > > >
> > > > > > John, I have been running into an error when I use NCEP
2.5
> > > > > > deg climo for the climo reference. Below is the command
line:
> > > > > >
> > > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > > 2017061800/grib2.2017061800.0024
/h/data/global/WXQC/data/met/
> > > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > > >
> > > > > > When I run this I get the following error:
> > > > > >
> > > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > > > created new Met2dDataFile object of type "FileType_NcMet".
> > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > DEBUG 4: lat_ll = 90
> > > > > > DEBUG 4: lon_ll = -0
> > > > > > DEBUG 4: delta_lat = 2.5
> > > > > > DEBUG 4: delta_lon = 2.5
> > > > > > DEBUG 4: Nlat = 73
> > > > > > DEBUG 4: Nlon = 144
> > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > > object of type "FileType_NcMet".
> > > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
verification
> > grid.
> > > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > > > climatology
> > > > > > field(s) valid at 20170619_000000.
> > > > > > DEBUG 3: Found 1 climatology fields.
> > > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
> PWS15/Z10.
> > > > > > DEBUG 2: Processing masking regions.
> > > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
parse_grid_mask()
> > > > > > -> parsing grid mask "FULL"
> > > > > > DEBUG 2:
> > > > > > DEBUG 2:
> > > > > >
------------------------------------------------------------
> > > > > > --------------------
> > > > > > DEBUG 2:
> > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > RectangularTemplate interpolation method.
> > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > RectangularTemplate interpolation method.
> > > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > > interpolation method NEAREST(1), over region FULL, using
65160
> > pairs.
> > > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > > ERROR :
> > > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
> > > > > > value
> > ...
> > > > > > -9999 ERROR :
> > > > > >
> > > > > > I have sent you all the files involved. When I run the
code
> > using a
> > > > > > netCDF file which is the same resolution as the FCST which
> > > > > > contains the average frequency of the event I don't get
this
> > > > > > error. If I switch to using the ncep 2.5 deg climo and
remap
> > > > > > it to the FCST, I get
> > > > > the error.
> > > > > > Looking in all the files I don't see any values of -9999.
So
> > > > > > I don't know where this is coming from.
> > > > > >
> > > > > > Thanks
> > > > > > Bob Craig
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat Missing Value
From: robert.craig.2 at us.af.mil
Time: Mon Jun 26 13:35:32 2017
John, got it compiled and the error went away.
Thanks for your continued help.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, June 26, 2017 10:39 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
Missing Value
Bob,
OK, you can find the patch file named "met-
6.0_patches_20170623.tar.gz" in your directory on the ftp site.
Thanks,
John
On Mon, Jun 26, 2017 at 7:47 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, I am glad you found that issue. Please place the fix on the
ftp
> site in my directory since the download site is blocked here.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, June 23, 2017 5:04 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> I was finally able to replicate your error. Turns out that while
> Grid-Stat was running without error, it wasn't actually producing
any
> output. So I wasn't actually exercising the problematic piece of
code.
>
> Yes, I can confirm that the problem is caused by missing data values
> in the climatology field.
>
> To make the testing even easier, I regridded your climo data to a
very
> small grid so that almost all the climo values would be bad data.
>
> I'm working on updating Grid-Stat to make sure that when the user
> specifies a climatology field, we only compute stats at grid points
> where the forecast, observation, and climatology all contain valid
data.
>
> Unfortunately, this will require a patch and recompiling.
>
> John
>
>
> On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > You should have it now. By the way, the forecast file has 65160
> > data points but in the MET output listing it says it is using
64620
> > for the full region. Could the missing points here be the -9999
values?
> >
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 3:18 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > I still don't have the forecast file you're referencing:
> >
> >
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061
> > 90
> > 0
> > .0024
> >
> > The one of the ftp site is for the previous day:
> > grib2.2017061800.0024
> >
> > I tried using the previous day's file in Grid-Stat, but it runs
fine
> > (other than a warning about the times not matching). So I'm not
> > able to replicate your error. I really need to do that before I
can debug it.
> >
> > My guess is that this has to do with the grid definition issues
> > you've been working through. But it really would be nice to
> > replicate this so that I can make the code more robust in handling
missing data values.
> >
> > John
> >
> > On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > Just set config file - it didn't seem to go the first time I
sent.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, June 23, 2017 1:52 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
> > > Stat Missing Value
> > >
> > > Bob,
> > >
> > > I don't see your input files on the ftp site. Looking in:
> > > incoming/irap/met_help
> > >
> > > The following files do not exist:
> > > grib2.2017061900.0024
> > > GridStatConfig_geps_updated
> > >
> > > Could you please post them to make sure I'm running exactly the
> > > same case as you?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via
RT
> > > < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
> > > >
> > > > John, With your help (thanks for that) I fixed the observation
> > > > file and climo file issues and they now look right (the files
> > > > are on the server). I also placed the most recent config file
there.
> > > > When I run the following
> > > > command:
> > > >
> > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > 2017061900/grib2.2017061900.0024
/h/data/global/WXQC/data/met/
> > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > >
> > > > I get the same error:
> > > >
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = -89.9219
> > > > DEBUG 4: lon_ll = -0.117187
> > > > DEBUG 4: delta_lat = 0.15625
> > > > DEBUG 4: delta_lon = 0.234375
> > > > DEBUG 4: Nlat = 1152
> > > > DEBUG 4: Nlon = 1536
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_Gb2".
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 3: Use the forecast grid.
> > > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 181
lat_ll:
> > > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > > > GSL_RNG_TYPE=mt19937
> > > > GSL_RNG_SEED=18446744072977206617 DEBUG 1: Forecast File:
> > > > /h/data/global/WXQC/data/geps_
> > > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
> > <(201)%20706-1900>.0024
> > > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > DEBUG 3: Reading forecast data for PWS15/Z10.
> > > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match
for
> > > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
> > <(201)%20706-1900>/
> > > > grib2.2017061900 <(201)%20706-1900>.0024'
> > > > DEBUG 4:
> > > > DEBUG 4: Data plane information:
> > > > DEBUG 4: plane min: 0
> > > > DEBUG 4: plane max: 100
> > > > DEBUG 4: scan mode: 0
> > > > DEBUG 4: valid time: 20170620_000000
> > > > DEBUG 4: lead time: 240000
> > > > DEBUG 4: init time: 20170619_000000
> > > > DEBUG 4: bitmap flag: 255
> > > > DEBUG 4:
> > > > DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll = -90
> > > > DEBUG 4: lon_ll = -0
> > > > DEBUG 4: delta_lat = 2.5
> > > > DEBUG 4: delta_lon = 2.5
> > > > DEBUG 4: Nlat = 73
> > > > DEBUG 4: Nlon = 144
> > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > > object of type "FileType_NcMet".
> > > > DEBUG 1: Regridding climotology PWS15(*,*) to the verification
grid.
> > > > DEBUG 3: For forecast valid at 20170620_000000, found 1
> > > > climatology
> > > > field(s) valid at 20170620_000000.
> > > > DEBUG 3: Found 1 climatology fields.
> > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > DEBUG 2: Processing masking regions.
> > > > DEBUG 3: Processing grid mask: FULL DEBUG 4: parse_grid_mask()
> > > > -> parsing grid mask "FULL"
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > > ------------------------------------------------------------
> > > > --------------------
> > > > DEBUG 2:
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > RectangularTemplate interpolation method.
> > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > interpolation method NEAREST(1), over region FULL, using 64620
pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > ERROR :
> > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value ...
> > > > -9999 ERROR :
> > > >
> > > > Let me know if you get the same error?
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, June 23, 2017 11:10 AM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988]
Grid
> > > > Stat Missing Value
> > > >
> > > > Bob,
> > > >
> > > > Thanks for sending the sample observation data file. I ran it
> > > > through plot_data_plane and see what you mean. The earth
> > > > doesn't fit the data well. I suspect there's something wrong
in
> > > > your grid
> > definition.
> > > >
> > > > Let's think about this. The lower left corner is defined as
> > > > (lat,
> > > > lon) = (-89.92, 0.117). Let's calculate the upper-right
corner:
> > > >
> > > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117 +
> > > > (.15625
> > > > *
> > > > 1536) = 240.117
> > > >
> > > > You have an upper-right (lat, lon) = (180.51, 240.117), but
you
> > > > want it to be about (90, 360).
> > > >
> > > > Swapping the delta_lat and delta_lon values results in an
> > > > upper-right corner of (90.078, 360.117). I suspect that's the
> problem.
> > > >
> > > > I swapped those values using ncatted and replotted using
> > plot_data_plane.
> > > > The resulting plot (attached) looks a lot better:
> > > >
> > > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > > > -a delta_lon,global,o,c,"0.234375 degrees" \
> > > > -o obs_new.nc \
> > > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> > > >
> > > > met-6.0/bin/plot_data_plane \
> > > > obs_new.nc obs_new.ps \
> > > > 'name="PWS15"; level="(*,*)";'
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil via
RT
> > > > < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > >
> > > > >
> > > > > John, I fixed the problem you found with my observation
file.
> > > > > After I converted it to a netCDF file, I displayed it (on
the
> > > > > ftp
> > server).
> > > > > Note the map does not fit right. I placed the netCDF file
on
> > > > > the server. The header for the file seems correct:
> > > > >
> > > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > > > dimensions:
> > > > > lat = 1152 ;
> > > > > lon = 1536 ;
> > > > > variables:
> > > > > float PWS15(lat, lon) ;
> > > > > PWS15:name = "PWS15" ;
> > > > > PWS15:init_time_ut = "1497916800" ;
> > > > > PWS15:valid_time = "20170620_000000" ;
> > > > > PWS15:valid_time_ut = "1497916800" ;
> > > > > PWS15:init_time = "20170620_000000" ;
> > > > > PWS15:units = "%" ;
> > > > > PWS15:level = "Z10" ;
> > > > >
> > > > > // global attributes:
> > > > > :FileOrigins = "Model Data" ;
> > > > > :MET_version = "V6.0" ;
> > > > > :Projection = "LatLon" ;
> > > > > :lat_ll = "-89.921875 degrees_north" ;
> > > > > :lon_ll = "0.117187 degrees_east" ;
> > > > > :delta_lat = "0.234375 degrees" ;
> > > > > :delta_lon = "0.15625 degrees" ;
> > > > > :Nlat = "1152 grid_points" ;
> > > > > :Nlon = "1536 grid_points" ;
> > > > >
> > > > > Any idea why the map is wrong - should I worry?
> > > > >
> > > > > Thanks
> > > > > Bob
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Thursday, June 22, 2017 3:04 PM
> > > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > > <robert.craig.2 at us.af.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
> > > > > Stat Missing Value
> > > > >
> > > > > Bob,
> > > > >
> > > > > I grabbed the files you posted to our ftp site, but I
noticed
> > > > > that the timestamp of this file is older:
> > > > > GridStatConfig_geps_updated
> > > > >
> > > > > Can you please make sure that file on the ftp site exactly
> > > > > matches the one you're using:
> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > >
> > > > > I was able to get it to run using met-6.0 but unable to
> > > > > replicate your error.
> > > > >
> > > > > So I want to make sure I'm testing with the same
configuration.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil
via
> > > > > RT < met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > > Queue: met_help
> > > > > > Subject: Grid Stat Missing Value
> > > > > > Owner: Nobody
> > > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > > >
> > > > > >
> > > > > >
> > > > > > John, I have been running into an error when I use NCEP
2.5
> > > > > > deg climo for the climo reference. Below is the command
line:
> > > > > >
> > > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > > 2017061800/grib2.2017061800.0024
> > > > > > /h/data/global/WXQC/data/met/
> > > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > > >
> > > > > > When I run this I get the following error:
> > > > > >
> > > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > > > created new Met2dDataFile object of type "FileType_NcMet".
> > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > DEBUG 4: lat_ll = 90
> > > > > > DEBUG 4: lon_ll = -0
> > > > > > DEBUG 4: delta_lat = 2.5
> > > > > > DEBUG 4: delta_lon = 2.5
> > > > > > DEBUG 4: Nlat = 73
> > > > > > DEBUG 4: Nlon = 144
> > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
> > > > > > VarInfo object of type "FileType_NcMet".
> > > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
> > > > > > verification
> > grid.
> > > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > > > climatology
> > > > > > field(s) valid at 20170619_000000.
> > > > > > DEBUG 3: Found 1 climatology fields.
> > > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
> PWS15/Z10.
> > > > > > DEBUG 2: Processing masking regions.
> > > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
> > > > > > parse_grid_mask()
> > > > > > -> parsing grid mask "FULL"
> > > > > > DEBUG 2:
> > > > > > DEBUG 2:
> > > > > >
------------------------------------------------------------
> > > > > > --------------------
> > > > > > DEBUG 2:
> > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > RectangularTemplate interpolation method.
> > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > RectangularTemplate interpolation method.
> > > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > > interpolation method NEAREST(1), over region FULL, using
> > > > > > 65160
> > pairs.
> > > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > > ERROR :
> > > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
> > > > > > value
> > ...
> > > > > > -9999 ERROR :
> > > > > >
> > > > > > I have sent you all the files involved. When I run the
code
> > using a
> > > > > > netCDF file which is the same resolution as the FCST which
> > > > > > contains the average frequency of the event I don't get
this
> > > > > > error. If I switch to using the ncep 2.5 deg climo and
> > > > > > remap it to the FCST, I get
> > > > > the error.
> > > > > > Looking in all the files I don't see any values of -9999.
> > > > > > So I don't know where this is coming from.
> > > > > >
> > > > > > Thanks
> > > > > > Bob Craig
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Missing Value
From: John Halley Gotway
Time: Tue Jun 27 13:25:49 2017
Great, thanks for confirming. I'll resolve this ticket.
John
On Mon, Jun 26, 2017 at 1:35 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
>
> John, got it compiled and the error went away.
>
> Thanks for your continued help.
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, June 26, 2017 10:39 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid Stat
> Missing Value
>
> Bob,
>
> OK, you can find the patch file named "met-
6.0_patches_20170623.tar.gz"
> in your directory on the ftp site.
>
> Thanks,
> John
>
> On Mon, Jun 26, 2017 at 7:47 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> >
> > John, I am glad you found that issue. Please place the fix on the
ftp
> > site in my directory since the download site is blocked here.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, June 23, 2017 5:04 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > Missing Value
> >
> > Bob,
> >
> > I was finally able to replicate your error. Turns out that while
> > Grid-Stat was running without error, it wasn't actually producing
any
> > output. So I wasn't actually exercising the problematic piece of
code.
> >
> > Yes, I can confirm that the problem is caused by missing data
values
> > in the climatology field.
> >
> > To make the testing even easier, I regridded your climo data to a
very
> > small grid so that almost all the climo values would be bad data.
> >
> > I'm working on updating Grid-Stat to make sure that when the user
> > specifies a climatology field, we only compute stats at grid
points
> > where the forecast, observation, and climatology all contain valid
data.
> >
> > Unfortunately, this will require a patch and recompiling.
> >
> > John
> >
> >
> > On Fri, Jun 23, 2017 at 2:29 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > >
> > > You should have it now. By the way, the forecast file has 65160
> > > data points but in the MET output listing it says it is using
64620
> > > for the full region. Could the missing points here be the -9999
> values?
> > >
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, June 23, 2017 3:18 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988] Grid
Stat
> > > Missing Value
> > >
> > > Bob,
> > >
> > > I still don't have the forecast file you're referencing:
> > >
> > >
/h/data/global/WXQC/data/geps_verify/qcsave/2017061900/grib2.2017061
> > > 90
> > > 0
> > > .0024
> > >
> > > The one of the ftp site is for the previous day:
> > > grib2.2017061800.0024
> > >
> > > I tried using the previous day's file in Grid-Stat, but it runs
fine
> > > (other than a warning about the times not matching). So I'm not
> > > able to replicate your error. I really need to do that before I
can
> debug it.
> > >
> > > My guess is that this has to do with the grid definition issues
> > > you've been working through. But it really would be nice to
> > > replicate this so that I can make the code more robust in
handling
> missing data values.
> > >
> > > John
> > >
> > > On Fri, Jun 23, 2017 at 1:25 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
>
> > > >
> > > > Just set config file - it didn't seem to go the first time I
sent.
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, June 23, 2017 1:52 PM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988]
Grid
> > > > Stat Missing Value
> > > >
> > > > Bob,
> > > >
> > > > I don't see your input files on the ftp site. Looking in:
> > > > incoming/irap/met_help
> > > >
> > > > The following files do not exist:
> > > > grib2.2017061900.0024
> > > > GridStatConfig_geps_updated
> > > >
> > > > Could you please post them to make sure I'm running exactly
the
> > > > same case as you?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Fri, Jun 23, 2017 at 12:37 PM, robert.craig.2 at us.af.mil via
RT
> > > > < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988 >
> > > > >
> > > > > John, With your help (thanks for that) I fixed the
observation
> > > > > file and climo file issues and they now look right (the
files
> > > > > are on the server). I also placed the most recent config
file
> there.
> > > > > When I run the following
> > > > > command:
> > > > >
> > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > 2017061900/grib2.2017061900.0024
/h/data/global/WXQC/data/met/
> > > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > >
> > > > > I get the same error:
> > > > >
> > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > DEBUG 4: lat_ll = -89.9219
> > > > > DEBUG 4: lon_ll = -0.117187
> > > > > DEBUG 4: delta_lat = 0.15625
> > > > > DEBUG 4: delta_lon = 0.234375
> > > > > DEBUG 4: Nlat = 1152
> > > > > DEBUG 4: Nlon = 1536
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > object of type "FileType_Gb2".
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > object of type "FileType_NcMet".
> > > > > DEBUG 3: Use the forecast grid.
> > > > > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny:
181
> lat_ll:
> > > > > -90.000 lon_ll: -0.000 delta_lat: 1.000 delta_lon: 1.000
> > > > > GSL_RNG_TYPE=mt19937
> > > > > GSL_RNG_SEED=18446744072977206617 DEBUG 1: Forecast File:
> > > > > /h/data/global/WXQC/data/geps_
> > > > > verify/qcsave/2017061900 <(201)%20706-1900>/grib2.2017061900
> > > <(201)%20706-1900>.0024
> > > > > DEBUG 1: Observation File: /h/data/global/WXQC/data/met/
> > > > > tmp/68bf9465-1763-4eb3-a42f-413092931b60
> > > > > DEBUG 3: Reading forecast data for PWS15/Z10.
> > > > > DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match
for
> > > > > 'PWS15/Z10' in GRIB2 record 242 field 1 of GRIB2 file
> > > > > '/h/data/global/WXQC/data/geps_verify/qcsave/2017061900
> > > <(201)%20706-1900>/
> > > > > grib2.2017061900 <(201)%20706-1900>.0024'
> > > > > DEBUG 4:
> > > > > DEBUG 4: Data plane information:
> > > > > DEBUG 4: plane min: 0
> > > > > DEBUG 4: plane max: 100
> > > > > DEBUG 4: scan mode: 0
> > > > > DEBUG 4: valid time: 20170620_000000
> > > > > DEBUG 4: lead time: 240000
> > > > > DEBUG 4: init time: 20170619_000000
> > > > > DEBUG 4: bitmap flag: 255
> > > > > DEBUG 4:
> > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > DEBUG 1: Regridding forecast PWS15(*,*) to the verification
grid.
> > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > DEBUG 4: lat_ll = -90
> > > > > DEBUG 4: lon_ll = -0
> > > > > DEBUG 4: delta_lat = 2.5
> > > > > DEBUG 4: delta_lon = 2.5
> > > > > DEBUG 4: Nlat = 73
> > > > > DEBUG 4: Nlon = 144
> > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> > > > > object of type "FileType_NcMet".
> > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
verification
> grid.
> > > > > DEBUG 3: For forecast valid at 20170620_000000, found 1
> > > > > climatology
> > > > > field(s) valid at 20170620_000000.
> > > > > DEBUG 3: Found 1 climatology fields.
> > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
PWS15/Z10.
> > > > > DEBUG 2: Processing masking regions.
> > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
parse_grid_mask()
> > > > > -> parsing grid mask "FULL"
> > > > > DEBUG 2:
> > > > > DEBUG 2:
> > > > > ------------------------------------------------------------
> > > > > --------------------
> > > > > DEBUG 2:
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > RectangularTemplate interpolation method.
> > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > interpolation method NEAREST(1), over region FULL, using
64620
> pairs.
> > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > ERROR :
> > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) -> bad
value
> ...
> > > > > -9999 ERROR :
> > > > >
> > > > > Let me know if you get the same error?
> > > > >
> > > > > Thanks
> > > > > Bob
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, June 23, 2017 11:10 AM
> > > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > > <robert.craig.2 at us.af.mil>
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988]
Grid
> > > > > Stat Missing Value
> > > > >
> > > > > Bob,
> > > > >
> > > > > Thanks for sending the sample observation data file. I ran
it
> > > > > through plot_data_plane and see what you mean. The earth
> > > > > doesn't fit the data well. I suspect there's something
wrong in
> > > > > your grid
> > > definition.
> > > > >
> > > > > Let's think about this. The lower left corner is defined as
> > > > > (lat,
> > > > > lon) = (-89.92, 0.117). Let's calculate the upper-right
corner:
> > > > >
> > > > > UR Lat = -89.92 + (0.234375 * 1152) = 180.51 UR Lon = 0.117
+
> > > > > (.15625
> > > > > *
> > > > > 1536) = 240.117
> > > > >
> > > > > You have an upper-right (lat, lon) = (180.51, 240.117), but
you
> > > > > want it to be about (90, 360).
> > > > >
> > > > > Swapping the delta_lat and delta_lon values results in an
> > > > > upper-right corner of (90.078, 360.117). I suspect that's
the
> > problem.
> > > > >
> > > > > I swapped those values using ncatted and replotted using
> > > plot_data_plane.
> > > > > The resulting plot (attached) looks a lot better:
> > > > >
> > > > > ncatted -a delta_lat,global,o,c,"0.15625 degrees" \
> > > > > -a delta_lon,global,o,c,"0.234375 degrees" \
> > > > > -o obs_new.nc \
> > > > > 1d9c10f0-3c5a-4197-aaee-fb8b04b2a943
> > > > >
> > > > > met-6.0/bin/plot_data_plane \
> > > > > obs_new.nc obs_new.ps \
> > > > > 'name="PWS15"; level="(*,*)";'
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Fri, Jun 23, 2017 at 9:14 AM, robert.craig.2 at us.af.mil
via RT
> > > > > < met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > > >
> > > > > >
> > > > > > John, I fixed the problem you found with my observation
file.
> > > > > > After I converted it to a netCDF file, I displayed it (on
the
> > > > > > ftp
> > > server).
> > > > > > Note the map does not fit right. I placed the netCDF file
on
> > > > > > the server. The header for the file seems correct:
> > > > > >
> > > > > > netcdf \1d9c10f0-3c5a-4197-aaee-fb8b04b2a943 {
> > > > > > dimensions:
> > > > > > lat = 1152 ;
> > > > > > lon = 1536 ;
> > > > > > variables:
> > > > > > float PWS15(lat, lon) ;
> > > > > > PWS15:name = "PWS15" ;
> > > > > > PWS15:init_time_ut = "1497916800" ;
> > > > > > PWS15:valid_time = "20170620_000000" ;
> > > > > > PWS15:valid_time_ut = "1497916800" ;
> > > > > > PWS15:init_time = "20170620_000000" ;
> > > > > > PWS15:units = "%" ;
> > > > > > PWS15:level = "Z10" ;
> > > > > >
> > > > > > // global attributes:
> > > > > > :FileOrigins = "Model Data" ;
> > > > > > :MET_version = "V6.0" ;
> > > > > > :Projection = "LatLon" ;
> > > > > > :lat_ll = "-89.921875 degrees_north" ;
> > > > > > :lon_ll = "0.117187 degrees_east" ;
> > > > > > :delta_lat = "0.234375 degrees" ;
> > > > > > :delta_lon = "0.15625 degrees" ;
> > > > > > :Nlat = "1152 grid_points" ;
> > > > > > :Nlon = "1536 grid_points" ;
> > > > > >
> > > > > > Any idea why the map is wrong - should I worry?
> > > > > >
> > > > > > Thanks
> > > > > > Bob
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Thursday, June 22, 2017 3:04 PM
> > > > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > > > <robert.craig.2 at us.af.mil>
> > > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80988]
Grid
> > > > > > Stat Missing Value
> > > > > >
> > > > > > Bob,
> > > > > >
> > > > > > I grabbed the files you posted to our ftp site, but I
noticed
> > > > > > that the timestamp of this file is older:
> > > > > > GridStatConfig_geps_updated
> > > > > >
> > > > > > Can you please make sure that file on the ftp site exactly
> > > > > > matches the one you're using:
> > > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > >
> > > > > > I was able to get it to run using met-6.0 but unable to
> > > > > > replicate your error.
> > > > > >
> > > > > > So I want to make sure I'm testing with the same
configuration.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 1:25 PM, robert.craig.2 at us.af.mil
via
> > > > > > RT < met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Jun 22 13:25:27 2017: Request 80988 was acted upon.
> > > > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > > > Queue: met_help
> > > > > > > Subject: Grid Stat Missing Value
> > > > > > > Owner: Nobody
> > > > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80988
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > John, I have been running into an error when I use NCEP
2.5
> > > > > > > deg climo for the climo reference. Below is the command
line:
> > > > > > >
> > > > > > > /h/WXQC/met-6.0/bin/grid_stat
> > > > > > > /h/data/global/WXQC/data/geps_verify/qcsave/
> > > > > > > 2017061800/grib2.2017061800.0024
> > > > > > > /h/data/global/WXQC/data/met/
> > > > > > > tmp/a1262a1d-4201-48b4-bc5b-5cbf84af32bb
> > > > > > > /h/WXQC/met-6.0/data/config/GridStatConfig_geps_updated
> > > > > > > -outdir /h/data/global/WXQC/data/met/gdstat/geps -v 4
> > > > > > >
> > > > > > > When I run this I get the following error:
> > > > > > >
> > > > > > > DEBUG 3: Rescaling probabilistic field from [0,100] to
[0,1].
> > > > > > > DEBUG 3: Reading observation data for PWS15(*,*).
> > > > > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > > > > created new Met2dDataFile object of type
"FileType_NcMet".
> > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > DEBUG 4: lat_ll = 90
> > > > > > > DEBUG 4: lon_ll = -0
> > > > > > > DEBUG 4: delta_lat = 2.5
> > > > > > > DEBUG 4: delta_lon = 2.5
> > > > > > > DEBUG 4: Nlat = 73
> > > > > > > DEBUG 4: Nlon = 144
> > > > > > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
> > > > > > > VarInfo object of type "FileType_NcMet".
> > > > > > > DEBUG 1: Regridding climotology PWS15(*,*) to the
> > > > > > > verification
> > > grid.
> > > > > > > DEBUG 3: For forecast valid at 20170619_000000, found 1
> > > > > > > climatology
> > > > > > > field(s) valid at 20170619_000000.
> > > > > > > DEBUG 3: Found 1 climatology fields.
> > > > > > > DEBUG 3: Found 1 climatology mean field(s) for forecast
> > PWS15/Z10.
> > > > > > > DEBUG 2: Processing masking regions.
> > > > > > > DEBUG 3: Processing grid mask: FULL DEBUG 4:
> > > > > > > parse_grid_mask()
> > > > > > > -> parsing grid mask "FULL"
> > > > > > > DEBUG 2:
> > > > > > > DEBUG 2:
> > > > > > >
------------------------------------------------------------
> > > > > > > --------------------
> > > > > > > DEBUG 2:
> > > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > > RectangularTemplate interpolation method.
> > > > > > > DEBUG 3: Smoothing field using the NEAREST(1)
> > > > > > > RectangularTemplate interpolation method.
> > > > > > > DEBUG 2: Processing PWS15/Z10 versus PWS15(*,*), for
> > > > > > > interpolation method NEAREST(1), over region FULL, using
> > > > > > > 65160
> > > pairs.
> > > > > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > > > > ERROR :
> > > > > > > ERROR : Nx2ContingencyTable::inc_nonevent(double) ->
bad
> > > > > > > value
> > > ...
> > > > > > > -9999 ERROR :
> > > > > > >
> > > > > > > I have sent you all the files involved. When I run the
code
> > > using a
> > > > > > > netCDF file which is the same resolution as the FCST
which
> > > > > > > contains the average frequency of the event I don't get
this
> > > > > > > error. If I switch to using the ncep 2.5 deg climo and
> > > > > > > remap it to the FCST, I get
> > > > > > the error.
> > > > > > > Looking in all the files I don't see any values of
-9999.
> > > > > > > So I don't know where this is coming from.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bob Craig
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list