[Met_help] [rt.rap.ucar.edu #69380] History for point_stat issue
John Halley Gotway via RT
met_help at ucar.edu
Tue Oct 21 09:28:03 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
Here's my newest and latest headache using MET v4.1.
I'm trying to verify a gridded field using point_stat. The observations are formatted as NetCDF. MET appears to read in everything without a problem but, as far as I can tell, MET throws all the observations out so there's nothing left. Here's what I get:
=================
/home/qcteam/METv4.1/bin/point_stat /home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg -log ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats -v 4
DEBUG 1: Default Config File: /home/qcteam/METv4.1/data/config/PointStatConfig_default
DEBUG 1: User Config File: /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Gb1".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Gb1".
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=1
DEBUG 1: Forecast File: /home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib
DEBUG 1: Climatology File: none
DEBUG 1: Observation File: /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for U-GWD/L0.
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file "/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB records matching VarInfo "U-GWD/L0" in GRIB file "/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology levels.
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Searching 64800 observations from 64800 messages.
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type ADPSFC, over region FULL, for interpolation method BILIN(4), using 0 pairs.
DEBUG 3: Number of matched pairs = 0
DEBUG 3: Observations processed = 64800
DEBUG 3: Rejected: GRIB code = 0
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 43229
DEBUG 3: Rejected: off the grid = 21571
DEBUG 3: Rejected: level mismatch = 0
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 0
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V.stat
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_fho.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_ctc.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_cts.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mctc.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mcts.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_cnt.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_sl1l2.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_sal1l2.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_vl1l2.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_val1l2.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pct.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pstd.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pjc.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_prc.txt
DEBUG 1: Output file: /home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mpr.txt
============
What I'm trying to figure out is what 'off the grid' signifies. The observation data cover the entire global domain, as does the model. There shouldn't be any 'off the grid' data. Here's the model data information, courtesy degrib.
rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200 kpds7=0 levels=(0,0) grid=255 atmos col anl:
AODVIS=Aerosol optical depth (0.55 um) [numeric]
timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0 num_in_ave 0 missing 0
center 57 subcenter 6 process 15 Table 129 scan: WE:NS winds(grid)
latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan 0 mode 136 bdsgrid 1
min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945 DecScale 6 BinScale 0
Is it possible that MET is upset by the definition of longitude as running from 0 to -0.5 in positive steps of 1.003, which it will never reach? I'm not sure how MET establishes the model grid; I presume through the GriB header.
I would appreciate any thoughts you have on this. I can FTP the files as before if that will help.
Thanks,
Matt
// SIGNED //
Matthew C. Sittel
Associate Scientist
University Corporation for Atmospheric Research
16WS/WXN, Offutt AFB, NE
(402) 294-3473 DSN: 271-3473
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Wed Oct 15 10:38:52 2014
Matt,
It's possible that this issue is caused by the fact that it's a global
grid. Please try the following...
(1) Run plot_data_plane to make sure MET is reading the gridded data
the
way you expect:
plot_data_plane model.grb aodvis.ps 'name="AODVIS"; level="L0";'
And then look at that image (adovis.ps) to make sure it looks
good.
(2) Run plot_point_obs twice to see where MET is plotting the
observations:
plot_point_obs obs.nc obs.ps
plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
The first call plots all points on a default global grid. The
second
call plots all points, but using the grid it reads from model.grb
file.
Just look at the resulting images and see if anything jumps out at
you.
Perhaps you accidentally transposed the lat/lon values in the NetCDF
point
observation file? Or perhaps it's something to do with the range of
longitude values you used? Or something else I'm not thinking of.
If the problem doesn't become obvious after those steps, please send
me a
sample forecast grib file, NetCDF observation file, and Point-Stat
config
file.
Thanks,
John Halley Gotway
met_help at ucar.edu
On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> Queue: met_help
> Subject: point_stat issue
> Owner: Nobody
> Requestors: matthew.sittel.ctr at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
>
> Hi,
>
> Here's my newest and latest headache using MET v4.1.
>
> I'm trying to verify a gridded field using point_stat. The
observations
> are formatted as NetCDF. MET appears to read in everything without
a
> problem but, as far as I can tell, MET throws all the observations
out so
> there's nothing left. Here's what I get:
>
> =================
> /home/qcteam/METv4.1/bin/point_stat
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg -log
> ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats -v
4
> DEBUG 1: Default Config File:
> /home/qcteam/METv4.1/data/config/PointStatConfig_default
> DEBUG 1: User Config File:
> /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Gb1".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Gb1".
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=1
> DEBUG 1: Forecast File:
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for U-GWD/L0.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for
> VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> matching VarInfo "U-GWD/L0" in GRIB file
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Searching 64800 observations from 64800 messages.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type
ADPSFC,
> over region FULL, for interpolation method BILIN(4), using 0 pairs.
> DEBUG 3: Number of matched pairs = 0
> DEBUG 3: Observations processed = 64800
> DEBUG 3: Rejected: GRIB code = 0
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 43229
> DEBUG 3: Rejected: off the grid = 21571
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V.stat
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_fho.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_ctc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_cts.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mctc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mcts.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_cnt.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_sl1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_sal1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_vl1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_val1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pct.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pstd.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_pjc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_prc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000V_mpr.txt
> ============
>
> What I'm trying to figure out is what 'off the grid' signifies. The
> observation data cover the entire global domain, as does the model.
There
> shouldn't be any 'off the grid' data. Here's the model data
information,
> courtesy degrib.
>
> rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200 kpds7=0
> levels=(0,0) grid=255 atmos col anl:
> AODVIS=Aerosol optical depth (0.55 um) [numeric]
> timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0 num_in_ave
0
> missing 0
> center 57 subcenter 6 process 15 Table 129 scan: WE:NS winds(grid)
> latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan 0
mode
> 136 bdsgrid 1
> min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945 DecScale
6
> BinScale 0
>
> Is it possible that MET is upset by the definition of longitude as
running
> from 0 to -0.5 in positive steps of 1.003, which it will never
reach? I'm
> not sure how MET establishes the model grid; I presume through the
GriB
> header.
>
> I would appreciate any thoughts you have on this. I can FTP the
files as
> before if that will help.
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> Associate Scientist
> University Corporation for Atmospheric Research
> 16WS/WXN, Offutt AFB, NE
> (402) 294-3473 DSN: 271-3473
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Wed Oct 15 11:19:46 2014
Well the first two plots look just fine, but the third won't work:
DEBUG 2: Retrieving grid from file:
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib
DEBUG 1: Opening netCDF file:
/home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
DEBUG 2: Processing 64800 observations at 64800 locations.
DEBUG 2: Observation GRIB codes: ALL
DEBUG 2: Observation message types: ALL
DEBUG 1: Creating postscript file: obs_data_file.ps
DEBUG 2: Finished plotting 0 locations.
DEBUG 2: Skipped 64800 locations off the grid.
Since the second file looks fine, that implies the NetCDF is formatted
right. The GriB data seems okay too. But the pair together are not.
That's what leads me to question if the GriB header in the model GriB
file is a problem. Does MET take its dimensions literally? Does it
try to go from 0.0 to -0.5 in steps of 1.003 and never get there
because the end longitude is less than the start?
It's probably time to FTP the files. I'll push model and observation
from the above case along with config. I'll let you know when they're
done... I suspect it will take a couple hours given the FTP speed last
time we did this.
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 15, 2014 11:39 AM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
It's possible that this issue is caused by the fact that it's a global
grid. Please try the following...
(1) Run plot_data_plane to make sure MET is reading the gridded data
the way you expect:
plot_data_plane model.grb aodvis.ps 'name="AODVIS"; level="L0";'
And then look at that image (adovis.ps) to make sure it looks
good.
(2) Run plot_point_obs twice to see where MET is plotting the
observations:
plot_point_obs obs.nc obs.ps
plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
The first call plots all points on a default global grid. The
second call plots all points, but using the grid it reads from
model.grb file.
Just look at the resulting images and see if anything jumps out at
you.
Perhaps you accidentally transposed the lat/lon values in the NetCDF
point observation file? Or perhaps it's something to do with the
range of longitude values you used? Or something else I'm not
thinking of.
If the problem doesn't become obvious after those steps, please send
me a sample forecast grib file, NetCDF observation file, and Point-
Stat config file.
Thanks,
John Halley Gotway
met_help at ucar.edu
On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> Queue: met_help
> Subject: point_stat issue
> Owner: Nobody
> Requestors: matthew.sittel.ctr at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >
>
>
> Hi,
>
> Here's my newest and latest headache using MET v4.1.
>
> I'm trying to verify a gridded field using point_stat. The
> observations are formatted as NetCDF. MET appears to read in
> everything without a problem but, as far as I can tell, MET throws
all
> the observations out so there's nothing left. Here's what I get:
>
> =================
> /home/qcteam/METv4.1/bin/point_stat
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> rib /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg -log
> ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats -v
4
> DEBUG 1: Default Config File:
> /home/qcteam/METv4.1/data/config/PointStatConfig_default
> DEBUG 1: User Config File:
> /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Gb1".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_Gb1".
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=1
> DEBUG 1: Forecast File:
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> rib
> DEBUG 1: Climatology File: none
> DEBUG 1: Observation File:
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> DEBUG 2:
> DEBUG 2:
>
----------------------------------------------------------------------
> ----------
> DEBUG 2:
> DEBUG 2: Reading data for U-GWD/L0.
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for
> VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> matching VarInfo "U-GWD/L0" in GRIB file
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> DEBUG 2:
> DEBUG 2:
>
----------------------------------------------------------------------
> ----------
> DEBUG 2:
> DEBUG 2: Searching 64800 observations from 64800 messages.
> DEBUG 2:
> DEBUG 2:
>
----------------------------------------------------------------------
> ----------
> DEBUG 2:
> DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
pairs.
> DEBUG 3: Number of matched pairs = 0
> DEBUG 3: Observations processed = 64800
> DEBUG 3: Rejected: GRIB code = 0
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 43229
> DEBUG 3: Rejected: off the grid = 21571
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2:
> DEBUG 2:
>
----------------------------------------------------------------------
> ----------
> DEBUG 2:
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V.stat
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_fho.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_ctc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_cts.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_mctc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_mcts.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_cnt.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_sl1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_sal1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_vl1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_val1l2.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_pct.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_pstd.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_pjc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_prc.txt
> DEBUG 1: Output file:
>
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> V_mpr.txt
> ============
>
> What I'm trying to figure out is what 'off the grid' signifies. The
> observation data cover the entire global domain, as does the model.
> There shouldn't be any 'off the grid' data. Here's the model data
> information, courtesy degrib.
>
> rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200 kpds7=0
> levels=(0,0) grid=255 atmos col anl:
> AODVIS=Aerosol optical depth (0.55 um) [numeric]
> timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0 num_in_ave
0
> missing 0
> center 57 subcenter 6 process 15 Table 129 scan: WE:NS winds(grid)
> latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan 0
> mode
> 136 bdsgrid 1
> min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945 DecScale
6
> BinScale 0
>
> Is it possible that MET is upset by the definition of longitude as
> running from 0 to -0.5 in positive steps of 1.003, which it will
never
> reach? I'm not sure how MET establishes the model grid; I presume
> through the GriB header.
>
> I would appreciate any thoughts you have on this. I can FTP the
files
> as before if that will help.
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> Associate Scientist
> University Corporation for Atmospheric Research 16WS/WXN, Offutt
AFB,
> NE
> (402) 294-3473 DSN: 271-3473
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Wed Oct 15 11:28:27 2014
Matt,
OK, sounds good. Just let me know when the transfer is complete and
I'll
go grab the data.
Thanks,
John
On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Well the first two plots look just fine, but the third won't work:
>
> DEBUG 2: Retrieving grid from file:
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib
> DEBUG 1: Opening netCDF file:
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> DEBUG 2: Processing 64800 observations at 64800 locations.
> DEBUG 2: Observation GRIB codes: ALL
> DEBUG 2: Observation message types: ALL
> DEBUG 1: Creating postscript file: obs_data_file.ps
> DEBUG 2: Finished plotting 0 locations.
> DEBUG 2: Skipped 64800 locations off the grid.
>
> Since the second file looks fine, that implies the NetCDF is
formatted
> right. The GriB data seems okay too. But the pair together are
not.
> That's what leads me to question if the GriB header in the model
GriB file
> is a problem. Does MET take its dimensions literally? Does it try
to go
> from 0.0 to -0.5 in steps of 1.003 and never get there because the
end
> longitude is less than the start?
>
> It's probably time to FTP the files. I'll push model and
observation from
> the above case along with config. I'll let you know when they're
done... I
> suspect it will take a couple hours given the FTP speed last time we
did
> this.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> It's possible that this issue is caused by the fact that it's a
global
> grid. Please try the following...
>
> (1) Run plot_data_plane to make sure MET is reading the gridded data
the
> way you expect:
> plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> And then look at that image (adovis.ps) to make sure it looks
good.
>
> (2) Run plot_point_obs twice to see where MET is plotting the
observations:
> plot_point_obs obs.nc obs.ps
> plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
> The first call plots all points on a default global grid. The
second
> call plots all points, but using the grid it reads from model.grb
file.
>
> Just look at the resulting images and see if anything jumps out at
you.
> Perhaps you accidentally transposed the lat/lon values in the NetCDF
point
> observation file? Or perhaps it's something to do with the range of
> longitude values you used? Or something else I'm not thinking of.
>
> If the problem doesn't become obvious after those steps, please send
me a
> sample forecast grib file, NetCDF observation file, and Point-Stat
config
> file.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: point_stat issue
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >
> >
> >
> > Hi,
> >
> > Here's my newest and latest headache using MET v4.1.
> >
> > I'm trying to verify a gridded field using point_stat. The
> > observations are formatted as NetCDF. MET appears to read in
> > everything without a problem but, as far as I can tell, MET throws
all
> > the observations out so there's nothing left. Here's what I get:
> >
> > =================
> > /home/qcteam/METv4.1/bin/point_stat
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> > rib /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
-log
> > ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats
-v 4
> > DEBUG 1: Default Config File:
> > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > DEBUG 1: User Config File:
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Gb1".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_Gb1".
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=1
> > DEBUG 1: Forecast File:
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> > rib
> > DEBUG 1: Climatology File: none
> > DEBUG 1: Observation File:
> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > DEBUG 2:
> > DEBUG 2:
> >
----------------------------------------------------------------------
> > ----------
> > DEBUG 2:
> > DEBUG 2: Reading data for U-GWD/L0.
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
for
> > VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> > matching VarInfo "U-GWD/L0" in GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> > DEBUG 2:
> > DEBUG 2:
> >
----------------------------------------------------------------------
> > ----------
> > DEBUG 2:
> > DEBUG 2: Searching 64800 observations from 64800 messages.
> > DEBUG 2:
> > DEBUG 2:
> >
----------------------------------------------------------------------
> > ----------
> > DEBUG 2:
> > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> pairs.
> > DEBUG 3: Number of matched pairs = 0
> > DEBUG 3: Observations processed = 64800
> > DEBUG 3: Rejected: GRIB code = 0
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 43229
> > DEBUG 3: Rejected: off the grid = 21571
> > DEBUG 3: Rejected: level mismatch = 0
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2:
> > DEBUG 2:
> >
----------------------------------------------------------------------
> > ----------
> > DEBUG 2:
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V.stat
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_fho.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_ctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_cts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_mctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_mcts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_cnt.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_sl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_sal1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_vl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_val1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_pct.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_pstd.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_pjc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_prc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_000000
> > V_mpr.txt
> > ============
> >
> > What I'm trying to figure out is what 'off the grid' signifies.
The
> > observation data cover the entire global domain, as does the
model.
> > There shouldn't be any 'off the grid' data. Here's the model data
> > information, courtesy degrib.
> >
> > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
kpds7=0
> > levels=(0,0) grid=255 atmos col anl:
> > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
num_in_ave 0
> > missing 0
> > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> > long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan
0
> > mode
> > 136 bdsgrid 1
> > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
DecScale 6
> > BinScale 0
> >
> > Is it possible that MET is upset by the definition of longitude as
> > running from 0 to -0.5 in positive steps of 1.003, which it will
never
> > reach? I'm not sure how MET establishes the model grid; I presume
> > through the GriB header.
> >
> > I would appreciate any thoughts you have on this. I can FTP the
files
> > as before if that will help.
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > Associate Scientist
> > University Corporation for Atmospheric Research 16WS/WXN, Offutt
AFB,
> > NE
> > (402) 294-3473 DSN: 271-3473
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Wed Oct 15 12:39:01 2014
No sooner did I send that... the model file has finished transferring
at 1210 MT.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 15, 2014 12:28 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
OK, sounds good. Just let me know when the transfer is complete and
I'll go grab the data.
Thanks,
John
On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Well the first two plots look just fine, but the third won't work:
>
> DEBUG 2: Retrieving grid from file:
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> rib
> DEBUG 1: Opening netCDF file:
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> DEBUG 2: Processing 64800 observations at 64800 locations.
> DEBUG 2: Observation GRIB codes: ALL
> DEBUG 2: Observation message types: ALL DEBUG 1: Creating postscript
> file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
> DEBUG 2: Skipped 64800 locations off the grid.
>
> Since the second file looks fine, that implies the NetCDF is
formatted
> right. The GriB data seems okay too. But the pair together are
not.
> That's what leads me to question if the GriB header in the model
GriB
> file is a problem. Does MET take its dimensions literally? Does it
> try to go from 0.0 to -0.5 in steps of 1.003 and never get there
> because the end longitude is less than the start?
>
> It's probably time to FTP the files. I'll push model and
observation
> from the above case along with config. I'll let you know when
they're
> done... I suspect it will take a couple hours given the FTP speed
last
> time we did this.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> It's possible that this issue is caused by the fact that it's a
global
> grid. Please try the following...
>
> (1) Run plot_data_plane to make sure MET is reading the gridded data
> the way you expect:
> plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> And then look at that image (adovis.ps) to make sure it looks
good.
>
> (2) Run plot_point_obs twice to see where MET is plotting the
observations:
> plot_point_obs obs.nc obs.ps
> plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
> The first call plots all points on a default global grid. The
> second call plots all points, but using the grid it reads from
model.grb file.
>
> Just look at the resulting images and see if anything jumps out at
you.
> Perhaps you accidentally transposed the lat/lon values in the NetCDF
> point observation file? Or perhaps it's something to do with the
> range of longitude values you used? Or something else I'm not
thinking of.
>
> If the problem doesn't become obvious after those steps, please send
> me a sample forecast grib file, NetCDF observation file, and
> Point-Stat config file.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: point_stat issue
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >
> >
> >
> > Hi,
> >
> > Here's my newest and latest headache using MET v4.1.
> >
> > I'm trying to verify a gridded field using point_stat. The
> > observations are formatted as NetCDF. MET appears to read in
> > everything without a problem but, as far as I can tell, MET throws
> > all the observations out so there's nothing left. Here's what I
get:
> >
> > =================
> > /home/qcteam/METv4.1/bin/point_stat
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > .g rib /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
-log
> > ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats
-v
> > 4 DEBUG 1: Default Config File:
> > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > DEBUG 1: User Config File:
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Gb1".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_Gb1".
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=1
> > DEBUG 1: Forecast File:
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > .g
> > rib
> > DEBUG 1: Climatology File: none
> > DEBUG 1: Observation File:
> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Reading data for U-GWD/L0.
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
> > for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
> > records matching VarInfo "U-GWD/L0" in GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Searching 64800 observations from 64800 messages.
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> pairs.
> > DEBUG 3: Number of matched pairs = 0
> > DEBUG 3: Observations processed = 64800
> > DEBUG 3: Rejected: GRIB code = 0
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 43229
> > DEBUG 3: Rejected: off the grid = 21571
> > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected: quality
> > marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad fcst
> > value = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V.stat
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_fho.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_ctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_cts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mcts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_cnt.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_sl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_sal1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_vl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_val1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pct.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pstd.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pjc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_prc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mpr.txt
> > ============
> >
> > What I'm trying to figure out is what 'off the grid' signifies.
The
> > observation data cover the entire global domain, as does the
model.
> > There shouldn't be any 'off the grid' data. Here's the model data
> > information, courtesy degrib.
> >
> > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
kpds7=0
> > levels=(0,0) grid=255 atmos col anl:
> > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
num_in_ave
> > 0 missing 0
> > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> > long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan
0
> > mode
> > 136 bdsgrid 1
> > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
DecScale
> > 6 BinScale 0
> >
> > Is it possible that MET is upset by the definition of longitude as
> > running from 0 to -0.5 in positive steps of 1.003, which it will
> > never reach? I'm not sure how MET establishes the model grid; I
> > presume through the GriB header.
> >
> > I would appreciate any thoughts you have on this. I can FTP the
> > files as before if that will help.
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > Associate Scientist
> > University Corporation for Atmospheric Research 16WS/WXN, Offutt
> > AFB, NE
> > (402) 294-3473 DSN: 271-3473
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Wed Oct 15 12:39:01 2014
While we wait... here's a curiosity that might be noteworthy, or
not...
When I plot the model GriB data, specifically the field kpds5=147
(AODVIS) using plot_data_plane I get the PDF output I've attached.
When the model developer visualizes the data, he swears that the GIF
I've attached is the correct layout... a version seemingly rotated 180
degrees along the Equatorial axis.
I'm not sure what to believe, but I'm wondering now if plot_data_plane
(or MET for that matter) assumes anything regarding the north-south
orientation of the data. I can't imagine that it does... but I
thought I'd mention it just the same.
The observation and config file have transferred... just waiting on
the ~370MB model file to finish.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 15, 2014 12:28 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
OK, sounds good. Just let me know when the transfer is complete and
I'll go grab the data.
Thanks,
John
On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Well the first two plots look just fine, but the third won't work:
>
> DEBUG 2: Retrieving grid from file:
>
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> rib
> DEBUG 1: Opening netCDF file:
> /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> DEBUG 2: Processing 64800 observations at 64800 locations.
> DEBUG 2: Observation GRIB codes: ALL
> DEBUG 2: Observation message types: ALL DEBUG 1: Creating postscript
> file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
> DEBUG 2: Skipped 64800 locations off the grid.
>
> Since the second file looks fine, that implies the NetCDF is
formatted
> right. The GriB data seems okay too. But the pair together are
not.
> That's what leads me to question if the GriB header in the model
GriB
> file is a problem. Does MET take its dimensions literally? Does it
> try to go from 0.0 to -0.5 in steps of 1.003 and never get there
> because the end longitude is less than the start?
>
> It's probably time to FTP the files. I'll push model and
observation
> from the above case along with config. I'll let you know when
they're
> done... I suspect it will take a couple hours given the FTP speed
last
> time we did this.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> It's possible that this issue is caused by the fact that it's a
global
> grid. Please try the following...
>
> (1) Run plot_data_plane to make sure MET is reading the gridded data
> the way you expect:
> plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> And then look at that image (adovis.ps) to make sure it looks
good.
>
> (2) Run plot_point_obs twice to see where MET is plotting the
observations:
> plot_point_obs obs.nc obs.ps
> plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
> The first call plots all points on a default global grid. The
> second call plots all points, but using the grid it reads from
model.grb file.
>
> Just look at the resulting images and see if anything jumps out at
you.
> Perhaps you accidentally transposed the lat/lon values in the NetCDF
> point observation file? Or perhaps it's something to do with the
> range of longitude values you used? Or something else I'm not
thinking of.
>
> If the problem doesn't become obvious after those steps, please send
> me a sample forecast grib file, NetCDF observation file, and
> Point-Stat config file.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: point_stat issue
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >
> >
> >
> > Hi,
> >
> > Here's my newest and latest headache using MET v4.1.
> >
> > I'm trying to verify a gridded field using point_stat. The
> > observations are formatted as NetCDF. MET appears to read in
> > everything without a problem but, as far as I can tell, MET throws
> > all the observations out so there's nothing left. Here's what I
get:
> >
> > =================
> > /home/qcteam/METv4.1/bin/point_stat
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > .g rib /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
-log
> > ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats
-v
> > 4 DEBUG 1: Default Config File:
> > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > DEBUG 1: User Config File:
> > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Gb1".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > object of type "FileType_Gb1".
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=1
> > DEBUG 1: Forecast File:
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > .g
> > rib
> > DEBUG 1: Climatology File: none
> > DEBUG 1: Observation File:
> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Reading data for U-GWD/L0.
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match
> > for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
> > records matching VarInfo "U-GWD/L0" in GRIB file
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Searching 64800 observations from 64800 messages.
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> pairs.
> > DEBUG 3: Number of matched pairs = 0
> > DEBUG 3: Observations processed = 64800
> > DEBUG 3: Rejected: GRIB code = 0
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 43229
> > DEBUG 3: Rejected: off the grid = 21571
> > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected: quality
> > marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad fcst
> > value = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2:
> > DEBUG 2:
> >
--------------------------------------------------------------------
> > --
> > ----------
> > DEBUG 2:
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V.stat
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_fho.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_ctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_cts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mctc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mcts.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_cnt.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_sl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_sal1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_vl1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_val1l2.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pct.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pstd.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_pjc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_prc.txt
> > DEBUG 1: Output file:
> >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > 00
> > V_mpr.txt
> > ============
> >
> > What I'm trying to figure out is what 'off the grid' signifies.
The
> > observation data cover the entire global domain, as does the
model.
> > There shouldn't be any 'off the grid' data. Here's the model data
> > information, courtesy degrib.
> >
> > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
kpds7=0
> > levels=(0,0) grid=255 atmos col anl:
> > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
num_in_ave
> > 0 missing 0
> > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> > long 0.000000 to -0.500000 by 1.003000, (359 x 180) scan
0
> > mode
> > 136 bdsgrid 1
> > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
DecScale
> > 6 BinScale 0
> >
> > Is it possible that MET is upset by the definition of longitude as
> > running from 0 to -0.5 in positive steps of 1.003, which it will
> > never reach? I'm not sure how MET establishes the model grid; I
> > presume through the GriB header.
> >
> > I would appreciate any thoughts you have on this. I can FTP the
> > files as before if that will help.
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > Associate Scientist
> > University Corporation for Atmospheric Research 16WS/WXN, Offutt
> > AFB, NE
> > (402) 294-3473 DSN: 271-3473
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Wed Oct 15 12:51:31 2014
Matt,
When I see aodvis.pdf, I do see an obvious problem. There are no
coastlines or map data. So MET is able to read the 2D data fine, but
it
just doesn't know where on earth to place it. So there's a problem
with
our grid definition logic. I'll take a closer look and try to nail it
down.
Thanks,
John
On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> While we wait... here's a curiosity that might be noteworthy, or
not...
>
> When I plot the model GriB data, specifically the field kpds5=147
(AODVIS)
> using plot_data_plane I get the PDF output I've attached.
>
> When the model developer visualizes the data, he swears that the GIF
I've
> attached is the correct layout... a version seemingly rotated 180
degrees
> along the Equatorial axis.
>
> I'm not sure what to believe, but I'm wondering now if
plot_data_plane (or
> MET for that matter) assumes anything regarding the north-south
orientation
> of the data. I can't imagine that it does... but I thought I'd
mention it
> just the same.
>
> The observation and config file have transferred... just waiting on
the
> ~370MB model file to finish.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 12:28 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> OK, sounds good. Just let me know when the transfer is complete and
I'll
> go grab the data.
>
> Thanks,
> John
>
> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >
> > Well the first two plots look just fine, but the third won't work:
> >
> > DEBUG 2: Retrieving grid from file:
> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
> > rib
> > DEBUG 1: Opening netCDF file:
> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > DEBUG 2: Processing 64800 observations at 64800 locations.
> > DEBUG 2: Observation GRIB codes: ALL
> > DEBUG 2: Observation message types: ALL DEBUG 1: Creating
postscript
> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
> > DEBUG 2: Skipped 64800 locations off the grid.
> >
> > Since the second file looks fine, that implies the NetCDF is
formatted
> > right. The GriB data seems okay too. But the pair together are
not.
> > That's what leads me to question if the GriB header in the model
GriB
> > file is a problem. Does MET take its dimensions literally? Does
it
> > try to go from 0.0 to -0.5 in steps of 1.003 and never get there
> > because the end longitude is less than the start?
> >
> > It's probably time to FTP the files. I'll push model and
observation
> > from the above case along with config. I'll let you know when
they're
> > done... I suspect it will take a couple hours given the FTP speed
last
> > time we did this.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, October 15, 2014 11:39 AM
> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >
> > Matt,
> >
> > It's possible that this issue is caused by the fact that it's a
global
> > grid. Please try the following...
> >
> > (1) Run plot_data_plane to make sure MET is reading the gridded
data
> > the way you expect:
> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> > And then look at that image (adovis.ps) to make sure it looks
good.
> >
> > (2) Run plot_point_obs twice to see where MET is plotting the
> observations:
> > plot_point_obs obs.nc obs.ps
> > plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
> > The first call plots all points on a default global grid.
The
> > second call plots all points, but using the grid it reads from
model.grb
> file.
> >
> > Just look at the resulting images and see if anything jumps out at
you.
> > Perhaps you accidentally transposed the lat/lon values in the
NetCDF
> > point observation file? Or perhaps it's something to do with the
> > range of longitude values you used? Or something else I'm not
thinking
> of.
> >
> > If the problem doesn't become obvious after those steps, please
send
> > me a sample forecast grib file, NetCDF observation file, and
> > Point-Stat config file.
> >
> > Thanks,
> > John Halley Gotway
> > met_help at ucar.edu
> >
> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > Queue: met_help
> > > Subject: point_stat issue
> > > Owner: Nobody
> > > Requestors: matthew.sittel.ctr at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > > >
> > >
> > >
> > > Hi,
> > >
> > > Here's my newest and latest headache using MET v4.1.
> > >
> > > I'm trying to verify a gridded field using point_stat. The
> > > observations are formatted as NetCDF. MET appears to read in
> > > everything without a problem but, as far as I can tell, MET
throws
> > > all the observations out so there's nothing left. Here's what I
get:
> > >
> > > =================
> > > /home/qcteam/METv4.1/bin/point_stat
> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > > .g rib /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
-log
> > > ./point_stat_log.txt -outdir /home/qcteam/METv4.1/data/aod/stats
-v
> > > 4 DEBUG 1: Default Config File:
> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > > DEBUG 1: User Config File:
> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_Gb1".
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
> > > object of type "FileType_Gb1".
> > > GSL_RNG_TYPE=mt19937
> > > GSL_RNG_SEED=1
> > > DEBUG 1: Forecast File:
> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
> > > .g
> > > rib
> > > DEBUG 1: Climatology File: none
> > > DEBUG 1: Observation File:
> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> > > DEBUG 2:
> > > DEBUG 2:
> > >
--------------------------------------------------------------------
> > > --
> > > ----------
> > > DEBUG 2:
> > > DEBUG 2: Reading data for U-GWD/L0.
> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
match
> > > for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> > >
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
> > > records matching VarInfo "U-GWD/L0" in GRIB file
> > >
> >
>
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.grib".
> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
levels.
> > > DEBUG 2:
> > > DEBUG 2:
> > >
--------------------------------------------------------------------
> > > --
> > > ----------
> > > DEBUG 2:
> > > DEBUG 2: Searching 64800 observations from 64800 messages.
> > > DEBUG 2:
> > > DEBUG 2:
> > >
--------------------------------------------------------------------
> > > --
> > > ----------
> > > DEBUG 2:
> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation
type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > pairs.
> > > DEBUG 3: Number of matched pairs = 0
> > > DEBUG 3: Observations processed = 64800
> > > DEBUG 3: Rejected: GRIB code = 0
> > > DEBUG 3: Rejected: valid time = 0
> > > DEBUG 3: Rejected: bad obs value = 43229
> > > DEBUG 3: Rejected: off the grid = 21571
> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected: quality
> > > marker = 0
> > > DEBUG 3: Rejected: message type = 0
> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
fcst
> > > value = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > > DEBUG 2:
> > > DEBUG 2:
> > >
--------------------------------------------------------------------
> > > --
> > > ----------
> > > DEBUG 2:
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V.stat
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_fho.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_ctc.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_cts.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_mctc.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_mcts.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_cnt.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_sl1l2.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_sal1l2.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_vl1l2.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_val1l2.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_pct.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_pstd.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_pjc.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_prc.txt
> > > DEBUG 1: Output file:
> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
> > > 00
> > > V_mpr.txt
> > > ============
> > >
> > > What I'm trying to figure out is what 'off the grid' signifies.
The
> > > observation data cover the entire global domain, as does the
model.
> > > There shouldn't be any 'off the grid' data. Here's the model
data
> > > information, courtesy degrib.
> > >
> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
kpds7=0
> > > levels=(0,0) grid=255 atmos col anl:
> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
num_in_ave
> > > 0 missing 0
> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
> > > long 0.000000 to -0.500000 by 1.003000, (359 x 180)
scan 0
> > > mode
> > > 136 bdsgrid 1
> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
DecScale
> > > 6 BinScale 0
> > >
> > > Is it possible that MET is upset by the definition of longitude
as
> > > running from 0 to -0.5 in positive steps of 1.003, which it will
> > > never reach? I'm not sure how MET establishes the model grid; I
> > > presume through the GriB header.
> > >
> > > I would appreciate any thoughts you have on this. I can FTP the
> > > files as before if that will help.
> > >
> > > Thanks,
> > > Matt
> > >
> > > // SIGNED //
> > > Matthew C. Sittel
> > > Associate Scientist
> > > University Corporation for Atmospheric Research 16WS/WXN, Offutt
> > > AFB, NE
> > > (402) 294-3473 DSN: 271-3473
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Wed Oct 15 13:40:43 2014
Matt,
The problem is how the "scanning mode flag" is set in the GRIB1 file
you're
using. It is currently set to 0, which in bits is "00000000". Here's
a
GRIB1 table that describes this flag:
http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
Notice the following in your wgrib output:
scan: WE:NS winds(grid)
All zeros means that the data is stored in the +x and -y directions.
The
grid definition includes lat1 and lat2 values. When data is in the -y
direction, we use lat2 to define the latitude of the lower-left
corner.
But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting the
latitude of the lower-left corner to 90 (when you really want it to be
-90). The best fix would be for the creator of this GRIB1 file to set
that
flag in a way that's consistent with the grid definition.
For testing, I added the following after line 196 of the file
METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1, 3),
decode_lat_lon(gds.grid_type.latlon_grid.lat2,
3));
This just forces it to use the minimum of the two latitudes for the
lower-left corner.
That results in the attached image (aodvis_from_MET.png). Since the
map
data is on there, MET knows where it is on the earth now.
***BUT*** when I plot the same dataset using IDV, I get the attached
image
(aodvis_from_IDV.png). Notice how they are flipped top to bottom.
I've always found this scanning mode flag to be pretty tedious. It
enables
you to pack the data in whatever order you prefer, but that just makes
the
code to read the data more complex and causes issues like this to pop
up.
So there's an inconsistency between the scanning mode flag and the
grid
lat1 and lat2 values. Either lat1 and lat2 need to be swapped or the
scanning mode flag needs to be set differently. My guess is that it's
the
scanning mode flag, because one would naturally assume that 0 is a
good
default value - but it's not. The most common scanning mode I see is
64,
which in bits is 0100000 (i.e. 64 = 2^6, so that means the 2nd bit is
1,
from most-to-least significant bits). And that means data is stored
in the
+x and +y directions.
My suggestion is to have the creator of this GRIB file set the
scanning
mode flag to 64. I manually did that in the MET code, removed the
data.lat_ll line I mentioned above and reran plot_data_plane. It
resulted
in the same aodvis_from_met.png image that I attached.
But you guys need to decide which way is "up" in your data! That's
easiest
if you have any fields whose values correspond well with land.
As a side note, if you to send me a single record from a larger GRIB
file,
you can extract using wgrib like this (use -d to pick the record
number):
wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
Hope that helps.
John
On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Matt,
>
> When I see aodvis.pdf, I do see an obvious problem. There are no
> coastlines or map data. So MET is able to read the 2D data fine,
but it
> just doesn't know where on earth to place it. So there's a problem
with
> our grid definition logic. I'll take a closer look and try to nail
it down.
>
> Thanks,
> John
>
>
> On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>>
>> While we wait... here's a curiosity that might be noteworthy, or
not...
>>
>> When I plot the model GriB data, specifically the field kpds5=147
>> (AODVIS) using plot_data_plane I get the PDF output I've attached.
>>
>> When the model developer visualizes the data, he swears that the
GIF I've
>> attached is the correct layout... a version seemingly rotated 180
degrees
>> along the Equatorial axis.
>>
>> I'm not sure what to believe, but I'm wondering now if
plot_data_plane
>> (or MET for that matter) assumes anything regarding the north-south
>> orientation of the data. I can't imagine that it does... but I
thought I'd
>> mention it just the same.
>>
>> The observation and config file have transferred... just waiting on
the
>> ~370MB model file to finish.
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, October 15, 2014 12:28 PM
>> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>>
>> Matt,
>>
>> OK, sounds good. Just let me know when the transfer is complete
and I'll
>> go grab the data.
>>
>> Thanks,
>> John
>>
>> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
>> WS/WXN via RT <met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>> >
>> > Well the first two plots look just fine, but the third won't
work:
>> >
>> > DEBUG 2: Retrieving grid from file:
>> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300.g
>> > rib
>> > DEBUG 1: Opening netCDF file:
>> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> > DEBUG 2: Processing 64800 observations at 64800 locations.
>> > DEBUG 2: Observation GRIB codes: ALL
>> > DEBUG 2: Observation message types: ALL DEBUG 1: Creating
postscript
>> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
>> > DEBUG 2: Skipped 64800 locations off the grid.
>> >
>> > Since the second file looks fine, that implies the NetCDF is
formatted
>> > right. The GriB data seems okay too. But the pair together are
not.
>> > That's what leads me to question if the GriB header in the model
GriB
>> > file is a problem. Does MET take its dimensions literally? Does
it
>> > try to go from 0.0 to -0.5 in steps of 1.003 and never get there
>> > because the end longitude is less than the start?
>> >
>> > It's probably time to FTP the files. I'll push model and
observation
>> > from the above case along with config. I'll let you know when
they're
>> > done... I suspect it will take a couple hours given the FTP speed
last
>> > time we did this.
>> >
>> > Thanks,
>> > Matt
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, October 15, 2014 11:39 AM
>> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >
>> > Matt,
>> >
>> > It's possible that this issue is caused by the fact that it's a
global
>> > grid. Please try the following...
>> >
>> > (1) Run plot_data_plane to make sure MET is reading the gridded
data
>> > the way you expect:
>> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
>> > And then look at that image (adovis.ps) to make sure it
looks
>> good.
>> >
>> > (2) Run plot_point_obs twice to see where MET is plotting the
>> observations:
>> > plot_point_obs obs.nc obs.ps
>> > plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
>> > The first call plots all points on a default global grid.
The
>> > second call plots all points, but using the grid it reads from
>> model.grb file.
>> >
>> > Just look at the resulting images and see if anything jumps out
at you.
>> > Perhaps you accidentally transposed the lat/lon values in the
NetCDF
>> > point observation file? Or perhaps it's something to do with the
>> > range of longitude values you used? Or something else I'm not
thinking
>> of.
>> >
>> > If the problem doesn't become obvious after those steps, please
send
>> > me a sample forecast grib file, NetCDF observation file, and
>> > Point-Stat config file.
>> >
>> > Thanks,
>> > John Halley Gotway
>> > met_help at ucar.edu
>> >
>> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
>> > WS/WXN via RT <met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
>> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>> > > Queue: met_help
>> > > Subject: point_stat issue
>> > > Owner: Nobody
>> > > Requestors: matthew.sittel.ctr at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>> > > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Here's my newest and latest headache using MET v4.1.
>> > >
>> > > I'm trying to verify a gridded field using point_stat. The
>> > > observations are formatted as NetCDF. MET appears to read in
>> > > everything without a problem but, as far as I can tell, MET
throws
>> > > all the observations out so there's nothing left. Here's what
I get:
>> > >
>> > > =================
>> > > /home/qcteam/METv4.1/bin/point_stat
>> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
>> > > .g rib /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
>> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
-log
>> > > ./point_stat_log.txt -outdir
/home/qcteam/METv4.1/data/aod/stats -v
>> > > 4 DEBUG 1: Default Config File:
>> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
>> > > DEBUG 1: User Config File:
>> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
>> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
>> > > Met2dDataFile object of type "FileType_Gb1".
>> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> > > object of type "FileType_Gb1".
>> > > GSL_RNG_TYPE=mt19937
>> > > GSL_RNG_SEED=1
>> > > DEBUG 1: Forecast File:
>> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
>> > > .g
>> > > rib
>> > > DEBUG 1: Climatology File: none
>> > > DEBUG 1: Observation File:
>> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
--------------------------------------------------------------------
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Reading data for U-GWD/L0.
>> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
match
>> > > for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>> > >
>> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
>> .grib".
>> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
>> > > records matching VarInfo "U-GWD/L0" in GRIB file
>> > >
>> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100300
>> .grib".
>> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
>> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
>> levels.
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
--------------------------------------------------------------------
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Searching 64800 observations from 64800 messages.
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
--------------------------------------------------------------------
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation
type
>> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
>> > pairs.
>> > > DEBUG 3: Number of matched pairs = 0
>> > > DEBUG 3: Observations processed = 64800
>> > > DEBUG 3: Rejected: GRIB code = 0
>> > > DEBUG 3: Rejected: valid time = 0
>> > > DEBUG 3: Rejected: bad obs value = 43229
>> > > DEBUG 3: Rejected: off the grid = 21571
>> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
quality
>> > > marker = 0
>> > > DEBUG 3: Rejected: message type = 0
>> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
fcst
>> > > value = 0
>> > > DEBUG 3: Rejected: duplicates = 0
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
--------------------------------------------------------------------
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V.stat
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_fho.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_ctc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_cts.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_mctc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_mcts.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_cnt.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_sl1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_sal1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_vl1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_val1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_pct.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_pstd.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_pjc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_prc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0000
>> > > 00
>> > > V_mpr.txt
>> > > ============
>> > >
>> > > What I'm trying to figure out is what 'off the grid' signifies.
The
>> > > observation data cover the entire global domain, as does the
model.
>> > > There shouldn't be any 'off the grid' data. Here's the model
data
>> > > information, courtesy degrib.
>> > >
>> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
kpds7=0
>> > > levels=(0,0) grid=255 atmos col anl:
>> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
>> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
num_in_ave
>> > > 0 missing 0
>> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
>> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
>> > > long 0.000000 to -0.500000 by 1.003000, (359 x 180)
scan 0
>> > > mode
>> > > 136 bdsgrid 1
>> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
DecScale
>> > > 6 BinScale 0
>> > >
>> > > Is it possible that MET is upset by the definition of longitude
as
>> > > running from 0 to -0.5 in positive steps of 1.003, which it
will
>> > > never reach? I'm not sure how MET establishes the model grid;
I
>> > > presume through the GriB header.
>> > >
>> > > I would appreciate any thoughts you have on this. I can FTP
the
>> > > files as before if that will help.
>> > >
>> > > Thanks,
>> > > Matt
>> > >
>> > > // SIGNED //
>> > > Matthew C. Sittel
>> > > Associate Scientist
>> > > University Corporation for Atmospheric Research 16WS/WXN,
Offutt
>> > > AFB, NE
>> > > (402) 294-3473 DSN: 271-3473
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Oct 16 06:32:29 2014
Hi John,
Feedback from the modeler:
"Well their plot is still wrong. The area that has the high
concentration of dust (the dark red spot) is Northern Africa. So the
map needs to be rotated 180 degrees. Which I don't understand why
their image isn’t doing it properly, doesn’t the girb file give
lat/lon of each value? I don’t get why other graphic utilities like
xconv have no issues with the grid. Also the idv data is actually
correct map wise (i.e. it is flipped the right way, just the location
is off (again needs to be rotated 180 degrees)"
The model data is built from software sent by NCAR, so it's not in-
house. The header information looks to be straightforward to adjust.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 15, 2014 2:41 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
The problem is how the "scanning mode flag" is set in the GRIB1 file
you're using. It is currently set to 0, which in bits is "00000000".
Here's a
GRIB1 table that describes this flag:
http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
Notice the following in your wgrib output:
scan: WE:NS winds(grid)
All zeros means that the data is stored in the +x and -y directions.
The grid definition includes lat1 and lat2 values. When data is in
the -y direction, we use lat2 to define the latitude of the lower-left
corner.
But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting the
latitude of the lower-left corner to 90 (when you really want it to be
-90). The best fix would be for the creator of this GRIB1 file to set
that flag in a way that's consistent with the grid definition.
For testing, I added the following after line 196 of the file
METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1, 3),
decode_lat_lon(gds.grid_type.latlon_grid.lat2,
3));
This just forces it to use the minimum of the two latitudes for the
lower-left corner.
That results in the attached image (aodvis_from_MET.png). Since the
map data is on there, MET knows where it is on the earth now.
***BUT*** when I plot the same dataset using IDV, I get the attached
image (aodvis_from_IDV.png). Notice how they are flipped top to
bottom.
I've always found this scanning mode flag to be pretty tedious. It
enables you to pack the data in whatever order you prefer, but that
just makes the code to read the data more complex and causes issues
like this to pop up.
So there's an inconsistency between the scanning mode flag and the
grid
lat1 and lat2 values. Either lat1 and lat2 need to be swapped or the
scanning mode flag needs to be set differently. My guess is that it's
the scanning mode flag, because one would naturally assume that 0 is a
good default value - but it's not. The most common scanning mode I
see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so that means the
2nd bit is 1, from most-to-least significant bits). And that means
data is stored in the
+x and +y directions.
My suggestion is to have the creator of this GRIB file set the
scanning mode flag to 64. I manually did that in the MET code,
removed the data.lat_ll line I mentioned above and reran
plot_data_plane. It resulted in the same aodvis_from_met.png image
that I attached.
But you guys need to decide which way is "up" in your data! That's
easiest if you have any fields whose values correspond well with land.
As a side note, if you to send me a single record from a larger GRIB
file, you can extract using wgrib like this (use -d to pick the record
number):
wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
Hope that helps.
John
On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Matt,
>
> When I see aodvis.pdf, I do see an obvious problem. There are no
> coastlines or map data. So MET is able to read the 2D data fine,
but
> it just doesn't know where on earth to place it. So there's a
problem
> with our grid definition logic. I'll take a closer look and try to
nail it down.
>
> Thanks,
> John
>
>
> On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>>
>> While we wait... here's a curiosity that might be noteworthy, or
not...
>>
>> When I plot the model GriB data, specifically the field kpds5=147
>> (AODVIS) using plot_data_plane I get the PDF output I've attached.
>>
>> When the model developer visualizes the data, he swears that the
GIF
>> I've attached is the correct layout... a version seemingly rotated
>> 180 degrees along the Equatorial axis.
>>
>> I'm not sure what to believe, but I'm wondering now if
>> plot_data_plane (or MET for that matter) assumes anything regarding
>> the north-south orientation of the data. I can't imagine that it
>> does... but I thought I'd mention it just the same.
>>
>> The observation and config file have transferred... just waiting on
>> the ~370MB model file to finish.
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, October 15, 2014 12:28 PM
>> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>>
>> Matt,
>>
>> OK, sounds good. Just let me know when the transfer is complete
and
>> I'll go grab the data.
>>
>> Thanks,
>> John
>>
>> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
>> WS/WXN via RT <met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>> >
>> > Well the first two plots look just fine, but the third won't
work:
>> >
>> > DEBUG 2: Retrieving grid from file:
>> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410030
>> > 0.g
>> > rib
>> > DEBUG 1: Opening netCDF file:
>> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> > DEBUG 2: Processing 64800 observations at 64800 locations.
>> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation message
>> > types: ALL DEBUG 1: Creating postscript
>> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
>> > DEBUG 2: Skipped 64800 locations off the grid.
>> >
>> > Since the second file looks fine, that implies the NetCDF is
>> > formatted right. The GriB data seems okay too. But the pair
together are not.
>> > That's what leads me to question if the GriB header in the model
>> > GriB file is a problem. Does MET take its dimensions literally?
>> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and never
get
>> > there because the end longitude is less than the start?
>> >
>> > It's probably time to FTP the files. I'll push model and
>> > observation from the above case along with config. I'll let you
>> > know when they're done... I suspect it will take a couple hours
>> > given the FTP speed last time we did this.
>> >
>> > Thanks,
>> > Matt
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Wednesday, October 15, 2014 11:39 AM
>> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >
>> > Matt,
>> >
>> > It's possible that this issue is caused by the fact that it's a
>> > global grid. Please try the following...
>> >
>> > (1) Run plot_data_plane to make sure MET is reading the gridded
>> > data the way you expect:
>> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
>> > And then look at that image (adovis.ps) to make sure it
looks
>> good.
>> >
>> > (2) Run plot_point_obs twice to see where MET is plotting the
>> observations:
>> > plot_point_obs obs.nc obs.ps
>> > plot_point_obs obs.nc obs_data_file.ps -data_file model.grb
>> > The first call plots all points on a default global grid.
The
>> > second call plots all points, but using the grid it reads from
>> model.grb file.
>> >
>> > Just look at the resulting images and see if anything jumps out
at you.
>> > Perhaps you accidentally transposed the lat/lon values in the
>> > NetCDF point observation file? Or perhaps it's something to do
>> > with the range of longitude values you used? Or something else
I'm
>> > not thinking
>> of.
>> >
>> > If the problem doesn't become obvious after those steps, please
>> > send me a sample forecast grib file, NetCDF observation file, and
>> > Point-Stat config file.
>> >
>> > Thanks,
>> > John Halley Gotway
>> > met_help at ucar.edu
>> >
>> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
>> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
>> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>> > > Queue: met_help
>> > > Subject: point_stat issue
>> > > Owner: Nobody
>> > > Requestors: matthew.sittel.ctr at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>> > > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Here's my newest and latest headache using MET v4.1.
>> > >
>> > > I'm trying to verify a gridded field using point_stat. The
>> > > observations are formatted as NetCDF. MET appears to read in
>> > > everything without a problem but, as far as I can tell, MET
>> > > throws all the observations out so there's nothing left.
Here's what I get:
>> > >
>> > > =================
>> > > /home/qcteam/METv4.1/bin/point_stat
>> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
>> > > 300 .g rib
>> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
>> > > -log ./point_stat_log.txt -outdir
>> > > /home/qcteam/METv4.1/data/aod/stats -v
>> > > 4 DEBUG 1: Default Config File:
>> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
>> > > DEBUG 1: User Config File:
>> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-MODIS.cfg
>> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> > > new Met2dDataFile object of type "FileType_Gb1".
>> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
>> > > object of type "FileType_Gb1".
>> > > GSL_RNG_TYPE=mt19937
>> > > GSL_RNG_SEED=1
>> > > DEBUG 1: Forecast File:
>> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
>> > > 300
>> > > .g
>> > > rib
>> > > DEBUG 1: Climatology File: none
>> > > DEBUG 1: Observation File:
>> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
-----------------------------------------------------------------
>> > > ---
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Reading data for U-GWD/L0.
>> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
>> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>> > >
>> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
>> > 00
>> .grib".
>> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
>> > > records matching VarInfo "U-GWD/L0" in GRIB file
>> > >
>> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
>> > 00
>> .grib".
>> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
>> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0 climatology
>> levels.
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
-----------------------------------------------------------------
>> > > ---
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Searching 64800 observations from 64800 messages.
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
-----------------------------------------------------------------
>> > > ---
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation
>> > > type ADPSFC, over region FULL, for interpolation method
BILIN(4),
>> > > using 0
>> > pairs.
>> > > DEBUG 3: Number of matched pairs = 0
>> > > DEBUG 3: Observations processed = 64800
>> > > DEBUG 3: Rejected: GRIB code = 0
>> > > DEBUG 3: Rejected: valid time = 0
>> > > DEBUG 3: Rejected: bad obs value = 43229
>> > > DEBUG 3: Rejected: off the grid = 21571
>> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
quality
>> > > marker = 0
>> > > DEBUG 3: Rejected: message type = 0
>> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
fcst
>> > > value = 0
>> > > DEBUG 3: Rejected: duplicates = 0
>> > > DEBUG 2:
>> > > DEBUG 2:
>> > >
-----------------------------------------------------------------
>> > > ---
>> > > --
>> > > ----------
>> > > DEBUG 2:
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V.stat
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_fho.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_ctc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_cts.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_mctc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_mcts.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_cnt.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_sl1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_sal1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_vl1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_val1l2.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_pct.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_pstd.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_pjc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_prc.txt
>> > > DEBUG 1: Output file:
>> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> > > 000
>> > > 00
>> > > V_mpr.txt
>> > > ============
>> > >
>> > > What I'm trying to figure out is what 'off the grid' signifies.
>> > > The observation data cover the entire global domain, as does
the model.
>> > > There shouldn't be any 'off the grid' data. Here's the model
>> > > data information, courtesy degrib.
>> > >
>> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
>> > > kpds7=0
>> > > levels=(0,0) grid=255 atmos col anl:
>> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
>> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
>> > > num_in_ave
>> > > 0 missing 0
>> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
>> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny 64620
>> > > long 0.000000 to -0.500000 by 1.003000, (359 x 180)
>> > > scan 0 mode
>> > > 136 bdsgrid 1
>> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
>> > > DecScale
>> > > 6 BinScale 0
>> > >
>> > > Is it possible that MET is upset by the definition of longitude
>> > > as running from 0 to -0.5 in positive steps of 1.003, which it
>> > > will never reach? I'm not sure how MET establishes the model
>> > > grid; I presume through the GriB header.
>> > >
>> > > I would appreciate any thoughts you have on this. I can FTP
the
>> > > files as before if that will help.
>> > >
>> > > Thanks,
>> > > Matt
>> > >
>> > > // SIGNED //
>> > > Matthew C. Sittel
>> > > Associate Scientist
>> > > University Corporation for Atmospheric Research 16WS/WXN,
Offutt
>> > > AFB, NE
>> > > (402) 294-3473 DSN: 271-3473
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Thu Oct 16 10:45:19 2014
Matt,
Would you be able to send me a plot that shows how the data should be
correctly oriented? I'm not familiar with the xconv tool and we don't
have
it on our system.
Thanks,
John
On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Hi John,
>
> Feedback from the modeler:
>
> "Well their plot is still wrong. The area that has the high
concentration
> of dust (the dark red spot) is Northern Africa. So the map needs to
be
> rotated 180 degrees. Which I don't understand why their image isn’t
doing
> it properly, doesn’t the girb file give lat/lon of each value? I
don’t get
> why other graphic utilities like xconv have no issues with the grid.
Also
> the idv data is actually correct map wise (i.e. it is flipped the
right
> way, just the location is off (again needs to be rotated 180
degrees)"
>
> The model data is built from software sent by NCAR, so it's not in-
house.
> The header information looks to be straightforward to adjust.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 2:41 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> The problem is how the "scanning mode flag" is set in the GRIB1 file
> you're using. It is currently set to 0, which in bits is
"00000000".
> Here's a
> GRIB1 table that describes this flag:
> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
>
> Notice the following in your wgrib output:
> scan: WE:NS winds(grid)
>
> All zeros means that the data is stored in the +x and -y directions.
The
> grid definition includes lat1 and lat2 values. When data is in the
-y
> direction, we use lat2 to define the latitude of the lower-left
corner.
> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
the
> latitude of the lower-left corner to 90 (when you really want it to
be
> -90). The best fix would be for the creator of this GRIB1 file to
set that
> flag in a way that's consistent with the grid definition.
>
> For testing, I added the following after line 196 of the file
> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
3),
>
decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> 3));
>
> This just forces it to use the minimum of the two latitudes for the
> lower-left corner.
>
> That results in the attached image (aodvis_from_MET.png). Since the
map
> data is on there, MET knows where it is on the earth now.
>
> ***BUT*** when I plot the same dataset using IDV, I get the attached
image
> (aodvis_from_IDV.png). Notice how they are flipped top to bottom.
>
> I've always found this scanning mode flag to be pretty tedious. It
> enables you to pack the data in whatever order you prefer, but that
just
> makes the code to read the data more complex and causes issues like
this to
> pop up.
>
> So there's an inconsistency between the scanning mode flag and the
grid
> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
the
> scanning mode flag needs to be set differently. My guess is that
it's the
> scanning mode flag, because one would naturally assume that 0 is a
good
> default value - but it's not. The most common scanning mode I see
is 64,
> which in bits is 0100000 (i.e. 64 = 2^6, so that means the 2nd bit
is 1,
> from most-to-least significant bits). And that means data is stored
in the
> +x and +y directions.
>
> My suggestion is to have the creator of this GRIB file set the
scanning
> mode flag to 64. I manually did that in the MET code, removed the
> data.lat_ll line I mentioned above and reran plot_data_plane. It
resulted
> in the same aodvis_from_met.png image that I attached.
>
> But you guys need to decide which way is "up" in your data! That's
> easiest if you have any fields whose values correspond well with
land.
>
> As a side note, if you to send me a single record from a larger GRIB
file,
> you can extract using wgrib like this (use -d to pick the record
number):
> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
>
> Hope that helps.
>
> John
>
>
>
>
>
>
>
>
> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Matt,
> >
> > When I see aodvis.pdf, I do see an obvious problem. There are no
> > coastlines or map data. So MET is able to read the 2D data fine,
but
> > it just doesn't know where on earth to place it. So there's a
problem
> > with our grid definition logic. I'll take a closer look and try
to nail
> it down.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >>
> >> While we wait... here's a curiosity that might be noteworthy, or
not...
> >>
> >> When I plot the model GriB data, specifically the field kpds5=147
> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> >>
> >> When the model developer visualizes the data, he swears that the
GIF
> >> I've attached is the correct layout... a version seemingly
rotated
> >> 180 degrees along the Equatorial axis.
> >>
> >> I'm not sure what to believe, but I'm wondering now if
> >> plot_data_plane (or MET for that matter) assumes anything
regarding
> >> the north-south orientation of the data. I can't imagine that it
> >> does... but I thought I'd mention it just the same.
> >>
> >> The observation and config file have transferred... just waiting
on
> >> the ~370MB model file to finish.
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, October 15, 2014 12:28 PM
> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >>
> >> Matt,
> >>
> >> OK, sounds good. Just let me know when the transfer is complete
and
> >> I'll go grab the data.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> >> WS/WXN via RT <met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >> >
> >> > Well the first two plots look just fine, but the third won't
work:
> >> >
> >> > DEBUG 2: Retrieving grid from file:
> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410030
> >> > 0.g
> >> > rib
> >> > DEBUG 1: Opening netCDF file:
> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
message
> >> > types: ALL DEBUG 1: Creating postscript
> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
> >> > DEBUG 2: Skipped 64800 locations off the grid.
> >> >
> >> > Since the second file looks fine, that implies the NetCDF is
> >> > formatted right. The GriB data seems okay too. But the pair
> together are not.
> >> > That's what leads me to question if the GriB header in the
model
> >> > GriB file is a problem. Does MET take its dimensions
literally?
> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and never
get
> >> > there because the end longitude is less than the start?
> >> >
> >> > It's probably time to FTP the files. I'll push model and
> >> > observation from the above case along with config. I'll let
you
> >> > know when they're done... I suspect it will take a couple hours
> >> > given the FTP speed last time we did this.
> >> >
> >> > Thanks,
> >> > Matt
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >
> >> > Matt,
> >> >
> >> > It's possible that this issue is caused by the fact that it's a
> >> > global grid. Please try the following...
> >> >
> >> > (1) Run plot_data_plane to make sure MET is reading the gridded
> >> > data the way you expect:
> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> >> > And then look at that image (adovis.ps) to make sure it
looks
> >> good.
> >> >
> >> > (2) Run plot_point_obs twice to see where MET is plotting the
> >> observations:
> >> > plot_point_obs obs.nc obs.ps
> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> >> > The first call plots all points on a default global grid.
The
> >> > second call plots all points, but using the grid it reads from
> >> model.grb file.
> >> >
> >> > Just look at the resulting images and see if anything jumps out
at
> you.
> >> > Perhaps you accidentally transposed the lat/lon values in the
> >> > NetCDF point observation file? Or perhaps it's something to do
> >> > with the range of longitude values you used? Or something else
I'm
> >> > not thinking
> >> of.
> >> >
> >> > If the problem doesn't become obvious after those steps, please
> >> > send me a sample forecast grib file, NetCDF observation file,
and
> >> > Point-Stat config file.
> >> >
> >> > Thanks,
> >> > John Halley Gotway
> >> > met_help at ucar.edu
> >> >
> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> >> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: point_stat issue
> >> > > Owner: Nobody
> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >> > > >
> >> > >
> >> > >
> >> > > Hi,
> >> > >
> >> > > Here's my newest and latest headache using MET v4.1.
> >> > >
> >> > > I'm trying to verify a gridded field using point_stat. The
> >> > > observations are formatted as NetCDF. MET appears to read in
> >> > > everything without a problem but, as far as I can tell, MET
> >> > > throws all the observations out so there's nothing left.
Here's
> what I get:
> >> > >
> >> > > =================
> >> > > /home/qcteam/METv4.1/bin/point_stat
> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
> >> > > 300 .g rib
> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
> >> > > -log ./point_stat_log.txt -outdir
> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> >> > > 4 DEBUG 1: Default Config File:
> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> >> > > DEBUG 1: User Config File:
> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> >> > > new Met2dDataFile object of type "FileType_Gb1".
> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> >> > > object of type "FileType_Gb1".
> >> > > GSL_RNG_TYPE=mt19937
> >> > > GSL_RNG_SEED=1
> >> > > DEBUG 1: Forecast File:
> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
> >> > > 300
> >> > > .g
> >> > > rib
> >> > > DEBUG 1: Climatology File: none
> >> > > DEBUG 1: Observation File:
> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
-----------------------------------------------------------------
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Reading data for U-GWD/L0.
> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> >> > >
> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
> >> > 00
> >> .grib".
> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
> >> > > records matching VarInfo "U-GWD/L0" in GRIB file
> >> > >
> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
> >> > 00
> >> .grib".
> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
climatology
> >> levels.
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
-----------------------------------------------------------------
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
-----------------------------------------------------------------
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation
> >> > > type ADPSFC, over region FULL, for interpolation method
BILIN(4),
> >> > > using 0
> >> > pairs.
> >> > > DEBUG 3: Number of matched pairs = 0
> >> > > DEBUG 3: Observations processed = 64800
> >> > > DEBUG 3: Rejected: GRIB code = 0
> >> > > DEBUG 3: Rejected: valid time = 0
> >> > > DEBUG 3: Rejected: bad obs value = 43229
> >> > > DEBUG 3: Rejected: off the grid = 21571
> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
quality
> >> > > marker = 0
> >> > > DEBUG 3: Rejected: message type = 0
> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
fcst
> >> > > value = 0
> >> > > DEBUG 3: Rejected: duplicates = 0
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
-----------------------------------------------------------------
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V.stat
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_fho.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_ctc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_cts.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_mctc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_mcts.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_cnt.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_sl1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_sal1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_vl1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_val1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_pct.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_pstd.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_pjc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_prc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
> >> > > 000
> >> > > 00
> >> > > V_mpr.txt
> >> > > ============
> >> > >
> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> >> > > The observation data cover the entire global domain, as does
the
> model.
> >> > > There shouldn't be any 'off the grid' data. Here's the model
> >> > > data information, courtesy degrib.
> >> > >
> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
> >> > > kpds7=0
> >> > > levels=(0,0) grid=255 atmos col anl:
> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
> >> > > num_in_ave
> >> > > 0 missing 0
> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x 180)
> >> > > scan 0 mode
> >> > > 136 bdsgrid 1
> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
> >> > > DecScale
> >> > > 6 BinScale 0
> >> > >
> >> > > Is it possible that MET is upset by the definition of
longitude
> >> > > as running from 0 to -0.5 in positive steps of 1.003, which
it
> >> > > will never reach? I'm not sure how MET establishes the model
> >> > > grid; I presume through the GriB header.
> >> > >
> >> > > I would appreciate any thoughts you have on this. I can FTP
the
> >> > > files as before if that will help.
> >> > >
> >> > > Thanks,
> >> > > Matt
> >> > >
> >> > > // SIGNED //
> >> > > Matthew C. Sittel
> >> > > Associate Scientist
> >> > > University Corporation for Atmospheric Research 16WS/WXN,
Offutt
> >> > > AFB, NE
> >> > > (402) 294-3473 DSN: 271-3473
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Oct 16 11:15:40 2014
Hey John,
Here is the correct layout of the data. The bright area in the middle
of the plot should be over north Africa.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 16, 2014 11:45 AM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
Would you be able to send me a plot that shows how the data should be
correctly oriented? I'm not familiar with the xconv tool and we don't
have it on our system.
Thanks,
John
On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Hi John,
>
> Feedback from the modeler:
>
> "Well their plot is still wrong. The area that has the high
> concentration of dust (the dark red spot) is Northern Africa. So the
> map needs to be rotated 180 degrees. Which I don't understand why
> their image isn’t doing it properly, doesn’t the girb file give
> lat/lon of each value? I don’t get why other graphic utilities like
> xconv have no issues with the grid. Also the idv data is actually
> correct map wise (i.e. it is flipped the right way, just the
location is off (again needs to be rotated 180 degrees)"
>
> The model data is built from software sent by NCAR, so it's not in-
house.
> The header information looks to be straightforward to adjust.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 2:41 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> The problem is how the "scanning mode flag" is set in the GRIB1 file
> you're using. It is currently set to 0, which in bits is
"00000000".
> Here's a
> GRIB1 table that describes this flag:
> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
>
> Notice the following in your wgrib output:
> scan: WE:NS winds(grid)
>
> All zeros means that the data is stored in the +x and -y directions.
> The grid definition includes lat1 and lat2 values. When data is in
> the -y direction, we use lat2 to define the latitude of the lower-
left corner.
> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
the
> latitude of the lower-left corner to 90 (when you really want it to
be
> -90). The best fix would be for the creator of this GRIB1 file to
set
> that flag in a way that's consistent with the grid definition.
>
> For testing, I added the following after line 196 of the file
> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
3),
>
> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> 3));
>
> This just forces it to use the minimum of the two latitudes for the
> lower-left corner.
>
> That results in the attached image (aodvis_from_MET.png). Since the
> map data is on there, MET knows where it is on the earth now.
>
> ***BUT*** when I plot the same dataset using IDV, I get the attached
> image (aodvis_from_IDV.png). Notice how they are flipped top to
bottom.
>
> I've always found this scanning mode flag to be pretty tedious. It
> enables you to pack the data in whatever order you prefer, but that
> just makes the code to read the data more complex and causes issues
> like this to pop up.
>
> So there's an inconsistency between the scanning mode flag and the
> grid
> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
the
> scanning mode flag needs to be set differently. My guess is that
it's
> the scanning mode flag, because one would naturally assume that 0 is
a
> good default value - but it's not. The most common scanning mode I
> see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so that means
the
> 2nd bit is 1, from most-to-least significant bits). And that means
> data is stored in the
> +x and +y directions.
>
> My suggestion is to have the creator of this GRIB file set the
> scanning mode flag to 64. I manually did that in the MET code,
> removed the data.lat_ll line I mentioned above and reran
> plot_data_plane. It resulted in the same aodvis_from_met.png image
that I attached.
>
> But you guys need to decide which way is "up" in your data! That's
> easiest if you have any fields whose values correspond well with
land.
>
> As a side note, if you to send me a single record from a larger GRIB
> file, you can extract using wgrib like this (use -d to pick the
record number):
> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
>
> Hope that helps.
>
> John
>
>
>
>
>
>
>
>
> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Matt,
> >
> > When I see aodvis.pdf, I do see an obvious problem. There are no
> > coastlines or map data. So MET is able to read the 2D data fine,
> > but it just doesn't know where on earth to place it. So there's a
> > problem with our grid definition logic. I'll take a closer look
and
> > try to nail
> it down.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >>
> >> While we wait... here's a curiosity that might be noteworthy, or
not...
> >>
> >> When I plot the model GriB data, specifically the field kpds5=147
> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> >>
> >> When the model developer visualizes the data, he swears that the
> >> GIF I've attached is the correct layout... a version seemingly
> >> rotated
> >> 180 degrees along the Equatorial axis.
> >>
> >> I'm not sure what to believe, but I'm wondering now if
> >> plot_data_plane (or MET for that matter) assumes anything
regarding
> >> the north-south orientation of the data. I can't imagine that it
> >> does... but I thought I'd mention it just the same.
> >>
> >> The observation and config file have transferred... just waiting
on
> >> the ~370MB model file to finish.
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, October 15, 2014 12:28 PM
> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >>
> >> Matt,
> >>
> >> OK, sounds good. Just let me know when the transfer is complete
> >> and I'll go grab the data.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >> >
> >> > Well the first two plots look just fine, but the third won't
work:
> >> >
> >> > DEBUG 2: Retrieving grid from file:
> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
> >> > 30
> >> > 0.g
> >> > rib
> >> > DEBUG 1: Opening netCDF file:
> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
message
> >> > types: ALL DEBUG 1: Creating postscript
> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
> >> > DEBUG 2: Skipped 64800 locations off the grid.
> >> >
> >> > Since the second file looks fine, that implies the NetCDF is
> >> > formatted right. The GriB data seems okay too. But the pair
> together are not.
> >> > That's what leads me to question if the GriB header in the
model
> >> > GriB file is a problem. Does MET take its dimensions
literally?
> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and never
> >> > get there because the end longitude is less than the start?
> >> >
> >> > It's probably time to FTP the files. I'll push model and
> >> > observation from the above case along with config. I'll let
you
> >> > know when they're done... I suspect it will take a couple hours
> >> > given the FTP speed last time we did this.
> >> >
> >> > Thanks,
> >> > Matt
> >> >
> >> > -----Original Message-----
> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >
> >> > Matt,
> >> >
> >> > It's possible that this issue is caused by the fact that it's a
> >> > global grid. Please try the following...
> >> >
> >> > (1) Run plot_data_plane to make sure MET is reading the gridded
> >> > data the way you expect:
> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
level="L0";'
> >> > And then look at that image (adovis.ps) to make sure it
> >> > looks
> >> good.
> >> >
> >> > (2) Run plot_point_obs twice to see where MET is plotting the
> >> observations:
> >> > plot_point_obs obs.nc obs.ps
> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> >> > The first call plots all points on a default global grid.
> >> > The second call plots all points, but using the grid it reads
> >> > from
> >> model.grb file.
> >> >
> >> > Just look at the resulting images and see if anything jumps out
> >> > at
> you.
> >> > Perhaps you accidentally transposed the lat/lon values in the
> >> > NetCDF point observation file? Or perhaps it's something to do
> >> > with the range of longitude values you used? Or something else
> >> > I'm not thinking
> >> of.
> >> >
> >> > If the problem doesn't become obvious after those steps, please
> >> > send me a sample forecast grib file, NetCDF observation file,
and
> >> > Point-Stat config file.
> >> >
> >> > Thanks,
> >> > John Halley Gotway
> >> > met_help at ucar.edu
> >> >
> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> >> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> >> > > Queue: met_help
> >> > > Subject: point_stat issue
> >> > > Owner: Nobody
> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> >> > > Status: new
> >> > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >> > > >
> >> > >
> >> > >
> >> > > Hi,
> >> > >
> >> > > Here's my newest and latest headache using MET v4.1.
> >> > >
> >> > > I'm trying to verify a gridded field using point_stat. The
> >> > > observations are formatted as NetCDF. MET appears to read in
> >> > > everything without a problem but, as far as I can tell, MET
> >> > > throws all the observations out so there's nothing left.
> >> > > Here's
> what I get:
> >> > >
> >> > > =================
> >> > > /home/qcteam/METv4.1/bin/point_stat
> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
> >> > > 00
> >> > > 300 .g rib
> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
> >> > > -log ./point_stat_log.txt -outdir
> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> >> > > 4 DEBUG 1: Default Config File:
> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> >> > > DEBUG 1: User Config File:
> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> >> > > created new Met2dDataFile object of type "FileType_Gb1".
> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> >> > > object of type "FileType_Gb1".
> >> > > GSL_RNG_TYPE=mt19937
> >> > > GSL_RNG_SEED=1
> >> > > DEBUG 1: Forecast File:
> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
> >> > > 00
> >> > > 300
> >> > > .g
> >> > > rib
> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation File:
> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
---------------------------------------------------------------
> >> > > --
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Reading data for U-GWD/L0.
> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
> >> > >
> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410
> >> > 03
> >> > 00
> >> .grib".
> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
> >> > > records matching VarInfo "U-GWD/L0" in GRIB file
> >> > >
> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410
> >> > 03
> >> > 00
> >> .grib".
> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
climatology
> >> levels.
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
---------------------------------------------------------------
> >> > > --
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
---------------------------------------------------------------
> >> > > --
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for observation
> >> > > type ADPSFC, over region FULL, for interpolation method
> >> > > BILIN(4), using 0
> >> > pairs.
> >> > > DEBUG 3: Number of matched pairs = 0
> >> > > DEBUG 3: Observations processed = 64800
> >> > > DEBUG 3: Rejected: GRIB code = 0
> >> > > DEBUG 3: Rejected: valid time = 0
> >> > > DEBUG 3: Rejected: bad obs value = 43229
> >> > > DEBUG 3: Rejected: off the grid = 21571
> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
> >> > > quality marker = 0
> >> > > DEBUG 3: Rejected: message type = 0
> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
> >> > > fcst value = 0
> >> > > DEBUG 3: Rejected: duplicates = 0
> >> > > DEBUG 2:
> >> > > DEBUG 2:
> >> > >
---------------------------------------------------------------
> >> > > --
> >> > > ---
> >> > > --
> >> > > ----------
> >> > > DEBUG 2:
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V.stat
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_fho.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_ctc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_cts.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_mctc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_mcts.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_cnt.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_sl1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_sal1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_vl1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_val1l2.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_pct.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_pstd.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_pjc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_prc.txt
> >> > > DEBUG 1: Output file:
> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003
> >> > > _0
> >> > > 000
> >> > > 00
> >> > > V_mpr.txt
> >> > > ============
> >> > >
> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> >> > > The observation data cover the entire global domain, as does
> >> > > the
> model.
> >> > > There shouldn't be any 'off the grid' data. Here's the model
> >> > > data information, courtesy degrib.
> >> > >
> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
> >> > > kpds7=0
> >> > > levels=(0,0) grid=255 atmos col anl:
> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
> >> > > num_in_ave
> >> > > 0 missing 0
> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
winds(grid)
> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x 180)
> >> > > scan 0 mode
> >> > > 136 bdsgrid 1
> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
> >> > > DecScale
> >> > > 6 BinScale 0
> >> > >
> >> > > Is it possible that MET is upset by the definition of
longitude
> >> > > as running from 0 to -0.5 in positive steps of 1.003, which
it
> >> > > will never reach? I'm not sure how MET establishes the model
> >> > > grid; I presume through the GriB header.
> >> > >
> >> > > I would appreciate any thoughts you have on this. I can FTP
> >> > > the files as before if that will help.
> >> > >
> >> > > Thanks,
> >> > > Matt
> >> > >
> >> > > // SIGNED //
> >> > > Matthew C. Sittel
> >> > > Associate Scientist
> >> > > University Corporation for Atmospheric Research 16WS/WXN,
> >> > > Offutt AFB, NE
> >> > > (402) 294-3473 DSN: 271-3473
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Thu Oct 16 11:28:07 2014
Matt,
I've attached two updated images.
"aodvis_64.png" shows how MET reads the data when the scanning mode
flag is
set to 64. The plot I sent yesterday was wrong... I manually changed
the
scan flag in the code to 64 in one spot but missed it in a second
spot.
Sorry for the confusion.
"aodvis_64_shift_180.png" shows how MET reads the data when the
scanning
mode flag is set to 64 and the lower-left longitude is set to -180,
instead
of 0.
Looks like the second image orients the data correctly and put it on
the
correct place on the earth. So perhaps the scanning mode flag and the
grid
definition are incorrect?
Thanks,
John
On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Matt,
>
> Would you be able to send me a plot that shows how the data should
be
> correctly oriented? I'm not familiar with the xconv tool and we
don't have
> it on our system.
>
> Thanks,
> John
>
> On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>>
>> Hi John,
>>
>> Feedback from the modeler:
>>
>> "Well their plot is still wrong. The area that has the high
concentration
>> of dust (the dark red spot) is Northern Africa. So the map needs to
be
>> rotated 180 degrees. Which I don't understand why their image isn’t
doing
>> it properly, doesn’t the girb file give lat/lon of each value? I
don’t get
>> why other graphic utilities like xconv have no issues with the
grid. Also
>> the idv data is actually correct map wise (i.e. it is flipped the
right
>> way, just the location is off (again needs to be rotated 180
degrees)"
>>
>> The model data is built from software sent by NCAR, so it's not
>> in-house. The header information looks to be straightforward to
adjust.
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, October 15, 2014 2:41 PM
>> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>>
>> Matt,
>>
>> The problem is how the "scanning mode flag" is set in the GRIB1
file
>> you're using. It is currently set to 0, which in bits is
"00000000".
>> Here's a
>> GRIB1 table that describes this flag:
>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
>>
>> Notice the following in your wgrib output:
>> scan: WE:NS winds(grid)
>>
>> All zeros means that the data is stored in the +x and -y
directions. The
>> grid definition includes lat1 and lat2 values. When data is in the
-y
>> direction, we use lat2 to define the latitude of the lower-left
corner.
>> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
the
>> latitude of the lower-left corner to 90 (when you really want it to
be
>> -90). The best fix would be for the creator of this GRIB1 file to
set that
>> flag in a way that's consistent with the grid definition.
>>
>> For testing, I added the following after line 196 of the file
>> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
>> data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
3),
>>
decode_lat_lon(gds.grid_type.latlon_grid.lat2,
>> 3));
>>
>> This just forces it to use the minimum of the two latitudes for the
>> lower-left corner.
>>
>> That results in the attached image (aodvis_from_MET.png). Since
the map
>> data is on there, MET knows where it is on the earth now.
>>
>> ***BUT*** when I plot the same dataset using IDV, I get the
attached
>> image (aodvis_from_IDV.png). Notice how they are flipped top to
bottom.
>>
>> I've always found this scanning mode flag to be pretty tedious. It
>> enables you to pack the data in whatever order you prefer, but that
just
>> makes the code to read the data more complex and causes issues like
this to
>> pop up.
>>
>> So there's an inconsistency between the scanning mode flag and the
grid
>> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
the
>> scanning mode flag needs to be set differently. My guess is that
it's the
>> scanning mode flag, because one would naturally assume that 0 is a
good
>> default value - but it's not. The most common scanning mode I see
is 64,
>> which in bits is 0100000 (i.e. 64 = 2^6, so that means the 2nd bit
is 1,
>> from most-to-least significant bits). And that means data is
stored in the
>> +x and +y directions.
>>
>> My suggestion is to have the creator of this GRIB file set the
scanning
>> mode flag to 64. I manually did that in the MET code, removed the
>> data.lat_ll line I mentioned above and reran plot_data_plane. It
resulted
>> in the same aodvis_from_met.png image that I attached.
>>
>> But you guys need to decide which way is "up" in your data! That's
>> easiest if you have any fields whose values correspond well with
land.
>>
>> As a side note, if you to send me a single record from a larger
GRIB
>> file, you can extract using wgrib like this (use -d to pick the
record
>> number):
>> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
>>
>> Hope that helps.
>>
>> John
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>
>> > Matt,
>> >
>> > When I see aodvis.pdf, I do see an obvious problem. There are no
>> > coastlines or map data. So MET is able to read the 2D data fine,
but
>> > it just doesn't know where on earth to place it. So there's a
problem
>> > with our grid definition logic. I'll take a closer look and try
to
>> nail it down.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
>> > WS/WXN via RT <met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>> >>
>> >> While we wait... here's a curiosity that might be noteworthy, or
not...
>> >>
>> >> When I plot the model GriB data, specifically the field
kpds5=147
>> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
>> >>
>> >> When the model developer visualizes the data, he swears that the
GIF
>> >> I've attached is the correct layout... a version seemingly
rotated
>> >> 180 degrees along the Equatorial axis.
>> >>
>> >> I'm not sure what to believe, but I'm wondering now if
>> >> plot_data_plane (or MET for that matter) assumes anything
regarding
>> >> the north-south orientation of the data. I can't imagine that
it
>> >> does... but I thought I'd mention it just the same.
>> >>
>> >> The observation and config file have transferred... just waiting
on
>> >> the ~370MB model file to finish.
>> >>
>> >> -----Original Message-----
>> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> >> Sent: Wednesday, October 15, 2014 12:28 PM
>> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >>
>> >> Matt,
>> >>
>> >> OK, sounds good. Just let me know when the transfer is complete
and
>> >> I'll go grab the data.
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA 16
>> >> WS/WXN via RT <met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>
>> >> >
>> >> > Well the first two plots look just fine, but the third won't
work:
>> >> >
>> >> > DEBUG 2: Retrieving grid from file:
>> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410030
>> >> > 0.g
>> >> > rib
>> >> > DEBUG 1: Opening netCDF file:
>> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
>> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
message
>> >> > types: ALL DEBUG 1: Creating postscript
>> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
>> >> > DEBUG 2: Skipped 64800 locations off the grid.
>> >> >
>> >> > Since the second file looks fine, that implies the NetCDF is
>> >> > formatted right. The GriB data seems okay too. But the pair
>> together are not.
>> >> > That's what leads me to question if the GriB header in the
model
>> >> > GriB file is a problem. Does MET take its dimensions
literally?
>> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and never
get
>> >> > there because the end longitude is less than the start?
>> >> >
>> >> > It's probably time to FTP the files. I'll push model and
>> >> > observation from the above case along with config. I'll let
you
>> >> > know when they're done... I suspect it will take a couple
hours
>> >> > given the FTP speed last time we did this.
>> >> >
>> >> > Thanks,
>> >> > Matt
>> >> >
>> >> > -----Original Message-----
>> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> >> > Sent: Wednesday, October 15, 2014 11:39 AM
>> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >> >
>> >> > Matt,
>> >> >
>> >> > It's possible that this issue is caused by the fact that it's
a
>> >> > global grid. Please try the following...
>> >> >
>> >> > (1) Run plot_data_plane to make sure MET is reading the
gridded
>> >> > data the way you expect:
>> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
>> level="L0";'
>> >> > And then look at that image (adovis.ps) to make sure it
looks
>> >> good.
>> >> >
>> >> > (2) Run plot_point_obs twice to see where MET is plotting the
>> >> observations:
>> >> > plot_point_obs obs.nc obs.ps
>> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
>> >> > The first call plots all points on a default global grid.
The
>> >> > second call plots all points, but using the grid it reads from
>> >> model.grb file.
>> >> >
>> >> > Just look at the resulting images and see if anything jumps
out at
>> you.
>> >> > Perhaps you accidentally transposed the lat/lon values in the
>> >> > NetCDF point observation file? Or perhaps it's something to
do
>> >> > with the range of longitude values you used? Or something
else I'm
>> >> > not thinking
>> >> of.
>> >> >
>> >> > If the problem doesn't become obvious after those steps,
please
>> >> > send me a sample forecast grib file, NetCDF observation file,
and
>> >> > Point-Stat config file.
>> >> >
>> >> > Thanks,
>> >> > John Halley Gotway
>> >> > met_help at ucar.edu
>> >> >
>> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA
>> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
>> >> >
>> >> > >
>> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
>> >> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>> >> > > Queue: met_help
>> >> > > Subject: point_stat issue
>> >> > > Owner: Nobody
>> >> > > Requestors: matthew.sittel.ctr at us.af.mil
>> >> > > Status: new
>> >> > > Ticket <URL:
>> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>> >> > > >
>> >> > >
>> >> > >
>> >> > > Hi,
>> >> > >
>> >> > > Here's my newest and latest headache using MET v4.1.
>> >> > >
>> >> > > I'm trying to verify a gridded field using point_stat. The
>> >> > > observations are formatted as NetCDF. MET appears to read
in
>> >> > > everything without a problem but, as far as I can tell, MET
>> >> > > throws all the observations out so there's nothing left.
Here's
>> what I get:
>> >> > >
>> >> > > =================
>> >> > > /home/qcteam/METv4.1/bin/point_stat
>> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
>> >> > > 300 .g rib
>> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
>> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
>> >> > > -log ./point_stat_log.txt -outdir
>> >> > > /home/qcteam/METv4.1/data/aod/stats -v
>> >> > > 4 DEBUG 1: Default Config File:
>> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
>> >> > > DEBUG 1: User Config File:
>> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cfg
>> >> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> >> > > new Met2dDataFile object of type "FileType_Gb1".
>> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>> >> > > object of type "FileType_Gb1".
>> >> > > GSL_RNG_TYPE=mt19937
>> >> > > GSL_RNG_SEED=1
>> >> > > DEBUG 1: Forecast File:
>> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014100
>> >> > > 300
>> >> > > .g
>> >> > > rib
>> >> > > DEBUG 1: Climatology File: none
>> >> > > DEBUG 1: Observation File:
>> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
-----------------------------------------------------------------
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Reading data for U-GWD/L0.
>> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
>> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>> >> > >
>> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
>> >> > 00
>> >> .grib".
>> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
>> >> > > records matching VarInfo "U-GWD/L0" in GRIB file
>> >> > >
>> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141003
>> >> > 00
>> >> .grib".
>> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
>> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
climatology
>> >> levels.
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
-----------------------------------------------------------------
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
-----------------------------------------------------------------
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
observation
>> >> > > type ADPSFC, over region FULL, for interpolation method
BILIN(4),
>> >> > > using 0
>> >> > pairs.
>> >> > > DEBUG 3: Number of matched pairs = 0
>> >> > > DEBUG 3: Observations processed = 64800
>> >> > > DEBUG 3: Rejected: GRIB code = 0
>> >> > > DEBUG 3: Rejected: valid time = 0
>> >> > > DEBUG 3: Rejected: bad obs value = 43229
>> >> > > DEBUG 3: Rejected: off the grid = 21571
>> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
quality
>> >> > > marker = 0
>> >> > > DEBUG 3: Rejected: message type = 0
>> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
fcst
>> >> > > value = 0
>> >> > > DEBUG 3: Rejected: duplicates = 0
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
-----------------------------------------------------------------
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V.stat
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_fho.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_ctc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_cts.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mctc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mcts.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_cnt.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_sl1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_sal1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_vl1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_val1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pct.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pstd.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pjc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_prc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141003_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mpr.txt
>> >> > > ============
>> >> > >
>> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
>> >> > > The observation data cover the entire global domain, as does
the
>> model.
>> >> > > There shouldn't be any 'off the grid' data. Here's the
model
>> >> > > data information, courtesy degrib.
>> >> > >
>> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
>> >> > > kpds7=0
>> >> > > levels=(0,0) grid=255 atmos col anl:
>> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
>> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
>> >> > > num_in_ave
>> >> > > 0 missing 0
>> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
>> winds(grid)
>> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
>> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
180)
>> >> > > scan 0 mode
>> >> > > 136 bdsgrid 1
>> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
>> >> > > DecScale
>> >> > > 6 BinScale 0
>> >> > >
>> >> > > Is it possible that MET is upset by the definition of
longitude
>> >> > > as running from 0 to -0.5 in positive steps of 1.003, which
it
>> >> > > will never reach? I'm not sure how MET establishes the
model
>> >> > > grid; I presume through the GriB header.
>> >> > >
>> >> > > I would appreciate any thoughts you have on this. I can FTP
the
>> >> > > files as before if that will help.
>> >> > >
>> >> > > Thanks,
>> >> > > Matt
>> >> > >
>> >> > > // SIGNED //
>> >> > > Matthew C. Sittel
>> >> > > Associate Scientist
>> >> > > University Corporation for Atmospheric Research 16WS/WXN,
Offutt
>> >> > > AFB, NE
>> >> > > (402) 294-3473 DSN: 271-3473
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Oct 16 12:15:32 2014
Hi John,
Since I'm not the modeler, I don't know all the details, but what I
think is happening is that the header is being set wrong. There are
no LAT or LON fields in the GriB file to check it against. I don't
think the lat and lon corner points in the header are automatically
set. He's trying one more run... maybe that's all this is. I'll let
you know.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 16, 2014 12:28 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
I've attached two updated images.
"aodvis_64.png" shows how MET reads the data when the scanning mode
flag is set to 64. The plot I sent yesterday was wrong... I manually
changed the scan flag in the code to 64 in one spot but missed it in a
second spot.
Sorry for the confusion.
"aodvis_64_shift_180.png" shows how MET reads the data when the
scanning mode flag is set to 64 and the lower-left longitude is set to
-180, instead of 0.
Looks like the second image orients the data correctly and put it on
the correct place on the earth. So perhaps the scanning mode flag and
the grid definition are incorrect?
Thanks,
John
On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Matt,
>
> Would you be able to send me a plot that shows how the data should
be
> correctly oriented? I'm not familiar with the xconv tool and we
don't
> have it on our system.
>
> Thanks,
> John
>
> On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>>
>> Hi John,
>>
>> Feedback from the modeler:
>>
>> "Well their plot is still wrong. The area that has the high
>> concentration of dust (the dark red spot) is Northern Africa. So
the
>> map needs to be rotated 180 degrees. Which I don't understand why
>> their image isn’t doing it properly, doesn’t the girb file give
>> lat/lon of each value? I don’t get why other graphic utilities like
>> xconv have no issues with the grid. Also the idv data is actually
>> correct map wise (i.e. it is flipped the right way, just the
location is off (again needs to be rotated 180 degrees)"
>>
>> The model data is built from software sent by NCAR, so it's not
>> in-house. The header information looks to be straightforward to
adjust.
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Wednesday, October 15, 2014 2:41 PM
>> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>>
>> Matt,
>>
>> The problem is how the "scanning mode flag" is set in the GRIB1
file
>> you're using. It is currently set to 0, which in bits is
"00000000".
>> Here's a
>> GRIB1 table that describes this flag:
>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
>>
>> Notice the following in your wgrib output:
>> scan: WE:NS winds(grid)
>>
>> All zeros means that the data is stored in the +x and -y
directions.
>> The grid definition includes lat1 and lat2 values. When data is in
>> the -y direction, we use lat2 to define the latitude of the lower-
left corner.
>> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
the
>> latitude of the lower-left corner to 90 (when you really want it to
>> be -90). The best fix would be for the creator of this GRIB1 file
to
>> set that flag in a way that's consistent with the grid definition.
>>
>> For testing, I added the following after line 196 of the file
>> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
>> data.lat_ll = min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
3),
>>
>> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
>> 3));
>>
>> This just forces it to use the minimum of the two latitudes for the
>> lower-left corner.
>>
>> That results in the attached image (aodvis_from_MET.png). Since
the
>> map data is on there, MET knows where it is on the earth now.
>>
>> ***BUT*** when I plot the same dataset using IDV, I get the
attached
>> image (aodvis_from_IDV.png). Notice how they are flipped top to
bottom.
>>
>> I've always found this scanning mode flag to be pretty tedious. It
>> enables you to pack the data in whatever order you prefer, but that
>> just makes the code to read the data more complex and causes issues
>> like this to pop up.
>>
>> So there's an inconsistency between the scanning mode flag and the
>> grid
>> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
the
>> scanning mode flag needs to be set differently. My guess is that
>> it's the scanning mode flag, because one would naturally assume
that
>> 0 is a good default value - but it's not. The most common scanning
>> mode I see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so that
>> means the 2nd bit is 1, from most-to-least significant bits). And
>> that means data is stored in the
>> +x and +y directions.
>>
>> My suggestion is to have the creator of this GRIB file set the
>> scanning mode flag to 64. I manually did that in the MET code,
>> removed the data.lat_ll line I mentioned above and reran
>> plot_data_plane. It resulted in the same aodvis_from_met.png image
that I attached.
>>
>> But you guys need to decide which way is "up" in your data! That's
>> easiest if you have any fields whose values correspond well with
land.
>>
>> As a side note, if you to send me a single record from a larger
GRIB
>> file, you can extract using wgrib like this (use -d to pick the
>> record
>> number):
>> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
>>
>> Hope that helps.
>>
>> John
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
>> <johnhg at ucar.edu>
>> wrote:
>>
>> > Matt,
>> >
>> > When I see aodvis.pdf, I do see an obvious problem. There are no
>> > coastlines or map data. So MET is able to read the 2D data fine,
>> > but it just doesn't know where on earth to place it. So there's
a
>> > problem with our grid definition logic. I'll take a closer look
>> > and try to
>> nail it down.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF AFWA
>> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>> >>
>> >> While we wait... here's a curiosity that might be noteworthy, or
not...
>> >>
>> >> When I plot the model GriB data, specifically the field
kpds5=147
>> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
>> >>
>> >> When the model developer visualizes the data, he swears that the
>> >> GIF I've attached is the correct layout... a version seemingly
>> >> rotated
>> >> 180 degrees along the Equatorial axis.
>> >>
>> >> I'm not sure what to believe, but I'm wondering now if
>> >> plot_data_plane (or MET for that matter) assumes anything
>> >> regarding the north-south orientation of the data. I can't
>> >> imagine that it does... but I thought I'd mention it just the
same.
>> >>
>> >> The observation and config file have transferred... just waiting
>> >> on the ~370MB model file to finish.
>> >>
>> >> -----Original Message-----
>> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> >> Sent: Wednesday, October 15, 2014 12:28 PM
>> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >>
>> >> Matt,
>> >>
>> >> OK, sounds good. Just let me know when the transfer is complete
>> >> and I'll go grab the data.
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
>> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>
>> >> >
>> >> > Well the first two plots look just fine, but the third won't
work:
>> >> >
>> >> > DEBUG 2: Retrieving grid from file:
>> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410
>> >> > 030
>> >> > 0.g
>> >> > rib
>> >> > DEBUG 1: Opening netCDF file:
>> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-MODIS_2014100300.nc
>> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
>> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
>> >> > message
>> >> > types: ALL DEBUG 1: Creating postscript
>> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0 locations.
>> >> > DEBUG 2: Skipped 64800 locations off the grid.
>> >> >
>> >> > Since the second file looks fine, that implies the NetCDF is
>> >> > formatted right. The GriB data seems okay too. But the pair
>> together are not.
>> >> > That's what leads me to question if the GriB header in the
model
>> >> > GriB file is a problem. Does MET take its dimensions
literally?
>> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and never
>> >> > get there because the end longitude is less than the start?
>> >> >
>> >> > It's probably time to FTP the files. I'll push model and
>> >> > observation from the above case along with config. I'll let
you
>> >> > know when they're done... I suspect it will take a couple
hours
>> >> > given the FTP speed last time we did this.
>> >> >
>> >> > Thanks,
>> >> > Matt
>> >> >
>> >> > -----Original Message-----
>> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> >> > Sent: Wednesday, October 15, 2014 11:39 AM
>> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
>> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>> >> >
>> >> > Matt,
>> >> >
>> >> > It's possible that this issue is caused by the fact that it's
a
>> >> > global grid. Please try the following...
>> >> >
>> >> > (1) Run plot_data_plane to make sure MET is reading the
gridded
>> >> > data the way you expect:
>> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
>> level="L0";'
>> >> > And then look at that image (adovis.ps) to make sure it
>> >> > looks
>> >> good.
>> >> >
>> >> > (2) Run plot_point_obs twice to see where MET is plotting the
>> >> observations:
>> >> > plot_point_obs obs.nc obs.ps
>> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
>> >> > The first call plots all points on a default global grid.
>> >> > The second call plots all points, but using the grid it reads
>> >> > from
>> >> model.grb file.
>> >> >
>> >> > Just look at the resulting images and see if anything jumps
out
>> >> > at
>> you.
>> >> > Perhaps you accidentally transposed the lat/lon values in the
>> >> > NetCDF point observation file? Or perhaps it's something to
do
>> >> > with the range of longitude values you used? Or something
else
>> >> > I'm not thinking
>> >> of.
>> >> >
>> >> > If the problem doesn't become obvious after those steps,
please
>> >> > send me a sample forecast grib file, NetCDF observation file,
>> >> > and Point-Stat config file.
>> >> >
>> >> > Thanks,
>> >> > John Halley Gotway
>> >> > met_help at ucar.edu
>> >> >
>> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
>> >> > AFWA
>> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
>> >> >
>> >> > >
>> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
>> >> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>> >> > > Queue: met_help
>> >> > > Subject: point_stat issue
>> >> > > Owner: Nobody
>> >> > > Requestors: matthew.sittel.ctr at us.af.mil
>> >> > > Status: new
>> >> > > Ticket <URL:
>> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>> >> > > >
>> >> > >
>> >> > >
>> >> > > Hi,
>> >> > >
>> >> > > Here's my newest and latest headache using MET v4.1.
>> >> > >
>> >> > > I'm trying to verify a gridded field using point_stat. The
>> >> > > observations are formatted as NetCDF. MET appears to read
in
>> >> > > everything without a problem but, as far as I can tell, MET
>> >> > > throws all the observations out so there's nothing left.
>> >> > > Here's
>> what I get:
>> >> > >
>> >> > > =================
>> >> > > /home/qcteam/METv4.1/bin/point_stat
>> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
>> >> > > 100
>> >> > > 300 .g rib
>> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
>> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cf
>> >> > > g -log ./point_stat_log.txt -outdir
>> >> > > /home/qcteam/METv4.1/data/aod/stats -v
>> >> > > 4 DEBUG 1: Default Config File:
>> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
>> >> > > DEBUG 1: User Config File:
>> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cf
>> >> > > g DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
>> >> > > created new Met2dDataFile object of type "FileType_Gb1".
>> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>> >> > > object of type "FileType_Gb1".
>> >> > > GSL_RNG_TYPE=mt19937
>> >> > > GSL_RNG_SEED=1
>> >> > > DEBUG 1: Forecast File:
>> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
>> >> > > 100
>> >> > > 300
>> >> > > .g
>> >> > > rib
>> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation File:
>> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
--------------------------------------------------------------
>> >> > > ---
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Reading data for U-GWD/L0.
>> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range
>> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB file
>> >> > >
>> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
>> >> > 003
>> >> > 00
>> >> .grib".
>> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
>> >> > > records matching VarInfo "U-GWD/L0" in GRIB file
>> >> > >
>> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
>> >> > 003
>> >> > 00
>> >> .grib".
>> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
>> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
>> >> > > climatology
>> >> levels.
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
--------------------------------------------------------------
>> >> > > ---
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
--------------------------------------------------------------
>> >> > > ---
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
observation
>> >> > > type ADPSFC, over region FULL, for interpolation method
>> >> > > BILIN(4), using 0
>> >> > pairs.
>> >> > > DEBUG 3: Number of matched pairs = 0
>> >> > > DEBUG 3: Observations processed = 64800
>> >> > > DEBUG 3: Rejected: GRIB code = 0
>> >> > > DEBUG 3: Rejected: valid time = 0
>> >> > > DEBUG 3: Rejected: bad obs value = 43229
>> >> > > DEBUG 3: Rejected: off the grid = 21571
>> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
>> >> > > quality marker = 0
>> >> > > DEBUG 3: Rejected: message type = 0
>> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected: bad
>> >> > > fcst value = 0
>> >> > > DEBUG 3: Rejected: duplicates = 0
>> >> > > DEBUG 2:
>> >> > > DEBUG 2:
>> >> > >
--------------------------------------------------------------
>> >> > > ---
>> >> > > ---
>> >> > > --
>> >> > > ----------
>> >> > > DEBUG 2:
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V.stat
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_fho.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_ctc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_cts.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mctc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mcts.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_cnt.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_sl1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_sal1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_vl1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_val1l2.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pct.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pstd.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_pjc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_prc.txt
>> >> > > DEBUG 1: Output file:
>> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
>> >> > > 3_0
>> >> > > 000
>> >> > > 00
>> >> > > V_mpr.txt
>> >> > > ============
>> >> > >
>> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
>> >> > > The observation data cover the entire global domain, as does
>> >> > > the
>> model.
>> >> > > There shouldn't be any 'off the grid' data. Here's the
model
>> >> > > data information, courtesy degrib.
>> >> > >
>> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147 kpds6=200
>> >> > > kpds7=0
>> >> > > levels=(0,0) grid=255 atmos col anl:
>> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
>> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
>> >> > > num_in_ave
>> >> > > 0 missing 0
>> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
>> winds(grid)
>> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
>> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
180)
>> >> > > scan 0 mode
>> >> > > 136 bdsgrid 1
>> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
>> >> > > DecScale
>> >> > > 6 BinScale 0
>> >> > >
>> >> > > Is it possible that MET is upset by the definition of
>> >> > > longitude as running from 0 to -0.5 in positive steps of
>> >> > > 1.003, which it will never reach? I'm not sure how MET
>> >> > > establishes the model grid; I presume through the GriB
header.
>> >> > >
>> >> > > I would appreciate any thoughts you have on this. I can FTP
>> >> > > the files as before if that will help.
>> >> > >
>> >> > > Thanks,
>> >> > > Matt
>> >> > >
>> >> > > // SIGNED //
>> >> > > Matthew C. Sittel
>> >> > > Associate Scientist
>> >> > > University Corporation for Atmospheric Research 16WS/WXN,
>> >> > > Offutt AFB, NE
>> >> > > (402) 294-3473 DSN: 271-3473
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Thu Oct 16 12:24:54 2014
Matt,
That's right. GRIB files do not include actual values for latitude
and
longitude, as NetCDF files often do. The lat/lon values are computed
using
the grid definition information contained in the GDS (grid description
section) of each GRIB record.
John
On Thu, Oct 16, 2014 at 12:15 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Hi John,
>
> Since I'm not the modeler, I don't know all the details, but what I
think
> is happening is that the header is being set wrong. There are no
LAT or
> LON fields in the GriB file to check it against. I don't think the
lat and
> lon corner points in the header are automatically set. He's trying
one
> more run... maybe that's all this is. I'll let you know.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 16, 2014 12:28 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> I've attached two updated images.
>
> "aodvis_64.png" shows how MET reads the data when the scanning mode
flag
> is set to 64. The plot I sent yesterday was wrong... I manually
changed
> the scan flag in the code to 64 in one spot but missed it in a
second spot.
> Sorry for the confusion.
>
> "aodvis_64_shift_180.png" shows how MET reads the data when the
scanning
> mode flag is set to 64 and the lower-left longitude is set to -180,
instead
> of 0.
>
> Looks like the second image orients the data correctly and put it on
the
> correct place on the earth. So perhaps the scanning mode flag and
the grid
> definition are incorrect?
>
> Thanks,
> John
>
> On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Matt,
> >
> > Would you be able to send me a plot that shows how the data should
be
> > correctly oriented? I'm not familiar with the xconv tool and we
don't
> > have it on our system.
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >>
> >> Hi John,
> >>
> >> Feedback from the modeler:
> >>
> >> "Well their plot is still wrong. The area that has the high
> >> concentration of dust (the dark red spot) is Northern Africa. So
the
> >> map needs to be rotated 180 degrees. Which I don't understand why
> >> their image isn’t doing it properly, doesn’t the girb file give
> >> lat/lon of each value? I don’t get why other graphic utilities
like
> >> xconv have no issues with the grid. Also the idv data is
actually
> >> correct map wise (i.e. it is flipped the right way, just the
location
> is off (again needs to be rotated 180 degrees)"
> >>
> >> The model data is built from software sent by NCAR, so it's not
> >> in-house. The header information looks to be straightforward to
adjust.
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, October 15, 2014 2:41 PM
> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >>
> >> Matt,
> >>
> >> The problem is how the "scanning mode flag" is set in the GRIB1
file
> >> you're using. It is currently set to 0, which in bits is
"00000000".
> >> Here's a
> >> GRIB1 table that describes this flag:
> >> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
> >>
> >> Notice the following in your wgrib output:
> >> scan: WE:NS winds(grid)
> >>
> >> All zeros means that the data is stored in the +x and -y
directions.
> >> The grid definition includes lat1 and lat2 values. When data is
in
> >> the -y direction, we use lat2 to define the latitude of the
lower-left
> corner.
> >> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
the
> >> latitude of the lower-left corner to 90 (when you really want it
to
> >> be -90). The best fix would be for the creator of this GRIB1
file to
> >> set that flag in a way that's consistent with the grid
definition.
> >>
> >> For testing, I added the following after line 196 of the file
> >> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> >> data.lat_ll =
min(decode_lat_lon(gds.grid_type.latlon_grid.lat1, 3),
> >>
> >> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> >> 3));
> >>
> >> This just forces it to use the minimum of the two latitudes for
the
> >> lower-left corner.
> >>
> >> That results in the attached image (aodvis_from_MET.png). Since
the
> >> map data is on there, MET knows where it is on the earth now.
> >>
> >> ***BUT*** when I plot the same dataset using IDV, I get the
attached
> >> image (aodvis_from_IDV.png). Notice how they are flipped top to
bottom.
> >>
> >> I've always found this scanning mode flag to be pretty tedious.
It
> >> enables you to pack the data in whatever order you prefer, but
that
> >> just makes the code to read the data more complex and causes
issues
> >> like this to pop up.
> >>
> >> So there's an inconsistency between the scanning mode flag and
the
> >> grid
> >> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
the
> >> scanning mode flag needs to be set differently. My guess is that
> >> it's the scanning mode flag, because one would naturally assume
that
> >> 0 is a good default value - but it's not. The most common
scanning
> >> mode I see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so
that
> >> means the 2nd bit is 1, from most-to-least significant bits).
And
> >> that means data is stored in the
> >> +x and +y directions.
> >>
> >> My suggestion is to have the creator of this GRIB file set the
> >> scanning mode flag to 64. I manually did that in the MET code,
> >> removed the data.lat_ll line I mentioned above and reran
> >> plot_data_plane. It resulted in the same aodvis_from_met.png
image
> that I attached.
> >>
> >> But you guys need to decide which way is "up" in your data!
That's
> >> easiest if you have any fields whose values correspond well with
land.
> >>
> >> As a side note, if you to send me a single record from a larger
GRIB
> >> file, you can extract using wgrib like this (use -d to pick the
> >> record
> >> number):
> >> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
> >>
> >> Hope that helps.
> >>
> >> John
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
> >> <johnhg at ucar.edu>
> >> wrote:
> >>
> >> > Matt,
> >> >
> >> > When I see aodvis.pdf, I do see an obvious problem. There are
no
> >> > coastlines or map data. So MET is able to read the 2D data
fine,
> >> > but it just doesn't know where on earth to place it. So
there's a
> >> > problem with our grid definition logic. I'll take a closer
look
> >> > and try to
> >> nail it down.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF
AFWA
> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>
> >> >>
> >> >> While we wait... here's a curiosity that might be noteworthy,
or
> not...
> >> >>
> >> >> When I plot the model GriB data, specifically the field
kpds5=147
> >> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> >> >>
> >> >> When the model developer visualizes the data, he swears that
the
> >> >> GIF I've attached is the correct layout... a version seemingly
> >> >> rotated
> >> >> 180 degrees along the Equatorial axis.
> >> >>
> >> >> I'm not sure what to believe, but I'm wondering now if
> >> >> plot_data_plane (or MET for that matter) assumes anything
> >> >> regarding the north-south orientation of the data. I can't
> >> >> imagine that it does... but I thought I'd mention it just the
same.
> >> >>
> >> >> The observation and config file have transferred... just
waiting
> >> >> on the ~370MB model file to finish.
> >> >>
> >> >> -----Original Message-----
> >> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> >> Sent: Wednesday, October 15, 2014 12:28 PM
> >> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >>
> >> >> Matt,
> >> >>
> >> >> OK, sounds good. Just let me know when the transfer is
complete
> >> >> and I'll go grab the data.
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> >> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >>
> >> >> >
> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >> >> >
> >> >> > Well the first two plots look just fine, but the third won't
work:
> >> >> >
> >> >> > DEBUG 2: Retrieving grid from file:
> >> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201410
> >> >> > 030
> >> >> > 0.g
> >> >> > rib
> >> >> > DEBUG 1: Opening netCDF file:
> >> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> >> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
> >> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
> >> >> > message
> >> >> > types: ALL DEBUG 1: Creating postscript
> >> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0
locations.
> >> >> > DEBUG 2: Skipped 64800 locations off the grid.
> >> >> >
> >> >> > Since the second file looks fine, that implies the NetCDF is
> >> >> > formatted right. The GriB data seems okay too. But the
pair
> >> together are not.
> >> >> > That's what leads me to question if the GriB header in the
model
> >> >> > GriB file is a problem. Does MET take its dimensions
literally?
> >> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and
never
> >> >> > get there because the end longitude is less than the start?
> >> >> >
> >> >> > It's probably time to FTP the files. I'll push model and
> >> >> > observation from the above case along with config. I'll let
you
> >> >> > know when they're done... I suspect it will take a couple
hours
> >> >> > given the FTP speed last time we did this.
> >> >> >
> >> >> > Thanks,
> >> >> > Matt
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> >> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >> >
> >> >> > Matt,
> >> >> >
> >> >> > It's possible that this issue is caused by the fact that
it's a
> >> >> > global grid. Please try the following...
> >> >> >
> >> >> > (1) Run plot_data_plane to make sure MET is reading the
gridded
> >> >> > data the way you expect:
> >> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
> >> level="L0";'
> >> >> > And then look at that image (adovis.ps) to make sure it
> >> >> > looks
> >> >> good.
> >> >> >
> >> >> > (2) Run plot_point_obs twice to see where MET is plotting
the
> >> >> observations:
> >> >> > plot_point_obs obs.nc obs.ps
> >> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> >> >> > The first call plots all points on a default global
grid.
> >> >> > The second call plots all points, but using the grid it
reads
> >> >> > from
> >> >> model.grb file.
> >> >> >
> >> >> > Just look at the resulting images and see if anything jumps
out
> >> >> > at
> >> you.
> >> >> > Perhaps you accidentally transposed the lat/lon values in
the
> >> >> > NetCDF point observation file? Or perhaps it's something to
do
> >> >> > with the range of longitude values you used? Or something
else
> >> >> > I'm not thinking
> >> >> of.
> >> >> >
> >> >> > If the problem doesn't become obvious after those steps,
please
> >> >> > send me a sample forecast grib file, NetCDF observation
file,
> >> >> > and Point-Stat config file.
> >> >> >
> >> >> > Thanks,
> >> >> > John Halley Gotway
> >> >> > met_help at ucar.edu
> >> >> >
> >> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> >> >> > AFWA
> >> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >> >
> >> >> > >
> >> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> >> >> > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> >> >> > > Queue: met_help
> >> >> > > Subject: point_stat issue
> >> >> > > Owner: Nobody
> >> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> >> >> > > Status: new
> >> >> > > Ticket <URL:
> >> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > Hi,
> >> >> > >
> >> >> > > Here's my newest and latest headache using MET v4.1.
> >> >> > >
> >> >> > > I'm trying to verify a gridded field using point_stat.
The
> >> >> > > observations are formatted as NetCDF. MET appears to read
in
> >> >> > > everything without a problem but, as far as I can tell,
MET
> >> >> > > throws all the observations out so there's nothing left.
> >> >> > > Here's
> >> what I get:
> >> >> > >
> >> >> > > =================
> >> >> > > /home/qcteam/METv4.1/bin/point_stat
> >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
> >> >> > > 100
> >> >> > > 300 .g rib
> >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cf
> >> >> > > g -log ./point_stat_log.txt -outdir
> >> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> >> >> > > 4 DEBUG 1: Default Config File:
> >> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> >> >> > > DEBUG 1: User Config File:
> >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.cf
> >> >> > > g DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> >> >> > > created new Met2dDataFile object of type "FileType_Gb1".
> >> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> >> >> > > object of type "FileType_Gb1".
> >> >> > > GSL_RNG_TYPE=mt19937
> >> >> > > GSL_RNG_SEED=1
> >> >> > > DEBUG 1: Forecast File:
> >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
> >> >> > > 100
> >> >> > > 300
> >> >> > > .g
> >> >> > > rib
> >> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation File:
> >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
--------------------------------------------------------------
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Reading data for U-GWD/L0.
> >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found
range
> >> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB
file
> >> >> > >
> >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
> >> >> > 003
> >> >> > 00
> >> >> .grib".
> >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
> >> >> > > records matching VarInfo "U-GWD/L0" in GRIB file
> >> >> > >
> >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20141
> >> >> > 003
> >> >> > 00
> >> >> .grib".
> >> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> >> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
> >> >> > > climatology
> >> >> levels.
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
--------------------------------------------------------------
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
--------------------------------------------------------------
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
observation
> >> >> > > type ADPSFC, over region FULL, for interpolation method
> >> >> > > BILIN(4), using 0
> >> >> > pairs.
> >> >> > > DEBUG 3: Number of matched pairs = 0
> >> >> > > DEBUG 3: Observations processed = 64800
> >> >> > > DEBUG 3: Rejected: GRIB code = 0
> >> >> > > DEBUG 3: Rejected: valid time = 0
> >> >> > > DEBUG 3: Rejected: bad obs value = 43229
> >> >> > > DEBUG 3: Rejected: off the grid = 21571
> >> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
> >> >> > > quality marker = 0
> >> >> > > DEBUG 3: Rejected: message type = 0
> >> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected:
bad
> >> >> > > fcst value = 0
> >> >> > > DEBUG 3: Rejected: duplicates = 0
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
--------------------------------------------------------------
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V.stat
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_fho.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_ctc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_cts.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mctc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mcts.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_cnt.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_sl1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_sal1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_vl1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_val1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pct.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pstd.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pjc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_prc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_2014100
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mpr.txt
> >> >> > > ============
> >> >> > >
> >> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> >> >> > > The observation data cover the entire global domain, as
does
> >> >> > > the
> >> model.
> >> >> > > There shouldn't be any 'off the grid' data. Here's the
model
> >> >> > > data information, courtesy degrib.
> >> >> > >
> >> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147
kpds6=200
> >> >> > > kpds7=0
> >> >> > > levels=(0,0) grid=255 atmos col anl:
> >> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> >> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
> >> >> > > num_in_ave
> >> >> > > 0 missing 0
> >> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
> >> winds(grid)
> >> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> >> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
180)
> >> >> > > scan 0 mode
> >> >> > > 136 bdsgrid 1
> >> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
> >> >> > > DecScale
> >> >> > > 6 BinScale 0
> >> >> > >
> >> >> > > Is it possible that MET is upset by the definition of
> >> >> > > longitude as running from 0 to -0.5 in positive steps of
> >> >> > > 1.003, which it will never reach? I'm not sure how MET
> >> >> > > establishes the model grid; I presume through the GriB
header.
> >> >> > >
> >> >> > > I would appreciate any thoughts you have on this. I can
FTP
> >> >> > > the files as before if that will help.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Matt
> >> >> > >
> >> >> > > // SIGNED //
> >> >> > > Matthew C. Sittel
> >> >> > > Associate Scientist
> >> >> > > University Corporation for Atmospheric Research 16WS/WXN,
> >> >> > > Offutt AFB, NE
> >> >> > > (402) 294-3473 DSN: 271-3473
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Oct 16 12:28:21 2014
But you can generate them; we typically ask the ensembles guys to make
sure there are included in any GriB file. Now how those are generated
I'm not sure... but we read them when we need to, such as in this WRF
file:
rec 108:3165146:date 2014101200 LAT kpds5=230 kpds6=1 kpds7=0
levels=(0,0) grid=255 sfc 48hr fcst:
LAT= Latitude [deg]
timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3 num_in_ave
0 missing 0
center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
min/max data 5.0097 67.3016 num bits 20 BDS_Ref 50097 DecScale 4
BinScale 0
rec 109:3231984:date 2014101200 LON kpds5=231 kpds6=1 kpds7=0
levels=(0,0) grid=255 sfc 48hr fcst:
LON= Longitude [deg]
timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3 num_in_ave
0 missing 0
center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
min/max data -165.726 -26.2738 num bits 21 BDS_Ref -1.65726e+06
DecScale 4 BinScale 0
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 16, 2014 1:25 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Matt,
That's right. GRIB files do not include actual values for latitude
and longitude, as NetCDF files often do. The lat/lon values are
computed using the grid definition information contained in the GDS
(grid description
section) of each GRIB record.
John
On Thu, Oct 16, 2014 at 12:15 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> Hi John,
>
> Since I'm not the modeler, I don't know all the details, but what I
> think is happening is that the header is being set wrong. There are
> no LAT or LON fields in the GriB file to check it against. I don't
> think the lat and lon corner points in the header are automatically
> set. He's trying one more run... maybe that's all this is. I'll
let you know.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 16, 2014 12:28 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> I've attached two updated images.
>
> "aodvis_64.png" shows how MET reads the data when the scanning mode
> flag is set to 64. The plot I sent yesterday was wrong... I
manually
> changed the scan flag in the code to 64 in one spot but missed it in
a second spot.
> Sorry for the confusion.
>
> "aodvis_64_shift_180.png" shows how MET reads the data when the
> scanning mode flag is set to 64 and the lower-left longitude is set
to
> -180, instead of 0.
>
> Looks like the second image orients the data correctly and put it on
> the correct place on the earth. So perhaps the scanning mode flag
and
> the grid definition are incorrect?
>
> Thanks,
> John
>
> On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Matt,
> >
> > Would you be able to send me a plot that shows how the data should
> > be correctly oriented? I'm not familiar with the xconv tool and
we
> > don't have it on our system.
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >>
> >> Hi John,
> >>
> >> Feedback from the modeler:
> >>
> >> "Well their plot is still wrong. The area that has the high
> >> concentration of dust (the dark red spot) is Northern Africa. So
> >> the map needs to be rotated 180 degrees. Which I don't understand
> >> why their image isn’t doing it properly, doesn’t the girb file
give
> >> lat/lon of each value? I don’t get why other graphic utilities
like
> >> xconv have no issues with the grid. Also the idv data is
actually
> >> correct map wise (i.e. it is flipped the right way, just the
> >> location
> is off (again needs to be rotated 180 degrees)"
> >>
> >> The model data is built from software sent by NCAR, so it's not
> >> in-house. The header information looks to be straightforward to
adjust.
> >>
> >> -----Original Message-----
> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> Sent: Wednesday, October 15, 2014 2:41 PM
> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >>
> >> Matt,
> >>
> >> The problem is how the "scanning mode flag" is set in the GRIB1
> >> file you're using. It is currently set to 0, which in bits is
"00000000".
> >> Here's a
> >> GRIB1 table that describes this flag:
> >> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
> >>
> >> Notice the following in your wgrib output:
> >> scan: WE:NS winds(grid)
> >>
> >> All zeros means that the data is stored in the +x and -y
directions.
> >> The grid definition includes lat1 and lat2 values. When data is
in
> >> the -y direction, we use lat2 to define the latitude of the
> >> lower-left
> corner.
> >> But in your GRIB file lat1 = -90 and lat2 = 90. So we're setting
> >> the latitude of the lower-left corner to 90 (when you really want
> >> it to be -90). The best fix would be for the creator of this
GRIB1
> >> file to set that flag in a way that's consistent with the grid
definition.
> >>
> >> For testing, I added the following after line 196 of the file
> >> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> >> data.lat_ll =
min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
> >> 3),
> >>
> >> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> >> 3));
> >>
> >> This just forces it to use the minimum of the two latitudes for
the
> >> lower-left corner.
> >>
> >> That results in the attached image (aodvis_from_MET.png). Since
> >> the map data is on there, MET knows where it is on the earth now.
> >>
> >> ***BUT*** when I plot the same dataset using IDV, I get the
> >> attached image (aodvis_from_IDV.png). Notice how they are
flipped top to bottom.
> >>
> >> I've always found this scanning mode flag to be pretty tedious.
It
> >> enables you to pack the data in whatever order you prefer, but
that
> >> just makes the code to read the data more complex and causes
issues
> >> like this to pop up.
> >>
> >> So there's an inconsistency between the scanning mode flag and
the
> >> grid
> >> lat1 and lat2 values. Either lat1 and lat2 need to be swapped or
> >> the scanning mode flag needs to be set differently. My guess is
> >> that it's the scanning mode flag, because one would naturally
> >> assume that
> >> 0 is a good default value - but it's not. The most common
scanning
> >> mode I see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so
that
> >> means the 2nd bit is 1, from most-to-least significant bits).
And
> >> that means data is stored in the
> >> +x and +y directions.
> >>
> >> My suggestion is to have the creator of this GRIB file set the
> >> scanning mode flag to 64. I manually did that in the MET code,
> >> removed the data.lat_ll line I mentioned above and reran
> >> plot_data_plane. It resulted in the same aodvis_from_met.png
image
> that I attached.
> >>
> >> But you guys need to decide which way is "up" in your data!
That's
> >> easiest if you have any fields whose values correspond well with
land.
> >>
> >> As a side note, if you to send me a single record from a larger
> >> GRIB file, you can extract using wgrib like this (use -d to pick
> >> the record
> >> number):
> >> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
> >>
> >> Hope that helps.
> >>
> >> John
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
> >> <johnhg at ucar.edu>
> >> wrote:
> >>
> >> > Matt,
> >> >
> >> > When I see aodvis.pdf, I do see an obvious problem. There are
no
> >> > coastlines or map data. So MET is able to read the 2D data
fine,
> >> > but it just doesn't know where on earth to place it. So
there's
> >> > a problem with our grid definition logic. I'll take a closer
> >> > look and try to
> >> nail it down.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF
AFWA
> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
>
> >> >>
> >> >> While we wait... here's a curiosity that might be noteworthy,
or
> not...
> >> >>
> >> >> When I plot the model GriB data, specifically the field
> >> >> kpds5=147
> >> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> >> >>
> >> >> When the model developer visualizes the data, he swears that
the
> >> >> GIF I've attached is the correct layout... a version seemingly
> >> >> rotated
> >> >> 180 degrees along the Equatorial axis.
> >> >>
> >> >> I'm not sure what to believe, but I'm wondering now if
> >> >> plot_data_plane (or MET for that matter) assumes anything
> >> >> regarding the north-south orientation of the data. I can't
> >> >> imagine that it does... but I thought I'd mention it just the
same.
> >> >>
> >> >> The observation and config file have transferred... just
waiting
> >> >> on the ~370MB model file to finish.
> >> >>
> >> >> -----Original Message-----
> >> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> >> Sent: Wednesday, October 15, 2014 12:28 PM
> >> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >>
> >> >> Matt,
> >> >>
> >> >> OK, sounds good. Just let me know when the transfer is
complete
> >> >> and I'll go grab the data.
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
> >> >> AFWA
> >> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >>
> >> >> >
> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >> >> > >
> >> >> >
> >> >> > Well the first two plots look just fine, but the third won't
work:
> >> >> >
> >> >> > DEBUG 2: Retrieving grid from file:
> >> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
> >> >> > 10
> >> >> > 030
> >> >> > 0.g
> >> >> > rib
> >> >> > DEBUG 1: Opening netCDF file:
> >> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> >> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
> >> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
> >> >> > message
> >> >> > types: ALL DEBUG 1: Creating postscript
> >> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0
locations.
> >> >> > DEBUG 2: Skipped 64800 locations off the grid.
> >> >> >
> >> >> > Since the second file looks fine, that implies the NetCDF is
> >> >> > formatted right. The GriB data seems okay too. But the
pair
> >> together are not.
> >> >> > That's what leads me to question if the GriB header in the
> >> >> > model GriB file is a problem. Does MET take its dimensions
literally?
> >> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and
never
> >> >> > get there because the end longitude is less than the start?
> >> >> >
> >> >> > It's probably time to FTP the files. I'll push model and
> >> >> > observation from the above case along with config. I'll let
> >> >> > you know when they're done... I suspect it will take a
couple
> >> >> > hours given the FTP speed last time we did this.
> >> >> >
> >> >> > Thanks,
> >> >> > Matt
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> >> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> >> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> >> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >> >> >
> >> >> > Matt,
> >> >> >
> >> >> > It's possible that this issue is caused by the fact that
it's
> >> >> > a global grid. Please try the following...
> >> >> >
> >> >> > (1) Run plot_data_plane to make sure MET is reading the
> >> >> > gridded data the way you expect:
> >> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
> >> level="L0";'
> >> >> > And then look at that image (adovis.ps) to make sure it
> >> >> > looks
> >> >> good.
> >> >> >
> >> >> > (2) Run plot_point_obs twice to see where MET is plotting
the
> >> >> observations:
> >> >> > plot_point_obs obs.nc obs.ps
> >> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> >> >> > The first call plots all points on a default global
grid.
> >> >> > The second call plots all points, but using the grid it
reads
> >> >> > from
> >> >> model.grb file.
> >> >> >
> >> >> > Just look at the resulting images and see if anything jumps
> >> >> > out at
> >> you.
> >> >> > Perhaps you accidentally transposed the lat/lon values in
the
> >> >> > NetCDF point observation file? Or perhaps it's something to
> >> >> > do with the range of longitude values you used? Or
something
> >> >> > else I'm not thinking
> >> >> of.
> >> >> >
> >> >> > If the problem doesn't become obvious after those steps,
> >> >> > please send me a sample forecast grib file, NetCDF
observation
> >> >> > file, and Point-Stat config file.
> >> >> >
> >> >> > Thanks,
> >> >> > John Halley Gotway
> >> >> > met_help at ucar.edu
> >> >> >
> >> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> >> >> > AFWA
> >> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> >> >> >
> >> >> > >
> >> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> >> >> > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> >> >> > > Queue: met_help
> >> >> > > Subject: point_stat issue
> >> >> > > Owner: Nobody
> >> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> >> >> > > Status: new
> >> >> > > Ticket <URL:
> >> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > Hi,
> >> >> > >
> >> >> > > Here's my newest and latest headache using MET v4.1.
> >> >> > >
> >> >> > > I'm trying to verify a gridded field using point_stat.
The
> >> >> > > observations are formatted as NetCDF. MET appears to read
> >> >> > > in everything without a problem but, as far as I can tell,
> >> >> > > MET throws all the observations out so there's nothing
left.
> >> >> > > Here's
> >> what I get:
> >> >> > >
> >> >> > > =================
> >> >> > > /home/qcteam/METv4.1/bin/point_stat
> >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20
> >> >> > > 14
> >> >> > > 100
> >> >> > > 300 .g rib
> >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.n
> >> >> > > c
> >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> >> >> > > cf g -log ./point_stat_log.txt -outdir
> >> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> >> >> > > 4 DEBUG 1: Default Config File:
> >> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> >> >> > > DEBUG 1: User Config File:
> >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> >> >> > > cf g DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file()
> >> >> > > -> created new Met2dDataFile object of type
"FileType_Gb1".
> >> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
> >> >> > > VarInfo object of type "FileType_Gb1".
> >> >> > > GSL_RNG_TYPE=mt19937
> >> >> > > GSL_RNG_SEED=1
> >> >> > > DEBUG 1: Forecast File:
> >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20
> >> >> > > 14
> >> >> > > 100
> >> >> > > 300
> >> >> > > .g
> >> >> > > rib
> >> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation File:
> >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.n
> >> >> > > c
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
------------------------------------------------------------
> >> >> > > --
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Reading data for U-GWD/L0.
> >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found
range
> >> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB
file
> >> >> > >
> >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201
> >> >> > 41
> >> >> > 003
> >> >> > 00
> >> >> .grib".
> >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
> >> >> > > GRIB records matching VarInfo "U-GWD/L0" in GRIB file
> >> >> > >
> >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201
> >> >> > 41
> >> >> > 003
> >> >> > 00
> >> >> .grib".
> >> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> >> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
> >> >> > > climatology
> >> >> levels.
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
------------------------------------------------------------
> >> >> > > --
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Searching 64800 observations from 64800 messages.
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
------------------------------------------------------------
> >> >> > > --
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
> >> >> > > observation type ADPSFC, over region FULL, for
interpolation
> >> >> > > method BILIN(4), using 0
> >> >> > pairs.
> >> >> > > DEBUG 3: Number of matched pairs = 0
> >> >> > > DEBUG 3: Observations processed = 64800
> >> >> > > DEBUG 3: Rejected: GRIB code = 0
> >> >> > > DEBUG 3: Rejected: valid time = 0
> >> >> > > DEBUG 3: Rejected: bad obs value = 43229
> >> >> > > DEBUG 3: Rejected: off the grid = 21571
> >> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
> >> >> > > quality marker = 0
> >> >> > > DEBUG 3: Rejected: message type = 0
> >> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected:
bad
> >> >> > > fcst value = 0
> >> >> > > DEBUG 3: Rejected: duplicates = 0
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 2:
> >> >> > >
------------------------------------------------------------
> >> >> > > --
> >> >> > > ---
> >> >> > > ---
> >> >> > > --
> >> >> > > ----------
> >> >> > > DEBUG 2:
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V.stat
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_fho.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_ctc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_cts.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mctc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mcts.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_cnt.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_sl1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_sal1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_vl1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_val1l2.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pct.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pstd.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_pjc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_prc.txt
> >> >> > > DEBUG 1: Output file:
> >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> >> >> > > 00
> >> >> > > 3_0
> >> >> > > 000
> >> >> > > 00
> >> >> > > V_mpr.txt
> >> >> > > ============
> >> >> > >
> >> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> >> >> > > The observation data cover the entire global domain, as
does
> >> >> > > the
> >> model.
> >> >> > > There shouldn't be any 'off the grid' data. Here's the
> >> >> > > model data information, courtesy degrib.
> >> >> > >
> >> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147
kpds6=200
> >> >> > > kpds7=0
> >> >> > > levels=(0,0) grid=255 atmos col anl:
> >> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> >> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid 0
> >> >> > > num_in_ave
> >> >> > > 0 missing 0
> >> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
> >> winds(grid)
> >> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> >> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
> >> >> > > 180) scan 0 mode
> >> >> > > 136 bdsgrid 1
> >> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref 9945
> >> >> > > DecScale
> >> >> > > 6 BinScale 0
> >> >> > >
> >> >> > > Is it possible that MET is upset by the definition of
> >> >> > > longitude as running from 0 to -0.5 in positive steps of
> >> >> > > 1.003, which it will never reach? I'm not sure how MET
> >> >> > > establishes the model grid; I presume through the GriB
header.
> >> >> > >
> >> >> > > I would appreciate any thoughts you have on this. I can
FTP
> >> >> > > the files as before if that will help.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Matt
> >> >> > >
> >> >> > > // SIGNED //
> >> >> > > Matthew C. Sittel
> >> >> > > Associate Scientist
> >> >> > > University Corporation for Atmospheric Research 16WS/WXN,
> >> >> > > Offutt AFB, NE
> >> >> > > (402) 294-3473 DSN: 271-3473
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
Subject: point_stat issue
From: John Halley Gotway
Time: Mon Oct 20 13:25:54 2014
Hello Matt,
Just wanted to check in. Is there any additional support you need
from me
on this issue?
Thanks,
John
On Thu, Oct 16, 2014 at 12:28 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> But you can generate them; we typically ask the ensembles guys to
make
> sure there are included in any GriB file. Now how those are
generated I'm
> not sure... but we read them when we need to, such as in this WRF
file:
>
> rec 108:3165146:date 2014101200 LAT kpds5=230 kpds6=1 kpds7=0
> levels=(0,0) grid=255 sfc 48hr fcst:
> LAT= Latitude [deg]
> timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3
num_in_ave 0
> missing 0
> center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
> Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
> Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
> North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
> min/max data 5.0097 67.3016 num bits 20 BDS_Ref 50097 DecScale
4
> BinScale 0
>
> rec 109:3231984:date 2014101200 LON kpds5=231 kpds6=1 kpds7=0
levels=(0,0)
> grid=255 sfc 48hr fcst:
> LON= Longitude [deg]
> timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3
num_in_ave 0
> missing 0
> center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
> Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
> Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
> North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
> min/max data -165.726 -26.2738 num bits 21 BDS_Ref -1.65726e+06
> DecScale 4 BinScale 0
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 16, 2014 1:25 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> That's right. GRIB files do not include actual values for latitude
and
> longitude, as NetCDF files often do. The lat/lon values are
computed using
> the grid definition information contained in the GDS (grid
description
> section) of each GRIB record.
>
> John
>
> On Thu, Oct 16, 2014 at 12:15 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >
> > Hi John,
> >
> > Since I'm not the modeler, I don't know all the details, but what
I
> > think is happening is that the header is being set wrong. There
are
> > no LAT or LON fields in the GriB file to check it against. I
don't
> > think the lat and lon corner points in the header are
automatically
> > set. He's trying one more run... maybe that's all this is. I'll
let
> you know.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, October 16, 2014 12:28 PM
> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >
> > Matt,
> >
> > I've attached two updated images.
> >
> > "aodvis_64.png" shows how MET reads the data when the scanning
mode
> > flag is set to 64. The plot I sent yesterday was wrong... I
manually
> > changed the scan flag in the code to 64 in one spot but missed it
in a
> second spot.
> > Sorry for the confusion.
> >
> > "aodvis_64_shift_180.png" shows how MET reads the data when the
> > scanning mode flag is set to 64 and the lower-left longitude is
set to
> > -180, instead of 0.
> >
> > Looks like the second image orients the data correctly and put it
on
> > the correct place on the earth. So perhaps the scanning mode flag
and
> > the grid definition are incorrect?
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Matt,
> > >
> > > Would you be able to send me a plot that shows how the data
should
> > > be correctly oriented? I'm not familiar with the xconv tool and
we
> > > don't have it on our system.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> > >>
> > >> Hi John,
> > >>
> > >> Feedback from the modeler:
> > >>
> > >> "Well their plot is still wrong. The area that has the high
> > >> concentration of dust (the dark red spot) is Northern Africa.
So
> > >> the map needs to be rotated 180 degrees. Which I don't
understand
> > >> why their image isn’t doing it properly, doesn’t the girb file
give
> > >> lat/lon of each value? I don’t get why other graphic utilities
like
> > >> xconv have no issues with the grid. Also the idv data is
actually
> > >> correct map wise (i.e. it is flipped the right way, just the
> > >> location
> > is off (again needs to be rotated 180 degrees)"
> > >>
> > >> The model data is built from software sent by NCAR, so it's not
> > >> in-house. The header information looks to be straightforward
to
> adjust.
> > >>
> > >> -----Original Message-----
> > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> Sent: Wednesday, October 15, 2014 2:41 PM
> > >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >>
> > >> Matt,
> > >>
> > >> The problem is how the "scanning mode flag" is set in the GRIB1
> > >> file you're using. It is currently set to 0, which in bits is
> "00000000".
> > >> Here's a
> > >> GRIB1 table that describes this flag:
> > >> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
> > >>
> > >> Notice the following in your wgrib output:
> > >> scan: WE:NS winds(grid)
> > >>
> > >> All zeros means that the data is stored in the +x and -y
directions.
> > >> The grid definition includes lat1 and lat2 values. When data
is in
> > >> the -y direction, we use lat2 to define the latitude of the
> > >> lower-left
> > corner.
> > >> But in your GRIB file lat1 = -90 and lat2 = 90. So we're
setting
> > >> the latitude of the lower-left corner to 90 (when you really
want
> > >> it to be -90). The best fix would be for the creator of this
GRIB1
> > >> file to set that flag in a way that's consistent with the grid
> definition.
> > >>
> > >> For testing, I added the following after line 196 of the file
> > >> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> > >> data.lat_ll =
min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
> > >> 3),
> > >>
> > >> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> > >> 3));
> > >>
> > >> This just forces it to use the minimum of the two latitudes for
the
> > >> lower-left corner.
> > >>
> > >> That results in the attached image (aodvis_from_MET.png).
Since
> > >> the map data is on there, MET knows where it is on the earth
now.
> > >>
> > >> ***BUT*** when I plot the same dataset using IDV, I get the
> > >> attached image (aodvis_from_IDV.png). Notice how they are
flipped
> top to bottom.
> > >>
> > >> I've always found this scanning mode flag to be pretty tedious.
It
> > >> enables you to pack the data in whatever order you prefer, but
that
> > >> just makes the code to read the data more complex and causes
issues
> > >> like this to pop up.
> > >>
> > >> So there's an inconsistency between the scanning mode flag and
the
> > >> grid
> > >> lat1 and lat2 values. Either lat1 and lat2 need to be swapped
or
> > >> the scanning mode flag needs to be set differently. My guess
is
> > >> that it's the scanning mode flag, because one would naturally
> > >> assume that
> > >> 0 is a good default value - but it's not. The most common
scanning
> > >> mode I see is 64, which in bits is 0100000 (i.e. 64 = 2^6, so
that
> > >> means the 2nd bit is 1, from most-to-least significant bits).
And
> > >> that means data is stored in the
> > >> +x and +y directions.
> > >>
> > >> My suggestion is to have the creator of this GRIB file set the
> > >> scanning mode flag to 64. I manually did that in the MET code,
> > >> removed the data.lat_ll line I mentioned above and reran
> > >> plot_data_plane. It resulted in the same aodvis_from_met.png
image
> > that I attached.
> > >>
> > >> But you guys need to decide which way is "up" in your data!
That's
> > >> easiest if you have any fields whose values correspond well
with land.
> > >>
> > >> As a side note, if you to send me a single record from a larger
> > >> GRIB file, you can extract using wgrib like this (use -d to
pick
> > >> the record
> > >> number):
> > >> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
> > >>
> > >> Hope that helps.
> > >>
> > >> John
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
> > >> <johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >> > Matt,
> > >> >
> > >> > When I see aodvis.pdf, I do see an obvious problem. There
are no
> > >> > coastlines or map data. So MET is able to read the 2D data
fine,
> > >> > but it just doesn't know where on earth to place it. So
there's
> > >> > a problem with our grid definition logic. I'll take a closer
> > >> > look and try to
> > >> nail it down.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> >
> > >> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF
AFWA
> > >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> > >> >>
> > >> >> While we wait... here's a curiosity that might be
noteworthy, or
> > not...
> > >> >>
> > >> >> When I plot the model GriB data, specifically the field
> > >> >> kpds5=147
> > >> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> > >> >>
> > >> >> When the model developer visualizes the data, he swears that
the
> > >> >> GIF I've attached is the correct layout... a version
seemingly
> > >> >> rotated
> > >> >> 180 degrees along the Equatorial axis.
> > >> >>
> > >> >> I'm not sure what to believe, but I'm wondering now if
> > >> >> plot_data_plane (or MET for that matter) assumes anything
> > >> >> regarding the north-south orientation of the data. I can't
> > >> >> imagine that it does... but I thought I'd mention it just
the same.
> > >> >>
> > >> >> The observation and config file have transferred... just
waiting
> > >> >> on the ~370MB model file to finish.
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> >> Sent: Wednesday, October 15, 2014 12:28 PM
> > >> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >> >>
> > >> >> Matt,
> > >> >>
> > >> >> OK, sounds good. Just let me know when the transfer is
complete
> > >> >> and I'll go grab the data.
> > >> >>
> > >> >> Thanks,
> > >> >> John
> > >> >>
> > >> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
> > >> >> AFWA
> > >> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >>
> > >> >> >
> > >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >> >> > >
> > >> >> >
> > >> >> > Well the first two plots look just fine, but the third
won't
> work:
> > >> >> >
> > >> >> > DEBUG 2: Retrieving grid from file:
> > >> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2014
> > >> >> > 10
> > >> >> > 030
> > >> >> > 0.g
> > >> >> > rib
> > >> >> > DEBUG 1: Opening netCDF file:
> > >> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.nc
> > >> >> > DEBUG 2: Processing 64800 observations at 64800 locations.
> > >> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
> > >> >> > message
> > >> >> > types: ALL DEBUG 1: Creating postscript
> > >> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0
locations.
> > >> >> > DEBUG 2: Skipped 64800 locations off the grid.
> > >> >> >
> > >> >> > Since the second file looks fine, that implies the NetCDF
is
> > >> >> > formatted right. The GriB data seems okay too. But the
pair
> > >> together are not.
> > >> >> > That's what leads me to question if the GriB header in the
> > >> >> > model GriB file is a problem. Does MET take its
dimensions
> literally?
> > >> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and
never
> > >> >> > get there because the end longitude is less than the
start?
> > >> >> >
> > >> >> > It's probably time to FTP the files. I'll push model and
> > >> >> > observation from the above case along with config. I'll
let
> > >> >> > you know when they're done... I suspect it will take a
couple
> > >> >> > hours given the FTP speed last time we did this.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Matt
> > >> >> >
> > >> >> > -----Original Message-----
> > >> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> > >> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >> >> >
> > >> >> > Matt,
> > >> >> >
> > >> >> > It's possible that this issue is caused by the fact that
it's
> > >> >> > a global grid. Please try the following...
> > >> >> >
> > >> >> > (1) Run plot_data_plane to make sure MET is reading the
> > >> >> > gridded data the way you expect:
> > >> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
> > >> level="L0";'
> > >> >> > And then look at that image (adovis.ps) to make sure
it
> > >> >> > looks
> > >> >> good.
> > >> >> >
> > >> >> > (2) Run plot_point_obs twice to see where MET is plotting
the
> > >> >> observations:
> > >> >> > plot_point_obs obs.nc obs.ps
> > >> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> > >> >> > The first call plots all points on a default global
grid.
> > >> >> > The second call plots all points, but using the grid it
reads
> > >> >> > from
> > >> >> model.grb file.
> > >> >> >
> > >> >> > Just look at the resulting images and see if anything
jumps
> > >> >> > out at
> > >> you.
> > >> >> > Perhaps you accidentally transposed the lat/lon values in
the
> > >> >> > NetCDF point observation file? Or perhaps it's something
to
> > >> >> > do with the range of longitude values you used? Or
something
> > >> >> > else I'm not thinking
> > >> >> of.
> > >> >> >
> > >> >> > If the problem doesn't become obvious after those steps,
> > >> >> > please send me a sample forecast grib file, NetCDF
observation
> > >> >> > file, and Point-Stat config file.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > John Halley Gotway
> > >> >> > met_help at ucar.edu
> > >> >> >
> > >> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR
USAF
> > >> >> > AFWA
> > >> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >> >
> > >> >> > >
> > >> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > >> >> > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > >> >> > > Queue: met_help
> > >> >> > > Subject: point_stat issue
> > >> >> > > Owner: Nobody
> > >> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> > >> >> > > Status: new
> > >> >> > > Ticket <URL:
> > >> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > > Here's my newest and latest headache using MET v4.1.
> > >> >> > >
> > >> >> > > I'm trying to verify a gridded field using point_stat.
The
> > >> >> > > observations are formatted as NetCDF. MET appears to
read
> > >> >> > > in everything without a problem but, as far as I can
tell,
> > >> >> > > MET throws all the observations out so there's nothing
left.
> > >> >> > > Here's
> > >> what I get:
> > >> >> > >
> > >> >> > > =================
> > >> >> > > /home/qcteam/METv4.1/bin/point_stat
> > >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20
> > >> >> > > 14
> > >> >> > > 100
> > >> >> > > 300 .g rib
> > >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.n
> > >> >> > > c
> > >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> > >> >> > > cf g -log ./point_stat_log.txt -outdir
> > >> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> > >> >> > > 4 DEBUG 1: Default Config File:
> > >> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > >> >> > > DEBUG 1: User Config File:
> > >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> > >> >> > > cf g DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file()
> > >> >> > > -> created new Met2dDataFile object of type
"FileType_Gb1".
> > >> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
> > >> >> > > VarInfo object of type "FileType_Gb1".
> > >> >> > > GSL_RNG_TYPE=mt19937
> > >> >> > > GSL_RNG_SEED=1
> > >> >> > > DEBUG 1: Forecast File:
> > >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20
> > >> >> > > 14
> > >> >> > > 100
> > >> >> > > 300
> > >> >> > > .g
> > >> >> > > rib
> > >> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation
File:
> > >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.n
> > >> >> > > c
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
------------------------------------------------------------
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Reading data for U-GWD/L0.
> > >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found
range
> > >> >> > > match for VarInfo "U-GWD/L0" in GRIB record 502 of GRIB
file
> > >> >> > >
> > >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201
> > >> >> > 41
> > >> >> > 003
> > >> >> > 00
> > >> >> .grib".
> > >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
> > >> >> > > GRIB records matching VarInfo "U-GWD/L0" in GRIB file
> > >> >> > >
> > >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_201
> > >> >> > 41
> > >> >> > 003
> > >> >> > 00
> > >> >> .grib".
> > >> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > >> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
> > >> >> > > climatology
> > >> >> levels.
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
------------------------------------------------------------
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Searching 64800 observations from 64800
messages.
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
------------------------------------------------------------
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
> > >> >> > > observation type ADPSFC, over region FULL, for
interpolation
> > >> >> > > method BILIN(4), using 0
> > >> >> > pairs.
> > >> >> > > DEBUG 3: Number of matched pairs = 0
> > >> >> > > DEBUG 3: Observations processed = 64800
> > >> >> > > DEBUG 3: Rejected: GRIB code = 0
> > >> >> > > DEBUG 3: Rejected: valid time = 0
> > >> >> > > DEBUG 3: Rejected: bad obs value = 43229
> > >> >> > > DEBUG 3: Rejected: off the grid = 21571
> > >> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
> > >> >> > > quality marker = 0
> > >> >> > > DEBUG 3: Rejected: message type = 0
> > >> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected:
bad
> > >> >> > > fcst value = 0
> > >> >> > > DEBUG 3: Rejected: duplicates = 0
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
------------------------------------------------------------
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V.stat
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_fho.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_ctc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_cts.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mctc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mcts.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_cnt.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_sl1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_sal1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_vl1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_val1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pct.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pstd.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pjc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_prc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_20141
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mpr.txt
> > >> >> > > ============
> > >> >> > >
> > >> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> > >> >> > > The observation data cover the entire global domain, as
does
> > >> >> > > the
> > >> model.
> > >> >> > > There shouldn't be any 'off the grid' data. Here's the
> > >> >> > > model data information, courtesy degrib.
> > >> >> > >
> > >> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147
kpds6=200
> > >> >> > > kpds7=0
> > >> >> > > levels=(0,0) grid=255 atmos col anl:
> > >> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > >> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid
0
> > >> >> > > num_in_ave
> > >> >> > > 0 missing 0
> > >> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
> > >> winds(grid)
> > >> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> > >> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
> > >> >> > > 180) scan 0 mode
> > >> >> > > 136 bdsgrid 1
> > >> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref
9945
> > >> >> > > DecScale
> > >> >> > > 6 BinScale 0
> > >> >> > >
> > >> >> > > Is it possible that MET is upset by the definition of
> > >> >> > > longitude as running from 0 to -0.5 in positive steps of
> > >> >> > > 1.003, which it will never reach? I'm not sure how MET
> > >> >> > > establishes the model grid; I presume through the GriB
header.
> > >> >> > >
> > >> >> > > I would appreciate any thoughts you have on this. I can
FTP
> > >> >> > > the files as before if that will help.
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > Matt
> > >> >> > >
> > >> >> > > // SIGNED //
> > >> >> > > Matthew C. Sittel
> > >> >> > > Associate Scientist
> > >> >> > > University Corporation for Atmospheric Research
16WS/WXN,
> > >> >> > > Offutt AFB, NE
> > >> >> > > (402) 294-3473 DSN: 271-3473
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #69380] point_stat issue
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Tue Oct 21 06:40:24 2014
Hey John,
I'm not sure yet. I got the modeler to make the header "make sense"
and run -180 to 180 on the longitude and to add 'scan 64' and, last we
checked, it looked good. I was out of office yesterday so need to see
where things stand now.
Modeler was putting in 'scan 0' because his boss said that's what his
code required... so I may still have a fight on my hands to get it to
say 'scan 64'. For the record, the ensembles run here at AFWA all
have 'scan 64' so using 'scan 0' appears to be non-standard... maybe I
can use that argument to convince them to change!
If you need to close out the issue so you don't get dinged for
unresolved ones, go ahead, and I'll open a new one if necessary.
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, October 20, 2014 2:26 PM
To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
Hello Matt,
Just wanted to check in. Is there any additional support you need
from me on this issue?
Thanks,
John
On Thu, Oct 16, 2014 at 12:28 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
>
> But you can generate them; we typically ask the ensembles guys to
make
> sure there are included in any GriB file. Now how those are
generated
> I'm not sure... but we read them when we need to, such as in this
WRF file:
>
> rec 108:3165146:date 2014101200 LAT kpds5=230 kpds6=1 kpds7=0
> levels=(0,0) grid=255 sfc 48hr fcst:
> LAT= Latitude [deg]
> timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3
num_in_ave
> 0 missing 0
> center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
> Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
> Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
> North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
> min/max data 5.0097 67.3016 num bits 20 BDS_Ref 50097 DecScale
4
> BinScale 0
>
> rec 109:3231984:date 2014101200 LON kpds5=231 kpds6=1 kpds7=0
> levels=(0,0)
> grid=255 sfc 48hr fcst:
> LON= Longitude [deg]
> timerange 0 P1 48 P2 0 TimeU 1 nx 192 ny 139 GDS grid 3
num_in_ave
> 0 missing 0
> center 57 subcenter 1 process 11 Table 133 scan: WE:SN winds(grid)
> Lambert Conf: Lat1 5.009000 Lon1 -129.001000 Lov -96.000000
> Latin1 60.000000 Latin2 30.000000 LatSP -90.000000 LonSP
0.000000
> North Pole (192 x 139) Dx 45.000000 Dy 45.000000 scan 64 mode
136
> min/max data -165.726 -26.2738 num bits 21 BDS_Ref -1.65726e+06
> DecScale 4 BinScale 0
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 16, 2014 1:25 PM
> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
>
> Matt,
>
> That's right. GRIB files do not include actual values for latitude
> and longitude, as NetCDF files often do. The lat/lon values are
> computed using the grid definition information contained in the GDS
> (grid description
> section) of each GRIB record.
>
> John
>
> On Thu, Oct 16, 2014 at 12:15 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> >
> > Hi John,
> >
> > Since I'm not the modeler, I don't know all the details, but what
I
> > think is happening is that the header is being set wrong. There
are
> > no LAT or LON fields in the GriB file to check it against. I
don't
> > think the lat and lon corner points in the header are
automatically
> > set. He's trying one more run... maybe that's all this is. I'll
> > let
> you know.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, October 16, 2014 12:28 PM
> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> >
> > Matt,
> >
> > I've attached two updated images.
> >
> > "aodvis_64.png" shows how MET reads the data when the scanning
mode
> > flag is set to 64. The plot I sent yesterday was wrong... I
> > manually changed the scan flag in the code to 64 in one spot but
> > missed it in a
> second spot.
> > Sorry for the confusion.
> >
> > "aodvis_64_shift_180.png" shows how MET reads the data when the
> > scanning mode flag is set to 64 and the lower-left longitude is
set
> > to -180, instead of 0.
> >
> > Looks like the second image orients the data correctly and put it
on
> > the correct place on the earth. So perhaps the scanning mode flag
> > and the grid definition are incorrect?
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 16, 2014 at 10:44 AM, John Halley Gotway
> > <johnhg at ucar.edu>
> > wrote:
> >
> > > Matt,
> > >
> > > Would you be able to send me a plot that shows how the data
should
> > > be correctly oriented? I'm not familiar with the xconv tool and
> > > we don't have it on our system.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Oct 16, 2014 at 6:32 AM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380 >
> > >>
> > >> Hi John,
> > >>
> > >> Feedback from the modeler:
> > >>
> > >> "Well their plot is still wrong. The area that has the high
> > >> concentration of dust (the dark red spot) is Northern Africa.
So
> > >> the map needs to be rotated 180 degrees. Which I don't
understand
> > >> why their image isn’t doing it properly, doesn’t the girb file
> > >> give lat/lon of each value? I don’t get why other graphic
> > >> utilities like xconv have no issues with the grid. Also the
idv
> > >> data is actually correct map wise (i.e. it is flipped the right
> > >> way, just the location
> > is off (again needs to be rotated 180 degrees)"
> > >>
> > >> The model data is built from software sent by NCAR, so it's not
> > >> in-house. The header information looks to be straightforward
to
> adjust.
> > >>
> > >> -----Original Message-----
> > >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> Sent: Wednesday, October 15, 2014 2:41 PM
> > >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >>
> > >> Matt,
> > >>
> > >> The problem is how the "scanning mode flag" is set in the GRIB1
> > >> file you're using. It is currently set to 0, which in bits is
> "00000000".
> > >> Here's a
> > >> GRIB1 table that describes this flag:
> > >> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table8.html
> > >>
> > >> Notice the following in your wgrib output:
> > >> scan: WE:NS winds(grid)
> > >>
> > >> All zeros means that the data is stored in the +x and -y
directions.
> > >> The grid definition includes lat1 and lat2 values. When data
is
> > >> in the -y direction, we use lat2 to define the latitude of the
> > >> lower-left
> > corner.
> > >> But in your GRIB file lat1 = -90 and lat2 = 90. So we're
setting
> > >> the latitude of the lower-left corner to 90 (when you really
want
> > >> it to be -90). The best fix would be for the creator of this
> > >> GRIB1 file to set that flag in a way that's consistent with the
> > >> grid
> definition.
> > >>
> > >> For testing, I added the following after line 196 of the file
> > >> METv4.1/src/libcode/vx_data2d_grib/grib_utils.cc:
> > >> data.lat_ll =
> > >> min(decode_lat_lon(gds.grid_type.latlon_grid.lat1,
> > >> 3),
> > >>
> > >> decode_lat_lon(gds.grid_type.latlon_grid.lat2,
> > >> 3));
> > >>
> > >> This just forces it to use the minimum of the two latitudes for
> > >> the lower-left corner.
> > >>
> > >> That results in the attached image (aodvis_from_MET.png).
Since
> > >> the map data is on there, MET knows where it is on the earth
now.
> > >>
> > >> ***BUT*** when I plot the same dataset using IDV, I get the
> > >> attached image (aodvis_from_IDV.png). Notice how they are
> > >> flipped
> top to bottom.
> > >>
> > >> I've always found this scanning mode flag to be pretty tedious.
> > >> It enables you to pack the data in whatever order you prefer,
but
> > >> that just makes the code to read the data more complex and
causes
> > >> issues like this to pop up.
> > >>
> > >> So there's an inconsistency between the scanning mode flag and
> > >> the grid
> > >> lat1 and lat2 values. Either lat1 and lat2 need to be swapped
or
> > >> the scanning mode flag needs to be set differently. My guess
is
> > >> that it's the scanning mode flag, because one would naturally
> > >> assume that
> > >> 0 is a good default value - but it's not. The most common
> > >> scanning mode I see is 64, which in bits is 0100000 (i.e. 64 =
> > >> 2^6, so that means the 2nd bit is 1, from most-to-least
> > >> significant bits). And that means data is stored in the
> > >> +x and +y directions.
> > >>
> > >> My suggestion is to have the creator of this GRIB file set the
> > >> scanning mode flag to 64. I manually did that in the MET code,
> > >> removed the data.lat_ll line I mentioned above and reran
> > >> plot_data_plane. It resulted in the same aodvis_from_met.png
> > >> image
> > that I attached.
> > >>
> > >> But you guys need to decide which way is "up" in your data!
> > >> That's easiest if you have any fields whose values correspond
well with land.
> > >>
> > >> As a side note, if you to send me a single record from a larger
> > >> GRIB file, you can extract using wgrib like this (use -d to
pick
> > >> the record
> > >> number):
> > >> wgrib -d 502 -grib -o aodvis.grb FIM_Chem_2014100300.grib
> > >>
> > >> Hope that helps.
> > >>
> > >> John
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Oct 15, 2014 at 12:51 PM, John Halley Gotway
> > >> <johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >> > Matt,
> > >> >
> > >> > When I see aodvis.pdf, I do see an obvious problem. There
are
> > >> > no coastlines or map data. So MET is able to read the 2D
data
> > >> > fine, but it just doesn't know where on earth to place it.
So
> > >> > there's a problem with our grid definition logic. I'll take
a
> > >> > closer look and try to
> > >> nail it down.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> >
> > >> > On Wed, Oct 15, 2014 at 12:39 PM, SITTEL, MATTHEW C CTR USAF
> > >> > AFWA
> > >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >> >> >
> > >> >>
> > >> >> While we wait... here's a curiosity that might be
noteworthy,
> > >> >> or
> > not...
> > >> >>
> > >> >> When I plot the model GriB data, specifically the field
> > >> >> kpds5=147
> > >> >> (AODVIS) using plot_data_plane I get the PDF output I've
attached.
> > >> >>
> > >> >> When the model developer visualizes the data, he swears that
> > >> >> the GIF I've attached is the correct layout... a version
> > >> >> seemingly rotated
> > >> >> 180 degrees along the Equatorial axis.
> > >> >>
> > >> >> I'm not sure what to believe, but I'm wondering now if
> > >> >> plot_data_plane (or MET for that matter) assumes anything
> > >> >> regarding the north-south orientation of the data. I can't
> > >> >> imagine that it does... but I thought I'd mention it just
the same.
> > >> >>
> > >> >> The observation and config file have transferred... just
> > >> >> waiting on the ~370MB model file to finish.
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> >> Sent: Wednesday, October 15, 2014 12:28 PM
> > >> >> To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> >> Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >> >>
> > >> >> Matt,
> > >> >>
> > >> >> OK, sounds good. Just let me know when the transfer is
> > >> >> complete and I'll go grab the data.
> > >> >>
> > >> >> Thanks,
> > >> >> John
> > >> >>
> > >> >> On Wed, Oct 15, 2014 at 11:19 AM, SITTEL, MATTHEW C CTR USAF
> > >> >> AFWA
> > >> >> 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >>
> > >> >> >
> > >> >> > <URL:
> > >> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >> >> > >
> > >> >> >
> > >> >> > Well the first two plots look just fine, but the third
won't
> work:
> > >> >> >
> > >> >> > DEBUG 2: Retrieving grid from file:
> > >> >> >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_20
> > >> >> > 14
> > >> >> > 10
> > >> >> > 030
> > >> >> > 0.g
> > >> >> > rib
> > >> >> > DEBUG 1: Opening netCDF file:
> > >> >> > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300.n
> > >> >> > c DEBUG 2: Processing 64800 observations at 64800
locations.
> > >> >> > DEBUG 2: Observation GRIB codes: ALL DEBUG 2: Observation
> > >> >> > message
> > >> >> > types: ALL DEBUG 1: Creating postscript
> > >> >> > file: obs_data_file.ps DEBUG 2: Finished plotting 0
locations.
> > >> >> > DEBUG 2: Skipped 64800 locations off the grid.
> > >> >> >
> > >> >> > Since the second file looks fine, that implies the NetCDF
is
> > >> >> > formatted right. The GriB data seems okay too. But the
> > >> >> > pair
> > >> together are not.
> > >> >> > That's what leads me to question if the GriB header in the
> > >> >> > model GriB file is a problem. Does MET take its
dimensions
> literally?
> > >> >> > Does it try to go from 0.0 to -0.5 in steps of 1.003 and
> > >> >> > never get there because the end longitude is less than the
start?
> > >> >> >
> > >> >> > It's probably time to FTP the files. I'll push model and
> > >> >> > observation from the above case along with config. I'll
let
> > >> >> > you know when they're done... I suspect it will take a
> > >> >> > couple hours given the FTP speed last time we did this.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Matt
> > >> >> >
> > >> >> > -----Original Message-----
> > >> >> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > >> >> > Sent: Wednesday, October 15, 2014 11:39 AM
> > >> >> > To: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
> > >> >> > Subject: Re: [rt.rap.ucar.edu #69380] point_stat issue
> > >> >> >
> > >> >> > Matt,
> > >> >> >
> > >> >> > It's possible that this issue is caused by the fact that
> > >> >> > it's a global grid. Please try the following...
> > >> >> >
> > >> >> > (1) Run plot_data_plane to make sure MET is reading the
> > >> >> > gridded data the way you expect:
> > >> >> > plot_data_plane model.grb aodvis.ps 'name="AODVIS";
> > >> level="L0";'
> > >> >> > And then look at that image (adovis.ps) to make sure
it
> > >> >> > looks
> > >> >> good.
> > >> >> >
> > >> >> > (2) Run plot_point_obs twice to see where MET is plotting
> > >> >> > the
> > >> >> observations:
> > >> >> > plot_point_obs obs.nc obs.ps
> > >> >> > plot_point_obs obs.nc obs_data_file.ps -data_file
model.grb
> > >> >> > The first call plots all points on a default global
grid.
> > >> >> > The second call plots all points, but using the grid it
> > >> >> > reads from
> > >> >> model.grb file.
> > >> >> >
> > >> >> > Just look at the resulting images and see if anything
jumps
> > >> >> > out at
> > >> you.
> > >> >> > Perhaps you accidentally transposed the lat/lon values in
> > >> >> > the NetCDF point observation file? Or perhaps it's
> > >> >> > something to do with the range of longitude values you
used?
> > >> >> > Or something else I'm not thinking
> > >> >> of.
> > >> >> >
> > >> >> > If the problem doesn't become obvious after those steps,
> > >> >> > please send me a sample forecast grib file, NetCDF
> > >> >> > observation file, and Point-Stat config file.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > John Halley Gotway
> > >> >> > met_help at ucar.edu
> > >> >> >
> > >> >> > On Wed, Oct 15, 2014 at 10:05 AM, SITTEL, MATTHEW C CTR
USAF
> > >> >> > AFWA
> > >> >> > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >> >> >
> > >> >> > >
> > >> >> > > Wed Oct 15 10:05:22 2014: Request 69380 was acted upon.
> > >> >> > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > >> >> > > Queue: met_help
> > >> >> > > Subject: point_stat issue
> > >> >> > > Owner: Nobody
> > >> >> > > Requestors: matthew.sittel.ctr at us.af.mil
> > >> >> > > Status: new
> > >> >> > > Ticket <URL:
> > >> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69380
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > > Here's my newest and latest headache using MET v4.1.
> > >> >> > >
> > >> >> > > I'm trying to verify a gridded field using point_stat.
> > >> >> > > The observations are formatted as NetCDF. MET appears
to
> > >> >> > > read in everything without a problem but, as far as I
can
> > >> >> > > tell, MET throws all the observations out so there's
nothing left.
> > >> >> > > Here's
> > >> what I get:
> > >> >> > >
> > >> >> > > =================
> > >> >> > > /home/qcteam/METv4.1/bin/point_stat
> > >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_
> > >> >> > > 20
> > >> >> > > 14
> > >> >> > > 100
> > >> >> > > 300 .g rib
> > >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300
> > >> >> > > .n
> > >> >> > > c
> > >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> > >> >> > > cf g -log ./point_stat_log.txt -outdir
> > >> >> > > /home/qcteam/METv4.1/data/aod/stats -v
> > >> >> > > 4 DEBUG 1: Default Config File:
> > >> >> > > /home/qcteam/METv4.1/data/config/PointStatConfig_default
> > >> >> > > DEBUG 1: User Config File:
> > >> >> > > /home/qcteam/METv4.1/data/aod/cfg/aod_point_stat_AQUA-
MODIS.
> > >> >> > > cf g DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file()
> > >> >> > > -> created new Met2dDataFile object of type
"FileType_Gb1".
> > >> >> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new
> > >> >> > > VarInfo object of type "FileType_Gb1".
> > >> >> > > GSL_RNG_TYPE=mt19937
> > >> >> > > GSL_RNG_SEED=1
> > >> >> > > DEBUG 1: Forecast File:
> > >> >> > >
/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_
> > >> >> > > 20
> > >> >> > > 14
> > >> >> > > 100
> > >> >> > > 300
> > >> >> > > .g
> > >> >> > > rib
> > >> >> > > DEBUG 1: Climatology File: none DEBUG 1: Observation
File:
> > >> >> > > /home/qcteam/METv4.1/data/aod/output/AQUA-
MODIS_2014100300
> > >> >> > > .n
> > >> >> > > c
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
----------------------------------------------------------
> > >> >> > > --
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Reading data for U-GWD/L0.
> > >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found
> > >> >> > > range match for VarInfo "U-GWD/L0" in GRIB record 502 of
> > >> >> > > GRIB file
> > >> >> > >
> > >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2
> > >> >> > 01
> > >> >> > 41
> > >> >> > 003
> > >> >> > 00
> > >> >> .grib".
> > >> >> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
> > >> >> > > GRIB records matching VarInfo "U-GWD/L0" in GRIB file
> > >> >> > >
> > >> >> >
"/home/bliujuss/WRF_post_process/DATA/20141002_00/FIM_Chem_2
> > >> >> > 01
> > >> >> > 41
> > >> >> > 003
> > >> >> > 00
> > >> >> .grib".
> > >> >> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > >> >> > > DEBUG 2: For U-GWD/L0 found 1 forecast levels and 0
> > >> >> > > climatology
> > >> >> levels.
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
----------------------------------------------------------
> > >> >> > > --
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Searching 64800 observations from 64800
messages.
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
----------------------------------------------------------
> > >> >> > > --
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2: Processing U-GWD/L0 versus U-GWD/L0, for
> > >> >> > > observation type ADPSFC, over region FULL, for
> > >> >> > > interpolation method BILIN(4), using 0
> > >> >> > pairs.
> > >> >> > > DEBUG 3: Number of matched pairs = 0
> > >> >> > > DEBUG 3: Observations processed = 64800
> > >> >> > > DEBUG 3: Rejected: GRIB code = 0
> > >> >> > > DEBUG 3: Rejected: valid time = 0
> > >> >> > > DEBUG 3: Rejected: bad obs value = 43229
> > >> >> > > DEBUG 3: Rejected: off the grid = 21571
> > >> >> > > DEBUG 3: Rejected: level mismatch = 0 DEBUG 3: Rejected:
> > >> >> > > quality marker = 0
> > >> >> > > DEBUG 3: Rejected: message type = 0
> > >> >> > > DEBUG 3: Rejected: masking region = 0 DEBUG 3: Rejected:
> > >> >> > > bad fcst value = 0
> > >> >> > > DEBUG 3: Rejected: duplicates = 0
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 2:
> > >> >> > >
----------------------------------------------------------
> > >> >> > > --
> > >> >> > > --
> > >> >> > > ---
> > >> >> > > ---
> > >> >> > > --
> > >> >> > > ----------
> > >> >> > > DEBUG 2:
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V.stat
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_fho.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_ctc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_cts.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mctc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mcts.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_cnt.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_sl1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_sal1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_vl1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_val1l2.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pct.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pstd.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_pjc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_prc.txt
> > >> >> > > DEBUG 1: Output file:
> > >> >> > >
/home/qcteam/METv4.1/data/aod/stats/point_stat_000000L_201
> > >> >> > > 41
> > >> >> > > 00
> > >> >> > > 3_0
> > >> >> > > 000
> > >> >> > > 00
> > >> >> > > V_mpr.txt
> > >> >> > > ============
> > >> >> > >
> > >> >> > > What I'm trying to figure out is what 'off the grid'
signifies.
> > >> >> > > The observation data cover the entire global domain, as
> > >> >> > > does the
> > >> model.
> > >> >> > > There shouldn't be any 'off the grid' data. Here's the
> > >> >> > > model data information, courtesy degrib.
> > >> >> > >
> > >> >> > > rec 502:69286792:date 2014100300 AODVIS kpds5=147
> > >> >> > > kpds6=200
> > >> >> > > kpds7=0
> > >> >> > > levels=(0,0) grid=255 atmos col anl:
> > >> >> > > AODVIS=Aerosol optical depth (0.55 um) [numeric]
> > >> >> > > timerange 0 P1 0 P2 0 TimeU 1 nx 359 ny 180 GDS grid
0
> > >> >> > > num_in_ave
> > >> >> > > 0 missing 0
> > >> >> > > center 57 subcenter 6 process 15 Table 129 scan: WE:NS
> > >> winds(grid)
> > >> >> > > latlon: lat -90.000000 to 90.000000 by 1.003000 nxny
64620
> > >> >> > > long 0.000000 to -0.500000 by 1.003000, (359 x
> > >> >> > > 180) scan 0 mode
> > >> >> > > 136 bdsgrid 1
> > >> >> > > min/max data 0.009945 1.13231 num bits 21 BDS_Ref
9945
> > >> >> > > DecScale
> > >> >> > > 6 BinScale 0
> > >> >> > >
> > >> >> > > Is it possible that MET is upset by the definition of
> > >> >> > > longitude as running from 0 to -0.5 in positive steps of
> > >> >> > > 1.003, which it will never reach? I'm not sure how MET
> > >> >> > > establishes the model grid; I presume through the GriB
header.
> > >> >> > >
> > >> >> > > I would appreciate any thoughts you have on this. I can
> > >> >> > > FTP the files as before if that will help.
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > Matt
> > >> >> > >
> > >> >> > > // SIGNED //
> > >> >> > > Matthew C. Sittel
> > >> >> > > Associate Scientist
> > >> >> > > University Corporation for Atmospheric Research
16WS/WXN,
> > >> >> > > Offutt AFB, NE
> > >> >> > > (402) 294-3473 DSN: 271-3473
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list