[Met_help] [rt.rap.ucar.edu #93532] History for point stat questions
John Halley Gotway via RT
met_help at ucar.edu
Fri Jan 17 14:11:08 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John,
A few questions about point stat:
1. Are grid relative winds automatically converted to Earth relative
winds?
2. For observation that fall below the model surface, are model
fields interpolated to observation location? If so, how?
Thanks!
Liz
Elizabeth Satterfield
Editor, Monthly Weather Review
Chair, AMS Committee on Probability and Statistics
Meteorologist, Code 7531
Marine Meteorology Division
U.S. Naval Research Laboratory
T 831.656.4056 F 831.656.4026
<http://www.nrl.navy.mil/> www.nrl.navy.mil
__________________________________________________________________
FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
Any misuse or unauthorized disclosure may result in both civil and criminal
penalties.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Thu Dec 19 07:04:54 2019
Liz,
(1) When reading data from GRIB1 and GRIB2 files, MET automatically
checks
the flag in the header indicating whether winds are defined as grid or
earth-relative. If grid relative, it searches the file for the
correspond
U or V pair and uses both U and V to rotate them to be earth-relative.
That logic does NOT work for python embedding. I have some ideas
about how
to make it work but those changes are not slated for the met-9.0
release.
I assume you’d like to see this enhancement? And perhaps we’ve
already
talked about this?
(2) Config options were added in met-8.1 for the Air Force which
address
this issue. By default this logic is disabled but can be enabled
using the
“topo_mask” config file dictionary. FYI, the “land_mask” dictionary
was
also added at the same time. Below is a link to the default point-
stat
config file followed by an excerpt from it.
John
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
// Topography
// For SURFACE message types, only use observations where the topo -
station
// elevation difference meets the use_obs_thresh threshold.
// For the observations kept, when interpolating forecast data to the
// observation location, only use forecast grid points where the topo
-
station
// difference meets the interp_fcst_thresh threshold.
// topo_mask.flag may be set separately in each "obs.field" entry.
//
topo_mask = {
flag = FALSE;
file_name = [];
field = { name = "TOPO"; level = "L0"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: point stat questions
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
>
> Hi John,
>
>
>
> A few questions about point stat:
>
> 1. Are grid relative winds automatically converted to Earth
> relative
> winds?
>
> 2. For observation that fall below the model surface, are
model
> fields interpolated to observation location? If so, how?
>
>
>
> Thanks!
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Fri Dec 20 17:18:07 2019
Thanks John,
When using topo_mask, will point_Stat understand a python embedded
file? The files we use are named '
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
These are projection gridded data of terrain heights. Would this
look something like this?
topo_mask = {
flag = FALSE;
file_name = ["./read_NRL_binary
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
field = { name = "TOPO"; level = "L0"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
And how does point_stat know which HDF5 varible to use as the point of
comparison to the model terrain height?
Happy holidays!
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, December 19, 2019 6:05 AM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Liz,
(1) When reading data from GRIB1 and GRIB2 files, MET automatically
checks
the flag in the header indicating whether winds are defined as grid or
earth-relative. If grid relative, it searches the file for the
correspond
U or V pair and uses both U and V to rotate them to be earth-relative.
That logic does NOT work for python embedding. I have some ideas
about how
to make it work but those changes are not slated for the met-9.0
release.
I assume you’d like to see this enhancement? And perhaps we’ve
already
talked about this?
(2) Config options were added in met-8.1 for the Air Force which
address
this issue. By default this logic is disabled but can be enabled
using the
“topo_mask” config file dictionary. FYI, the “land_mask” dictionary
was
also added at the same time. Below is a link to the default point-
stat
config file followed by an excerpt from it.
John
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
// Topography
// For SURFACE message types, only use observations where the topo -
station
// elevation difference meets the use_obs_thresh threshold.
// For the observations kept, when interpolating forecast data to the
// observation location, only use forecast grid points where the topo
-
station
// difference meets the interp_fcst_thresh threshold.
// topo_mask.flag may be set separately in each "obs.field" entry.
//
topo_mask = {
flag = FALSE;
file_name = [];
field = { name = "TOPO"; level = "L0"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: point stat questions
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
>
> Hi John,
>
>
>
> A few questions about point stat:
>
> 1. Are grid relative winds automatically converted to Earth
> relative
> winds?
>
> 2. For observation that fall below the model surface, are
model
> fields interpolated to observation location? If so, how?
>
>
>
> Thanks!
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Wed Jan 08 14:32:18 2020
Hey John,
This email probably got lost during the holidays. Let me know if you
can get around to helping me out with this problem!
Thanks and happy new year!
Justin
-----Original Message-----
From: Tsu, Mr. Justin
Sent: Friday, December 20, 2019 4:18 PM
To: 'met_help at ucar.edu'
Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
Thanks John,
When using topo_mask, will point_Stat understand a python embedded
file? The files we use are named '
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
These are projection gridded data of terrain heights. Would this
look something like this?
topo_mask = {
flag = FALSE;
file_name = ["./read_NRL_binary
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
field = { name = "TOPO"; level = "L0"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
And how does point_stat know which HDF5 varible to use as the point of
comparison to the model terrain height?
Happy holidays!
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, December 19, 2019 6:05 AM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Liz,
(1) When reading data from GRIB1 and GRIB2 files, MET automatically
checks
the flag in the header indicating whether winds are defined as grid or
earth-relative. If grid relative, it searches the file for the
correspond
U or V pair and uses both U and V to rotate them to be earth-relative.
That logic does NOT work for python embedding. I have some ideas
about how
to make it work but those changes are not slated for the met-9.0
release.
I assume you’d like to see this enhancement? And perhaps we’ve
already
talked about this?
(2) Config options were added in met-8.1 for the Air Force which
address
this issue. By default this logic is disabled but can be enabled
using the
“topo_mask” config file dictionary. FYI, the “land_mask” dictionary
was
also added at the same time. Below is a link to the default point-
stat
config file followed by an excerpt from it.
John
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
// Topography
// For SURFACE message types, only use observations where the topo -
station
// elevation difference meets the use_obs_thresh threshold.
// For the observations kept, when interpolating forecast data to the
// observation location, only use forecast grid points where the topo
-
station
// difference meets the interp_fcst_thresh threshold.
// topo_mask.flag may be set separately in each "obs.field" entry.
//
topo_mask = {
flag = FALSE;
file_name = [];
field = { name = "TOPO"; level = "L0"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: point stat questions
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
>
> Hi John,
>
>
>
> A few questions about point stat:
>
> 1. Are grid relative winds automatically converted to Earth
> relative
> winds?
>
> 2. For observation that fall below the model surface, are
model
> fields interpolated to observation location? If so, how?
>
>
>
> Thanks!
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Thu Jan 09 11:26:35 2020
Justin,
Sorry for losing this in the shuffle. It's a hectic time of year.
So your question is whether or not Python-embedding will work when
setting
the "topo_mask" dictionary in the Point-Stat config file. I just
tested
this out (albeit with the development version of the code) and it
seems to
have worked fine, just as expected. Remember though that for
python-embedding you set the file_name to the constant string
"PYTHON_NUMPY" and the field "name" to the python command you want to
execute to read the data.
It looks like you were setting file_name = python command, and that
won't
work.
Thanks,
John
//
// Topography
//
topo_mask = {
flag = FALSE;
file_name = [ "PYTHON_NUMPY" ];
field = { name = "read_NRL_binary.py
trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Hey John,
>
> This email probably got lost during the holidays. Let me know if
you can
> get around to helping me out with this problem!
>
> Thanks and happy new year!
> Justin
>
> -----Original Message-----
> From: Tsu, Mr. Justin
> Sent: Friday, December 20, 2019 4:18 PM
> To: 'met_help at ucar.edu'
> Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
>
> Thanks John,
>
> When using topo_mask, will point_Stat understand a python embedded
file?
> The files we use are named '
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> These are projection gridded data of terrain heights. Would this
look
> something like this?
>
> topo_mask = {
> flag = FALSE;
> file_name = ["./read_NRL_binary
> terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> field = { name = "TOPO"; level = "L0"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
> And how does point_stat know which HDF5 varible to use as the point
of
> comparison to the model terrain height?
>
> Happy holidays!
> Justin
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, December 19, 2019 6:05 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Liz,
>
> (1) When reading data from GRIB1 and GRIB2 files, MET automatically
checks
> the flag in the header indicating whether winds are defined as grid
or
> earth-relative. If grid relative, it searches the file for the
correspond
> U or V pair and uses both U and V to rotate them to be earth-
relative.
>
> That logic does NOT work for python embedding. I have some ideas
about how
> to make it work but those changes are not slated for the met-9.0
release.
>
> I assume you’d like to see this enhancement? And perhaps we’ve
already
> talked about this?
>
> (2) Config options were added in met-8.1 for the Air Force which
address
> this issue. By default this logic is disabled but can be enabled
using the
> “topo_mask” config file dictionary. FYI, the “land_mask” dictionary
was
> also added at the same time. Below is a link to the default point-
stat
> config file followed by an excerpt from it.
>
> John
>
>
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
>
> // Topography
> // For SURFACE message types, only use observations where the topo -
> station
> // elevation difference meets the use_obs_thresh threshold.
> // For the observations kept, when interpolating forecast data to
the
> // observation location, only use forecast grid points where the
topo -
> station
> // difference meets the interp_fcst_thresh threshold.
> // topo_mask.flag may be set separately in each "obs.field" entry.
> //
> topo_mask = {
> flag = FALSE;
> file_name = [];
> field = { name = "TOPO"; level = "L0"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
>
> On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: point stat questions
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> >
> > Hi John,
> >
> >
> >
> > A few questions about point stat:
> >
> > 1. Are grid relative winds automatically converted to
Earth
> > relative
> > winds?
> >
> > 2. For observation that fall below the model surface, are
model
> > fields interpolated to observation location? If so, how?
> >
> >
> >
> > Thanks!
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > Any misuse or unauthorized disclosure may result in both civil and
> criminal
> > penalties.
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Fri Jan 17 11:26:01 2020
Thanks John,
I just implemented your changes for a simple point_stat case verifying
wind direction. The point_stat debugging output looks exactly the
same as if I hadn't used topo_mask... how do I know if it actually
worked and terrain height got into point_stat?
Also, how does point_stat know what netCDF variable to check terrht
against?
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, January 9, 2020 10:27 AM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Justin,
Sorry for losing this in the shuffle. It's a hectic time of year.
So your question is whether or not Python-embedding will work when
setting
the "topo_mask" dictionary in the Point-Stat config file. I just
tested
this out (albeit with the development version of the code) and it
seems to
have worked fine, just as expected. Remember though that for
python-embedding you set the file_name to the constant string
"PYTHON_NUMPY" and the field "name" to the python command you want to
execute to read the data.
It looks like you were setting file_name = python command, and that
won't
work.
Thanks,
John
//
// Topography
//
topo_mask = {
flag = FALSE;
file_name = [ "PYTHON_NUMPY" ];
field = { name = "read_NRL_binary.py
trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
regrid = { method = BILIN; width = 2; }
use_obs_thresh = ge-100&&le100;
interp_fcst_thresh = ge-50&&le50;
}
On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Hey John,
>
> This email probably got lost during the holidays. Let me know if
you can
> get around to helping me out with this problem!
>
> Thanks and happy new year!
> Justin
>
> -----Original Message-----
> From: Tsu, Mr. Justin
> Sent: Friday, December 20, 2019 4:18 PM
> To: 'met_help at ucar.edu'
> Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
>
> Thanks John,
>
> When using topo_mask, will point_Stat understand a python embedded
file?
> The files we use are named '
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> These are projection gridded data of terrain heights. Would this
look
> something like this?
>
> topo_mask = {
> flag = FALSE;
> file_name = ["./read_NRL_binary
> terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> field = { name = "TOPO"; level = "L0"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
> And how does point_stat know which HDF5 varible to use as the point
of
> comparison to the model terrain height?
>
> Happy holidays!
> Justin
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, December 19, 2019 6:05 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Liz,
>
> (1) When reading data from GRIB1 and GRIB2 files, MET automatically
checks
> the flag in the header indicating whether winds are defined as grid
or
> earth-relative. If grid relative, it searches the file for the
correspond
> U or V pair and uses both U and V to rotate them to be earth-
relative.
>
> That logic does NOT work for python embedding. I have some ideas
about how
> to make it work but those changes are not slated for the met-9.0
release.
>
> I assume you’d like to see this enhancement? And perhaps we’ve
already
> talked about this?
>
> (2) Config options were added in met-8.1 for the Air Force which
address
> this issue. By default this logic is disabled but can be enabled
using the
> “topo_mask” config file dictionary. FYI, the “land_mask” dictionary
was
> also added at the same time. Below is a link to the default point-
stat
> config file followed by an excerpt from it.
>
> John
>
>
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
>
> // Topography
> // For SURFACE message types, only use observations where the topo -
> station
> // elevation difference meets the use_obs_thresh threshold.
> // For the observations kept, when interpolating forecast data to
the
> // observation location, only use forecast grid points where the
topo -
> station
> // difference meets the interp_fcst_thresh threshold.
> // topo_mask.flag may be set separately in each "obs.field" entry.
> //
> topo_mask = {
> flag = FALSE;
> file_name = [];
> field = { name = "TOPO"; level = "L0"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
>
> On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: point stat questions
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> >
> > Hi John,
> >
> >
> >
> > A few questions about point stat:
> >
> > 1. Are grid relative winds automatically converted to
Earth
> > relative
> > winds?
> >
> > 2. For observation that fall below the model surface, are
model
> > fields interpolated to observation location? If so, how?
> >
> >
> >
> > Thanks!
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > Any misuse or unauthorized disclosure may result in both civil and
> criminal
> > penalties.
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Fri Jan 17 12:05:55 2020
Justin,
Running Point-stat and verbosity level 3 causes these types of log
messages
to be printed:
DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using 3779
matched pairs.
DEBUG 3: Number of matched pairs = 3779
DEBUG 3: Observations processed = 116861
DEBUG 3: Rejected: SID exclusion = 0
DEBUG 3: Rejected: obs type = 103809
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 21
DEBUG 3: Rejected: topography = 548
DEBUG 3: Rejected: level mismatch = 7675
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 809
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 220
DEBUG 3: Rejected: duplicates = 0
The "topography" line indicates the number of point observations which
were
skipped due to the topography logic. And they are skipped when...
(1) the model topo value or point obs elevation value are bad data.
(2) their difference exceeds the "use_obs_thresh" threshold
And we also write a DEBUG(4) log message about it:
DEBUG 4: Skipping observation for topography difference since 119.598
is
not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
[ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
So please try rerunning with verbosity level 4 (-v 4) and check for
log
messages about "topography" or "geography".
Your next question...
Also, how does point_stat know what netCDF variable to check terrht
against?
Point-Stat is comparing the model "terrht" against the observation
station
elevation, which is the 6th column of the 11-column point observation
format:
The "met_point" ASCII format consists of 11 columns:
Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
Lat(Deg North) Lon(Deg East) Elevation(msl)
Var_Name(or GRIB_Code) Level Height(msl or agl)
QC_String Observation_Value
This logic is used to discard point obs with large differences from
the
model elevation.
John
On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Thanks John,
>
> I just implemented your changes for a simple point_stat case
verifying
> wind direction. The point_stat debugging output looks exactly the
same as
> if I hadn't used topo_mask... how do I know if it actually worked
and
> terrain height got into point_stat?
>
> Also, how does point_stat know what netCDF variable to check terrht
> against?
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, January 9, 2020 10:27 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Justin,
>
> Sorry for losing this in the shuffle. It's a hectic time of year.
>
> So your question is whether or not Python-embedding will work when
setting
> the "topo_mask" dictionary in the Point-Stat config file. I just
tested
> this out (albeit with the development version of the code) and it
seems to
> have worked fine, just as expected. Remember though that for
> python-embedding you set the file_name to the constant string
> "PYTHON_NUMPY" and the field "name" to the python command you want
to
> execute to read the data.
>
> It looks like you were setting file_name = python command, and that
won't
> work.
>
> Thanks,
> John
>
> //
> // Topography
> //
> topo_mask = {
> flag = FALSE;
> file_name = [ "PYTHON_NUMPY" ];
> field = { name = "read_NRL_binary.py
> trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
>
>
>
> On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Hey John,
> >
> > This email probably got lost during the holidays. Let me know if
you can
> > get around to helping me out with this problem!
> >
> > Thanks and happy new year!
> > Justin
> >
> > -----Original Message-----
> > From: Tsu, Mr. Justin
> > Sent: Friday, December 20, 2019 4:18 PM
> > To: 'met_help at ucar.edu'
> > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Thanks John,
> >
> > When using topo_mask, will point_Stat understand a python
embedded file?
> > The files we use are named '
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > These are projection gridded data of terrain heights. Would this
look
> > something like this?
> >
> > topo_mask = {
> > flag = FALSE;
> > file_name = ["./read_NRL_binary
> >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > field = { name = "TOPO"; level = "L0"; }
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> > And how does point_stat know which HDF5 varible to use as the
point of
> > comparison to the model terrain height?
> >
> > Happy holidays!
> > Justin
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, December 19, 2019 6:05 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Liz,
> >
> > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> checks
> > the flag in the header indicating whether winds are defined as
grid or
> > earth-relative. If grid relative, it searches the file for the
> correspond
> > U or V pair and uses both U and V to rotate them to be earth-
relative.
> >
> > That logic does NOT work for python embedding. I have some ideas
about
> how
> > to make it work but those changes are not slated for the met-9.0
release.
> >
> > I assume you’d like to see this enhancement? And perhaps we’ve
already
> > talked about this?
> >
> > (2) Config options were added in met-8.1 for the Air Force which
address
> > this issue. By default this logic is disabled but can be enabled
using
> the
> > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary was
> > also added at the same time. Below is a link to the default
point-stat
> > config file followed by an excerpt from it.
> >
> > John
> >
> >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> >
> > // Topography
> > // For SURFACE message types, only use observations where the topo
-
> > station
> > // elevation difference meets the use_obs_thresh threshold.
> > // For the observations kept, when interpolating forecast data to
the
> > // observation location, only use forecast grid points where the
topo -
> > station
> > // difference meets the interp_fcst_thresh threshold.
> > // topo_mask.flag may be set separately in each "obs.field" entry.
> > //
> > topo_mask = {
> > flag = FALSE;
> > file_name = [];
> > field = { name = "TOPO"; level = "L0"; }
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> >
> > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: point stat questions
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> >
> > >
> > >
> > > Hi John,
> > >
> > >
> > >
> > > A few questions about point stat:
> > >
> > > 1. Are grid relative winds automatically converted to
Earth
> > > relative
> > > winds?
> > >
> > > 2. For observation that fall below the model surface, are
model
> > > fields interpolated to observation location? If so, how?
> > >
> > >
> > >
> > > Thanks!
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > criminal
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Fri Jan 17 12:15:02 2020
Thanks for the quick reply. Okay, I set it to verbosity 4 and got
nothing in the way of topography. I just saw the line for rejected
topography and see that it's 0:
[tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
sfc_2019072100.nc testConfig -v 4
DEBUG 1: Default Config File: /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
DEBUG 1: User Config File: testConfig
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_Python_Numpy".
DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list contains 0
entries.
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744071898014302
DEBUG 1: Forecast File: PYTHON_NUMPY
DEBUG 1: Observation File: sfc_2019072100.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for uuwind/zht_000010.
DEBUG 3: Use the observation grid.
DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians: 0.8649
Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739 By:
-4.6309
DEBUG 2: Processing masking regions.
DEBUG 3: Processing grid mask: FULL
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2: Processing geography data.
DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0 climatology
mean levels, and 0 climatology standard deviation levels.
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Searching 18034 observations from 18034 messages.
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 4: set FCST_UNITS:UNKNOWN
DEBUG 4: set OBS_UNITS:NA
DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for observation
type SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000), using 3238 pairs.
DEBUG 3: Number of matched pairs = 3238
DEBUG 3: Observations processed = 18034
DEBUG 3: Rejected: SID exclusion = 0
DEBUG 3: Rejected: obs type = 14718
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 78
DEBUG 3: Rejected: topography = 0
DEBUG 3: Rejected: level mismatch = 0
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 0
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2: Computing Continuous Statistics.
Would a hail mary way to see if this works to set my use_obs_thresh
and interp_fcst_thresh to equal 0? That way the ob has to be exactly
at the vertical level to pass the topography mask? Ultimately I just
want to see if I am implementing the topo mask correctly
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, January 17, 2020 11:06 AM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Justin,
Running Point-stat and verbosity level 3 causes these types of log
messages
to be printed:
DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using 3779
matched pairs.
DEBUG 3: Number of matched pairs = 3779
DEBUG 3: Observations processed = 116861
DEBUG 3: Rejected: SID exclusion = 0
DEBUG 3: Rejected: obs type = 103809
DEBUG 3: Rejected: valid time = 0
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 21
DEBUG 3: Rejected: topography = 548
DEBUG 3: Rejected: level mismatch = 7675
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 809
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 220
DEBUG 3: Rejected: duplicates = 0
The "topography" line indicates the number of point observations which
were
skipped due to the topography logic. And they are skipped when...
(1) the model topo value or point obs elevation value are bad data.
(2) their difference exceeds the "use_obs_thresh" threshold
And we also write a DEBUG(4) log message about it:
DEBUG 4: Skipping observation for topography difference since 119.598
is
not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
[ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
So please try rerunning with verbosity level 4 (-v 4) and check for
log
messages about "topography" or "geography".
Your next question...
Also, how does point_stat know what netCDF variable to check terrht
against?
Point-Stat is comparing the model "terrht" against the observation
station
elevation, which is the 6th column of the 11-column point observation
format:
The "met_point" ASCII format consists of 11 columns:
Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
Lat(Deg North) Lon(Deg East) Elevation(msl)
Var_Name(or GRIB_Code) Level Height(msl or agl)
QC_String Observation_Value
This logic is used to discard point obs with large differences from
the
model elevation.
John
On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Thanks John,
>
> I just implemented your changes for a simple point_stat case
verifying
> wind direction. The point_stat debugging output looks exactly the
same as
> if I hadn't used topo_mask... how do I know if it actually worked
and
> terrain height got into point_stat?
>
> Also, how does point_stat know what netCDF variable to check terrht
> against?
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, January 9, 2020 10:27 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Justin,
>
> Sorry for losing this in the shuffle. It's a hectic time of year.
>
> So your question is whether or not Python-embedding will work when
setting
> the "topo_mask" dictionary in the Point-Stat config file. I just
tested
> this out (albeit with the development version of the code) and it
seems to
> have worked fine, just as expected. Remember though that for
> python-embedding you set the file_name to the constant string
> "PYTHON_NUMPY" and the field "name" to the python command you want
to
> execute to read the data.
>
> It looks like you were setting file_name = python command, and that
won't
> work.
>
> Thanks,
> John
>
> //
> // Topography
> //
> topo_mask = {
> flag = FALSE;
> file_name = [ "PYTHON_NUMPY" ];
> field = { name = "read_NRL_binary.py
> trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
> regrid = { method = BILIN; width = 2; }
> use_obs_thresh = ge-100&&le100;
> interp_fcst_thresh = ge-50&&le50;
> }
>
>
>
> On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Hey John,
> >
> > This email probably got lost during the holidays. Let me know if
you can
> > get around to helping me out with this problem!
> >
> > Thanks and happy new year!
> > Justin
> >
> > -----Original Message-----
> > From: Tsu, Mr. Justin
> > Sent: Friday, December 20, 2019 4:18 PM
> > To: 'met_help at ucar.edu'
> > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Thanks John,
> >
> > When using topo_mask, will point_Stat understand a python
embedded file?
> > The files we use are named '
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > These are projection gridded data of terrain heights. Would this
look
> > something like this?
> >
> > topo_mask = {
> > flag = FALSE;
> > file_name = ["./read_NRL_binary
> >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > field = { name = "TOPO"; level = "L0"; }
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> > And how does point_stat know which HDF5 varible to use as the
point of
> > comparison to the model terrain height?
> >
> > Happy holidays!
> > Justin
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, December 19, 2019 6:05 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Liz,
> >
> > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> checks
> > the flag in the header indicating whether winds are defined as
grid or
> > earth-relative. If grid relative, it searches the file for the
> correspond
> > U or V pair and uses both U and V to rotate them to be earth-
relative.
> >
> > That logic does NOT work for python embedding. I have some ideas
about
> how
> > to make it work but those changes are not slated for the met-9.0
release.
> >
> > I assume you’d like to see this enhancement? And perhaps we’ve
already
> > talked about this?
> >
> > (2) Config options were added in met-8.1 for the Air Force which
address
> > this issue. By default this logic is disabled but can be enabled
using
> the
> > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary was
> > also added at the same time. Below is a link to the default
point-stat
> > config file followed by an excerpt from it.
> >
> > John
> >
> >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> >
> > // Topography
> > // For SURFACE message types, only use observations where the topo
-
> > station
> > // elevation difference meets the use_obs_thresh threshold.
> > // For the observations kept, when interpolating forecast data to
the
> > // observation location, only use forecast grid points where the
topo -
> > station
> > // difference meets the interp_fcst_thresh threshold.
> > // topo_mask.flag may be set separately in each "obs.field" entry.
> > //
> > topo_mask = {
> > flag = FALSE;
> > file_name = [];
> > field = { name = "TOPO"; level = "L0"; }
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> >
> > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: point stat questions
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> >
> > >
> > >
> > > Hi John,
> > >
> > >
> > >
> > > A few questions about point stat:
> > >
> > > 1. Are grid relative winds automatically converted to
Earth
> > > relative
> > > winds?
> > >
> > > 2. For observation that fall below the model surface, are
model
> > > fields interpolated to observation location? If so, how?
> > >
> > >
> > >
> > > Thanks!
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > criminal
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Fri Jan 17 12:25:02 2020
Yeah definitely. You could try setting the thresholds that way. FYI,
you
should also see a "DEBUG 2" statement like this:
DEBUG 2: Processing geography data.
Feel free to send along your latest Point-Stat config file and I could
look
for issues in your settings.
Thanks,
John
On Fri, Jan 17, 2020 at 12:15 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Thanks for the quick reply. Okay, I set it to verbosity 4 and got
nothing
> in the way of topography. I just saw the line for rejected
topography and
> see that it's 0:
>
>
> [tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
sfc_2019072100.nc
> testConfig -v 4
> DEBUG 1: Default Config File:
> /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
> DEBUG 1: User Config File: testConfig
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Python_Numpy".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Python_Numpy".
> DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list contains
0
> entries.
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744071898014302
> DEBUG 1: Forecast File: PYTHON_NUMPY
> DEBUG 1: Observation File: sfc_2019072100.nc
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for uuwind/zht_000010.
> DEBUG 3: Use the observation grid.
> DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
> Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians: 0.8649
> Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739 By:
-4.6309
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2: Processing geography data.
> DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0
climatology mean
> levels, and 0 climatology standard deviation levels.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Searching 18034 observations from 18034 messages.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 4: set FCST_UNITS:UNKNOWN
> DEBUG 4: set OBS_UNITS:NA
> DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for
observation type
> SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000), using
> 3238 pairs.
> DEBUG 3: Number of matched pairs = 3238
> DEBUG 3: Observations processed = 18034
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 14718
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 78
> DEBUG 3: Rejected: topography = 0
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2: Computing Continuous Statistics.
>
> Would a hail mary way to see if this works to set my use_obs_thresh
and
> interp_fcst_thresh to equal 0? That way the ob has to be exactly at
the
> vertical level to pass the topography mask? Ultimately I just want
to see
> if I am implementing the topo mask correctly
>
> Justin
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, January 17, 2020 11:06 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Justin,
>
> Running Point-stat and verbosity level 3 causes these types of log
messages
> to be printed:
>
> DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type
ADPSFC, over
> region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using
3779
> matched pairs.
> DEBUG 3: Number of matched pairs = 3779
> DEBUG 3: Observations processed = 116861
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 103809
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 21
> DEBUG 3: Rejected: topography = 548
> DEBUG 3: Rejected: level mismatch = 7675
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 809
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 220
> DEBUG 3: Rejected: duplicates = 0
>
> The "topography" line indicates the number of point observations
which were
> skipped due to the topography logic. And they are skipped when...
> (1) the model topo value or point obs elevation value are bad data.
> (2) their difference exceeds the "use_obs_thresh" threshold
>
> And we also write a DEBUG(4) log message about it:
>
> DEBUG 4: Skipping observation for topography difference since
119.598 is
> not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
> [ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
>
> So please try rerunning with verbosity level 4 (-v 4) and check for
log
> messages about "topography" or "geography".
>
> Your next question...
> Also, how does point_stat know what netCDF variable to check
terrht
> against?
>
> Point-Stat is comparing the model "terrht" against the observation
station
> elevation, which is the 6th column of the 11-column point
observation
> format:
>
> The "met_point" ASCII format consists of 11 columns:
> Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
> Lat(Deg North) Lon(Deg East) Elevation(msl)
> Var_Name(or GRIB_Code) Level Height(msl or agl)
> QC_String Observation_Value
>
> This logic is used to discard point obs with large differences from
the
> model elevation.
>
> John
>
> On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Thanks John,
> >
> > I just implemented your changes for a simple point_stat case
verifying
> > wind direction. The point_stat debugging output looks exactly
the same
> as
> > if I hadn't used topo_mask... how do I know if it actually worked
and
> > terrain height got into point_stat?
> >
> > Also, how does point_stat know what netCDF variable to check
terrht
> > against?
> > Justin
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, January 9, 2020 10:27 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Justin,
> >
> > Sorry for losing this in the shuffle. It's a hectic time of year.
> >
> > So your question is whether or not Python-embedding will work when
> setting
> > the "topo_mask" dictionary in the Point-Stat config file. I just
tested
> > this out (albeit with the development version of the code) and it
seems
> to
> > have worked fine, just as expected. Remember though that for
> > python-embedding you set the file_name to the constant string
> > "PYTHON_NUMPY" and the field "name" to the python command you want
to
> > execute to read the data.
> >
> > It looks like you were setting file_name = python command, and
that won't
> > work.
> >
> > Thanks,
> > John
> >
> > //
> > // Topography
> > //
> > topo_mask = {
> > flag = FALSE;
> > file_name = [ "PYTHON_NUMPY" ];
> > field = { name = "read_NRL_binary.py
> > trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld";
}
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> >
> >
> >
> > On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > >
> > > Hey John,
> > >
> > > This email probably got lost during the holidays. Let me know
if you
> can
> > > get around to helping me out with this problem!
> > >
> > > Thanks and happy new year!
> > > Justin
> > >
> > > -----Original Message-----
> > > From: Tsu, Mr. Justin
> > > Sent: Friday, December 20, 2019 4:18 PM
> > > To: 'met_help at ucar.edu'
> > > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Thanks John,
> > >
> > > When using topo_mask, will point_Stat understand a python
embedded
> file?
> > > The files we use are named '
> > >
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > > These are projection gridded data of terrain heights. Would
this look
> > > something like this?
> > >
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = ["./read_NRL_binary
> > >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > > field = { name = "TOPO"; level = "L0"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > > And how does point_stat know which HDF5 varible to use as the
point of
> > > comparison to the model terrain height?
> > >
> > > Happy holidays!
> > > Justin
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, December 19, 2019 6:05 AM
> > > To: Satterfield, Dr. Elizabeth
> > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Liz,
> > >
> > > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> > checks
> > > the flag in the header indicating whether winds are defined as
grid or
> > > earth-relative. If grid relative, it searches the file for the
> > correspond
> > > U or V pair and uses both U and V to rotate them to be earth-
relative.
> > >
> > > That logic does NOT work for python embedding. I have some
ideas about
> > how
> > > to make it work but those changes are not slated for the met-9.0
> release.
> > >
> > > I assume you’d like to see this enhancement? And perhaps we’ve
already
> > > talked about this?
> > >
> > > (2) Config options were added in met-8.1 for the Air Force which
> address
> > > this issue. By default this logic is disabled but can be
enabled using
> > the
> > > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary
> was
> > > also added at the same time. Below is a link to the default
point-stat
> > > config file followed by an excerpt from it.
> > >
> > > John
> > >
> > >
> > >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> > >
> > > // Topography
> > > // For SURFACE message types, only use observations where the
topo -
> > > station
> > > // elevation difference meets the use_obs_thresh threshold.
> > > // For the observations kept, when interpolating forecast data
to the
> > > // observation location, only use forecast grid points where the
topo -
> > > station
> > > // difference meets the interp_fcst_thresh threshold.
> > > // topo_mask.flag may be set separately in each "obs.field"
entry.
> > > //
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = [];
> > > field = { name = "TOPO"; level = "L0"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > >
> > > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > > Queue: met_help
> > > > Subject: point stat questions
> > > > Owner: Nobody
> > > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> > >
> > > >
> > > >
> > > > Hi John,
> > > >
> > > >
> > > >
> > > > A few questions about point stat:
> > > >
> > > > 1. Are grid relative winds automatically converted to
Earth
> > > > relative
> > > > winds?
> > > >
> > > > 2. For observation that fall below the model surface,
are
> model
> > > > fields interpolated to observation location? If so, how?
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Liz
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Elizabeth Satterfield
> > > > Editor, Monthly Weather Review
> > > >
> > > > Chair, AMS Committee on Probability and Statistics
> > > >
> > > > Meteorologist, Code 7531
> > > > Marine Meteorology Division
> > > > U.S. Naval Research Laboratory
> > > > T 831.656.4056 F 831.656.4026
> > > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > > >
> > > >
__________________________________________________________________
> > > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > > Any misuse or unauthorized disclosure may result in both civil
and
> > > criminal
> > > > penalties.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Fri Jan 17 13:01:13 2020
Ive put all the files you need to run (USE testConfig2) what I'm
running with point-stat in "/incoming/irap/met_help/test"
I am still getting 0 rejected topography, which I find hard to believe
especially since I set my thresholds to 0.
I do see the processing geography data, but I am not sure if it's
reading in my model terrain height correctly
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, January 17, 2020 11:25 AM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Yeah definitely. You could try setting the thresholds that way. FYI,
you
should also see a "DEBUG 2" statement like this:
DEBUG 2: Processing geography data.
Feel free to send along your latest Point-Stat config file and I could
look
for issues in your settings.
Thanks,
John
On Fri, Jan 17, 2020 at 12:15 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Thanks for the quick reply. Okay, I set it to verbosity 4 and got
nothing
> in the way of topography. I just saw the line for rejected
topography and
> see that it's 0:
>
>
> [tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
sfc_2019072100.nc
> testConfig -v 4
> DEBUG 1: Default Config File:
> /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
> DEBUG 1: User Config File: testConfig
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Python_Numpy".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_Python_Numpy".
> DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list contains
0
> entries.
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744071898014302
> DEBUG 1: Forecast File: PYTHON_NUMPY
> DEBUG 1: Observation File: sfc_2019072100.nc
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for uuwind/zht_000010.
> DEBUG 3: Use the observation grid.
> DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
> Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians: 0.8649
> Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739 By:
-4.6309
> DEBUG 2: Processing masking regions.
> DEBUG 3: Processing grid mask: FULL
> DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> DEBUG 2: Processing geography data.
> DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0
climatology mean
> levels, and 0 climatology standard deviation levels.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Searching 18034 observations from 18034 messages.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 4: set FCST_UNITS:UNKNOWN
> DEBUG 4: set OBS_UNITS:NA
> DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for
observation type
> SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000), using
> 3238 pairs.
> DEBUG 3: Number of matched pairs = 3238
> DEBUG 3: Observations processed = 18034
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 14718
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 78
> DEBUG 3: Rejected: topography = 0
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2: Computing Continuous Statistics.
>
> Would a hail mary way to see if this works to set my use_obs_thresh
and
> interp_fcst_thresh to equal 0? That way the ob has to be exactly at
the
> vertical level to pass the topography mask? Ultimately I just want
to see
> if I am implementing the topo mask correctly
>
> Justin
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, January 17, 2020 11:06 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Justin,
>
> Running Point-stat and verbosity level 3 causes these types of log
messages
> to be printed:
>
> DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type
ADPSFC, over
> region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using
3779
> matched pairs.
> DEBUG 3: Number of matched pairs = 3779
> DEBUG 3: Observations processed = 116861
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 103809
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 21
> DEBUG 3: Rejected: topography = 548
> DEBUG 3: Rejected: level mismatch = 7675
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 809
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 220
> DEBUG 3: Rejected: duplicates = 0
>
> The "topography" line indicates the number of point observations
which were
> skipped due to the topography logic. And they are skipped when...
> (1) the model topo value or point obs elevation value are bad data.
> (2) their difference exceeds the "use_obs_thresh" threshold
>
> And we also write a DEBUG(4) log message about it:
>
> DEBUG 4: Skipping observation for topography difference since
119.598 is
> not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
> [ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
>
> So please try rerunning with verbosity level 4 (-v 4) and check for
log
> messages about "topography" or "geography".
>
> Your next question...
> Also, how does point_stat know what netCDF variable to check
terrht
> against?
>
> Point-Stat is comparing the model "terrht" against the observation
station
> elevation, which is the 6th column of the 11-column point
observation
> format:
>
> The "met_point" ASCII format consists of 11 columns:
> Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
> Lat(Deg North) Lon(Deg East) Elevation(msl)
> Var_Name(or GRIB_Code) Level Height(msl or agl)
> QC_String Observation_Value
>
> This logic is used to discard point obs with large differences from
the
> model elevation.
>
> John
>
> On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Thanks John,
> >
> > I just implemented your changes for a simple point_stat case
verifying
> > wind direction. The point_stat debugging output looks exactly
the same
> as
> > if I hadn't used topo_mask... how do I know if it actually worked
and
> > terrain height got into point_stat?
> >
> > Also, how does point_stat know what netCDF variable to check
terrht
> > against?
> > Justin
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, January 9, 2020 10:27 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Justin,
> >
> > Sorry for losing this in the shuffle. It's a hectic time of year.
> >
> > So your question is whether or not Python-embedding will work when
> setting
> > the "topo_mask" dictionary in the Point-Stat config file. I just
tested
> > this out (albeit with the development version of the code) and it
seems
> to
> > have worked fine, just as expected. Remember though that for
> > python-embedding you set the file_name to the constant string
> > "PYTHON_NUMPY" and the field "name" to the python command you want
to
> > execute to read the data.
> >
> > It looks like you were setting file_name = python command, and
that won't
> > work.
> >
> > Thanks,
> > John
> >
> > //
> > // Topography
> > //
> > topo_mask = {
> > flag = FALSE;
> > file_name = [ "PYTHON_NUMPY" ];
> > field = { name = "read_NRL_binary.py
> > trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld";
}
> > regrid = { method = BILIN; width = 2; }
> > use_obs_thresh = ge-100&&le100;
> > interp_fcst_thresh = ge-50&&le50;
> > }
> >
> >
> >
> > On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > >
> > > Hey John,
> > >
> > > This email probably got lost during the holidays. Let me know
if you
> can
> > > get around to helping me out with this problem!
> > >
> > > Thanks and happy new year!
> > > Justin
> > >
> > > -----Original Message-----
> > > From: Tsu, Mr. Justin
> > > Sent: Friday, December 20, 2019 4:18 PM
> > > To: 'met_help at ucar.edu'
> > > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Thanks John,
> > >
> > > When using topo_mask, will point_Stat understand a python
embedded
> file?
> > > The files we use are named '
> > >
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > > These are projection gridded data of terrain heights. Would
this look
> > > something like this?
> > >
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = ["./read_NRL_binary
> > >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > > field = { name = "TOPO"; level = "L0"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > > And how does point_stat know which HDF5 varible to use as the
point of
> > > comparison to the model terrain height?
> > >
> > > Happy holidays!
> > > Justin
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, December 19, 2019 6:05 AM
> > > To: Satterfield, Dr. Elizabeth
> > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Liz,
> > >
> > > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> > checks
> > > the flag in the header indicating whether winds are defined as
grid or
> > > earth-relative. If grid relative, it searches the file for the
> > correspond
> > > U or V pair and uses both U and V to rotate them to be earth-
relative.
> > >
> > > That logic does NOT work for python embedding. I have some
ideas about
> > how
> > > to make it work but those changes are not slated for the met-9.0
> release.
> > >
> > > I assume you’d like to see this enhancement? And perhaps we’ve
already
> > > talked about this?
> > >
> > > (2) Config options were added in met-8.1 for the Air Force which
> address
> > > this issue. By default this logic is disabled but can be
enabled using
> > the
> > > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary
> was
> > > also added at the same time. Below is a link to the default
point-stat
> > > config file followed by an excerpt from it.
> > >
> > > John
> > >
> > >
> > >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> > >
> > > // Topography
> > > // For SURFACE message types, only use observations where the
topo -
> > > station
> > > // elevation difference meets the use_obs_thresh threshold.
> > > // For the observations kept, when interpolating forecast data
to the
> > > // observation location, only use forecast grid points where the
topo -
> > > station
> > > // difference meets the interp_fcst_thresh threshold.
> > > // topo_mask.flag may be set separately in each "obs.field"
entry.
> > > //
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = [];
> > > field = { name = "TOPO"; level = "L0"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > >
> > > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > > Queue: met_help
> > > > Subject: point stat questions
> > > > Owner: Nobody
> > > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> > >
> > > >
> > > >
> > > > Hi John,
> > > >
> > > >
> > > >
> > > > A few questions about point stat:
> > > >
> > > > 1. Are grid relative winds automatically converted to
Earth
> > > > relative
> > > > winds?
> > > >
> > > > 2. For observation that fall below the model surface,
are
> model
> > > > fields interpolated to observation location? If so, how?
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Liz
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Elizabeth Satterfield
> > > > Editor, Monthly Weather Review
> > > >
> > > > Chair, AMS Committee on Probability and Statistics
> > > >
> > > > Meteorologist, Code 7531
> > > > Marine Meteorology Division
> > > > U.S. Naval Research Laboratory
> > > > T 831.656.4056 F 831.656.4026
> > > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > > >
> > > >
__________________________________________________________________
> > > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > > Any misuse or unauthorized disclosure may result in both civil
and
> > > criminal
> > > > penalties.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Fri Jan 17 13:11:01 2020
Justin,
Thanks for sending the config file. You currently have the topography
logic disabled in the config file:
topo_mask = {
flag = FALSE;
...
}
The flag must be set to TRUE to enable the application of this logic:
topo_mask = {
flag = TRUE;
...
}
Thanks,
John
On Fri, Jan 17, 2020 at 1:04 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Ive put all the files you need to run (USE testConfig2) what I'm
running
> with point-stat in "/incoming/irap/met_help/test"
>
> I am still getting 0 rejected topography, which I find hard to
believe
> especially since I set my thresholds to 0.
>
> I do see the processing geography data, but I am not sure if it's
reading
> in my model terrain height correctly
>
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, January 17, 2020 11:25 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Yeah definitely. You could try setting the thresholds that way.
FYI, you
> should also see a "DEBUG 2" statement like this:
> DEBUG 2: Processing geography data.
>
> Feel free to send along your latest Point-Stat config file and I
could look
> for issues in your settings.
>
> Thanks,
> John
>
> On Fri, Jan 17, 2020 at 12:15 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Thanks for the quick reply. Okay, I set it to verbosity 4 and got
> nothing
> > in the way of topography. I just saw the line for rejected
topography and
> > see that it's 0:
> >
> >
> > [tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
> sfc_2019072100.nc
> > testConfig -v 4
> > DEBUG 1: Default Config File:
> >
> /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
> > DEBUG 1: User Config File: testConfig
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Python_Numpy".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> > type "FileType_Python_Numpy".
> > DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list
contains 0
> > entries.
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744071898014302
> > DEBUG 1: Forecast File: PYTHON_NUMPY
> > DEBUG 1: Observation File: sfc_2019072100.nc
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Reading data for uuwind/zht_000010.
> > DEBUG 3: Use the observation grid.
> > DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
> > Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians:
0.8649
> > Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739 By:
-4.6309
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2: Processing geography data.
> > DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0
climatology
> mean
> > levels, and 0 climatology standard deviation levels.
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Searching 18034 observations from 18034 messages.
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 4: set FCST_UNITS:UNKNOWN
> > DEBUG 4: set OBS_UNITS:NA
> > DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for
observation
> type
> > SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000),
> using
> > 3238 pairs.
> > DEBUG 3: Number of matched pairs = 3238
> > DEBUG 3: Observations processed = 18034
> > DEBUG 3: Rejected: SID exclusion = 0
> > DEBUG 3: Rejected: obs type = 14718
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 0
> > DEBUG 3: Rejected: off the grid = 78
> > DEBUG 3: Rejected: topography = 0
> > DEBUG 3: Rejected: level mismatch = 0
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2: Computing Continuous Statistics.
> >
> > Would a hail mary way to see if this works to set my
use_obs_thresh and
> > interp_fcst_thresh to equal 0? That way the ob has to be exactly
at the
> > vertical level to pass the topography mask? Ultimately I just
want to
> see
> > if I am implementing the topo mask correctly
> >
> > Justin
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, January 17, 2020 11:06 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Justin,
> >
> > Running Point-stat and verbosity level 3 causes these types of log
> messages
> > to be printed:
> >
> > DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type
ADPSFC,
> over
> > region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using
3779
> > matched pairs.
> > DEBUG 3: Number of matched pairs = 3779
> > DEBUG 3: Observations processed = 116861
> > DEBUG 3: Rejected: SID exclusion = 0
> > DEBUG 3: Rejected: obs type = 103809
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 0
> > DEBUG 3: Rejected: off the grid = 21
> > DEBUG 3: Rejected: topography = 548
> > DEBUG 3: Rejected: level mismatch = 7675
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 809
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 220
> > DEBUG 3: Rejected: duplicates = 0
> >
> > The "topography" line indicates the number of point observations
which
> were
> > skipped due to the topography logic. And they are skipped when...
> > (1) the model topo value or point obs elevation value are bad
data.
> > (2) their difference exceeds the "use_obs_thresh" threshold
> >
> > And we also write a DEBUG(4) log message about it:
> >
> > DEBUG 4: Skipping observation for topography difference since
119.598 is
> > not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
> > [ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
> >
> > So please try rerunning with verbosity level 4 (-v 4) and check
for log
> > messages about "topography" or "geography".
> >
> > Your next question...
> > Also, how does point_stat know what netCDF variable to check
terrht
> > against?
> >
> > Point-Stat is comparing the model "terrht" against the observation
> station
> > elevation, which is the 6th column of the 11-column point
observation
> > format:
> >
> > The "met_point" ASCII format consists of 11 columns:
> > Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
> > Lat(Deg North) Lon(Deg East) Elevation(msl)
> > Var_Name(or GRIB_Code) Level Height(msl or agl)
> > QC_String Observation_Value
> >
> > This logic is used to discard point obs with large differences
from the
> > model elevation.
> >
> > John
> >
> > On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > >
> > > Thanks John,
> > >
> > > I just implemented your changes for a simple point_stat case
verifying
> > > wind direction. The point_stat debugging output looks exactly
the
> same
> > as
> > > if I hadn't used topo_mask... how do I know if it actually
worked and
> > > terrain height got into point_stat?
> > >
> > > Also, how does point_stat know what netCDF variable to check
terrht
> > > against?
> > > Justin
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, January 9, 2020 10:27 AM
> > > To: Satterfield, Dr. Elizabeth
> > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Justin,
> > >
> > > Sorry for losing this in the shuffle. It's a hectic time of
year.
> > >
> > > So your question is whether or not Python-embedding will work
when
> > setting
> > > the "topo_mask" dictionary in the Point-Stat config file. I
just
> tested
> > > this out (albeit with the development version of the code) and
it seems
> > to
> > > have worked fine, just as expected. Remember though that for
> > > python-embedding you set the file_name to the constant string
> > > "PYTHON_NUMPY" and the field "name" to the python command you
want to
> > > execute to read the data.
> > >
> > > It looks like you were setting file_name = python command, and
that
> won't
> > > work.
> > >
> > > Thanks,
> > > John
> > >
> > > //
> > > // Topography
> > > //
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = [ "PYTHON_NUMPY" ];
> > > field = { name = "read_NRL_binary.py
> > >
trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > >
> > >
> > >
> > > On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
>
> > > >
> > > > Hey John,
> > > >
> > > > This email probably got lost during the holidays. Let me know
if you
> > can
> > > > get around to helping me out with this problem!
> > > >
> > > > Thanks and happy new year!
> > > > Justin
> > > >
> > > > -----Original Message-----
> > > > From: Tsu, Mr. Justin
> > > > Sent: Friday, December 20, 2019 4:18 PM
> > > > To: 'met_help at ucar.edu'
> > > > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> > > >
> > > > Thanks John,
> > > >
> > > > When using topo_mask, will point_Stat understand a python
embedded
> > file?
> > > > The files we use are named '
> > > >
> > >
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > > > These are projection gridded data of terrain heights. Would
this
> look
> > > > something like this?
> > > >
> > > > topo_mask = {
> > > > flag = FALSE;
> > > > file_name = ["./read_NRL_binary
> > > >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > > > field = { name = "TOPO"; level = "L0"; }
> > > > regrid = { method = BILIN; width = 2; }
> > > > use_obs_thresh = ge-100&&le100;
> > > > interp_fcst_thresh = ge-50&&le50;
> > > > }
> > > > And how does point_stat know which HDF5 varible to use as the
point
> of
> > > > comparison to the model terrain height?
> > > >
> > > > Happy holidays!
> > > > Justin
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, December 19, 2019 6:05 AM
> > > > To: Satterfield, Dr. Elizabeth
> > > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > > >
> > > > Liz,
> > > >
> > > > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> > > checks
> > > > the flag in the header indicating whether winds are defined as
grid
> or
> > > > earth-relative. If grid relative, it searches the file for
the
> > > correspond
> > > > U or V pair and uses both U and V to rotate them to be
> earth-relative.
> > > >
> > > > That logic does NOT work for python embedding. I have some
ideas
> about
> > > how
> > > > to make it work but those changes are not slated for the met-
9.0
> > release.
> > > >
> > > > I assume you’d like to see this enhancement? And perhaps
we’ve
> already
> > > > talked about this?
> > > >
> > > > (2) Config options were added in met-8.1 for the Air Force
which
> > address
> > > > this issue. By default this logic is disabled but can be
enabled
> using
> > > the
> > > > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary
> > was
> > > > also added at the same time. Below is a link to the default
> point-stat
> > > > config file followed by an excerpt from it.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> > > >
> > > > // Topography
> > > > // For SURFACE message types, only use observations where the
topo -
> > > > station
> > > > // elevation difference meets the use_obs_thresh threshold.
> > > > // For the observations kept, when interpolating forecast data
to the
> > > > // observation location, only use forecast grid points where
the
> topo -
> > > > station
> > > > // difference meets the interp_fcst_thresh threshold.
> > > > // topo_mask.flag may be set separately in each "obs.field"
entry.
> > > > //
> > > > topo_mask = {
> > > > flag = FALSE;
> > > > file_name = [];
> > > > field = { name = "TOPO"; level = "L0"; }
> > > > regrid = { method = BILIN; width = 2; }
> > > > use_obs_thresh = ge-100&&le100;
> > > > interp_fcst_thresh = ge-50&&le50;
> > > > }
> > > >
> > > > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > > > Transaction: Ticket created by
> Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > Queue: met_help
> > > > > Subject: point stat questions
> > > > > Owner: Nobody
> > > > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> > > >
> > > > >
> > > > >
> > > > > Hi John,
> > > > >
> > > > >
> > > > >
> > > > > A few questions about point stat:
> > > > >
> > > > > 1. Are grid relative winds automatically converted
to Earth
> > > > > relative
> > > > > winds?
> > > > >
> > > > > 2. For observation that fall below the model surface,
are
> > model
> > > > > fields interpolated to observation location? If so, how?
> > > > >
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Liz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Elizabeth Satterfield
> > > > > Editor, Monthly Weather Review
> > > > >
> > > > > Chair, AMS Committee on Probability and Statistics
> > > > >
> > > > > Meteorologist, Code 7531
> > > > > Marine Meteorology Division
> > > > > U.S. Naval Research Laboratory
> > > > > T 831.656.4056 F 831.656.4026
> > > > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > > > >
> > > > >
__________________________________________________________________
> > > > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > > > Any misuse or unauthorized disclosure may result in both
civil and
> > > > criminal
> > > > > penalties.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: Tsu, Mr. Justin
Time: Fri Jan 17 13:13:37 2020
Awesome,
It is working now. Thanks a bunch John!
Justin
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, January 17, 2020 12:11 PM
To: Satterfield, Dr. Elizabeth
Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
Justin,
Thanks for sending the config file. You currently have the topography
logic disabled in the config file:
topo_mask = {
flag = FALSE;
...
}
The flag must be set to TRUE to enable the application of this logic:
topo_mask = {
flag = TRUE;
...
}
Thanks,
John
On Fri, Jan 17, 2020 at 1:04 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Ive put all the files you need to run (USE testConfig2) what I'm
running
> with point-stat in "/incoming/irap/met_help/test"
>
> I am still getting 0 rejected topography, which I find hard to
believe
> especially since I set my thresholds to 0.
>
> I do see the processing geography data, but I am not sure if it's
reading
> in my model terrain height correctly
>
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, January 17, 2020 11:25 AM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Yeah definitely. You could try setting the thresholds that way.
FYI, you
> should also see a "DEBUG 2" statement like this:
> DEBUG 2: Processing geography data.
>
> Feel free to send along your latest Point-Stat config file and I
could look
> for issues in your settings.
>
> Thanks,
> John
>
> On Fri, Jan 17, 2020 at 12:15 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Thanks for the quick reply. Okay, I set it to verbosity 4 and got
> nothing
> > in the way of topography. I just saw the line for rejected
topography and
> > see that it's 0:
> >
> >
> > [tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
> sfc_2019072100.nc
> > testConfig -v 4
> > DEBUG 1: Default Config File:
> >
> /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
> > DEBUG 1: User Config File: testConfig
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Python_Numpy".
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> > type "FileType_Python_Numpy".
> > DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list
contains 0
> > entries.
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744071898014302
> > DEBUG 1: Forecast File: PYTHON_NUMPY
> > DEBUG 1: Observation File: sfc_2019072100.nc
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Reading data for uuwind/zht_000010.
> > DEBUG 3: Use the observation grid.
> > DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
> > Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians:
0.8649
> > Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739 By:
-4.6309
> > DEBUG 2: Processing masking regions.
> > DEBUG 3: Processing grid mask: FULL
> > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > DEBUG 2: Processing geography data.
> > DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0
climatology
> mean
> > levels, and 0 climatology standard deviation levels.
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Searching 18034 observations from 18034 messages.
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 4: set FCST_UNITS:UNKNOWN
> > DEBUG 4: set OBS_UNITS:NA
> > DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for
observation
> type
> > SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000),
> using
> > 3238 pairs.
> > DEBUG 3: Number of matched pairs = 3238
> > DEBUG 3: Observations processed = 18034
> > DEBUG 3: Rejected: SID exclusion = 0
> > DEBUG 3: Rejected: obs type = 14718
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 0
> > DEBUG 3: Rejected: off the grid = 78
> > DEBUG 3: Rejected: topography = 0
> > DEBUG 3: Rejected: level mismatch = 0
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2: Computing Continuous Statistics.
> >
> > Would a hail mary way to see if this works to set my
use_obs_thresh and
> > interp_fcst_thresh to equal 0? That way the ob has to be exactly
at the
> > vertical level to pass the topography mask? Ultimately I just
want to
> see
> > if I am implementing the topo mask correctly
> >
> > Justin
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, January 17, 2020 11:06 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Justin,
> >
> > Running Point-stat and verbosity level 3 causes these types of log
> messages
> > to be printed:
> >
> > DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type
ADPSFC,
> over
> > region FULL, for interpolation method GEOG_MATCH_SQUARE(9), using
3779
> > matched pairs.
> > DEBUG 3: Number of matched pairs = 3779
> > DEBUG 3: Observations processed = 116861
> > DEBUG 3: Rejected: SID exclusion = 0
> > DEBUG 3: Rejected: obs type = 103809
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 0
> > DEBUG 3: Rejected: off the grid = 21
> > DEBUG 3: Rejected: topography = 548
> > DEBUG 3: Rejected: level mismatch = 7675
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 809
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 220
> > DEBUG 3: Rejected: duplicates = 0
> >
> > The "topography" line indicates the number of point observations
which
> were
> > skipped due to the topography logic. And they are skipped when...
> > (1) the model topo value or point obs elevation value are bad
data.
> > (2) their difference exceeds the "use_obs_thresh" threshold
> >
> > And we also write a DEBUG(4) log message about it:
> >
> > DEBUG 4: Skipping observation for topography difference since
119.598 is
> > not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
> > [ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
> >
> > So please try rerunning with verbosity level 4 (-v 4) and check
for log
> > messages about "topography" or "geography".
> >
> > Your next question...
> > Also, how does point_stat know what netCDF variable to check
terrht
> > against?
> >
> > Point-Stat is comparing the model "terrht" against the observation
> station
> > elevation, which is the 6th column of the 11-column point
observation
> > format:
> >
> > The "met_point" ASCII format consists of 11 columns:
> > Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS)
> > Lat(Deg North) Lon(Deg East) Elevation(msl)
> > Var_Name(or GRIB_Code) Level Height(msl or agl)
> > QC_String Observation_Value
> >
> > This logic is used to discard point obs with large differences
from the
> > model elevation.
> >
> > John
> >
> > On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > >
> > > Thanks John,
> > >
> > > I just implemented your changes for a simple point_stat case
verifying
> > > wind direction. The point_stat debugging output looks exactly
the
> same
> > as
> > > if I hadn't used topo_mask... how do I know if it actually
worked and
> > > terrain height got into point_stat?
> > >
> > > Also, how does point_stat know what netCDF variable to check
terrht
> > > against?
> > > Justin
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, January 9, 2020 10:27 AM
> > > To: Satterfield, Dr. Elizabeth
> > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Justin,
> > >
> > > Sorry for losing this in the shuffle. It's a hectic time of
year.
> > >
> > > So your question is whether or not Python-embedding will work
when
> > setting
> > > the "topo_mask" dictionary in the Point-Stat config file. I
just
> tested
> > > this out (albeit with the development version of the code) and
it seems
> > to
> > > have worked fine, just as expected. Remember though that for
> > > python-embedding you set the file_name to the constant string
> > > "PYTHON_NUMPY" and the field "name" to the python command you
want to
> > > execute to read the data.
> > >
> > > It looks like you were setting file_name = python command, and
that
> won't
> > > work.
> > >
> > > Thanks,
> > > John
> > >
> > > //
> > > // Topography
> > > //
> > > topo_mask = {
> > > flag = FALSE;
> > > file_name = [ "PYTHON_NUMPY" ];
> > > field = { name = "read_NRL_binary.py
> > >
trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
> > > regrid = { method = BILIN; width = 2; }
> > > use_obs_thresh = ge-100&&le100;
> > > interp_fcst_thresh = ge-50&&le50;
> > > }
> > >
> > >
> > >
> > > On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
>
> > > >
> > > > Hey John,
> > > >
> > > > This email probably got lost during the holidays. Let me know
if you
> > can
> > > > get around to helping me out with this problem!
> > > >
> > > > Thanks and happy new year!
> > > > Justin
> > > >
> > > > -----Original Message-----
> > > > From: Tsu, Mr. Justin
> > > > Sent: Friday, December 20, 2019 4:18 PM
> > > > To: 'met_help at ucar.edu'
> > > > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> > > >
> > > > Thanks John,
> > > >
> > > > When using topo_mask, will point_Stat understand a python
embedded
> > file?
> > > > The files we use are named '
> > > >
> > >
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > > > These are projection gridded data of terrain heights. Would
this
> look
> > > > something like this?
> > > >
> > > > topo_mask = {
> > > > flag = FALSE;
> > > > file_name = ["./read_NRL_binary
> > > >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > > > field = { name = "TOPO"; level = "L0"; }
> > > > regrid = { method = BILIN; width = 2; }
> > > > use_obs_thresh = ge-100&&le100;
> > > > interp_fcst_thresh = ge-50&&le50;
> > > > }
> > > > And how does point_stat know which HDF5 varible to use as the
point
> of
> > > > comparison to the model terrain height?
> > > >
> > > > Happy holidays!
> > > > Justin
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, December 19, 2019 6:05 AM
> > > > To: Satterfield, Dr. Elizabeth
> > > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > > >
> > > > Liz,
> > > >
> > > > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> > > checks
> > > > the flag in the header indicating whether winds are defined as
grid
> or
> > > > earth-relative. If grid relative, it searches the file for
the
> > > correspond
> > > > U or V pair and uses both U and V to rotate them to be
> earth-relative.
> > > >
> > > > That logic does NOT work for python embedding. I have some
ideas
> about
> > > how
> > > > to make it work but those changes are not slated for the met-
9.0
> > release.
> > > >
> > > > I assume you’d like to see this enhancement? And perhaps
we’ve
> already
> > > > talked about this?
> > > >
> > > > (2) Config options were added in met-8.1 for the Air Force
which
> > address
> > > > this issue. By default this logic is disabled but can be
enabled
> using
> > > the
> > > > “topo_mask” config file dictionary. FYI, the “land_mask”
dictionary
> > was
> > > > also added at the same time. Below is a link to the default
> point-stat
> > > > config file followed by an excerpt from it.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> > > >
> > > > // Topography
> > > > // For SURFACE message types, only use observations where the
topo -
> > > > station
> > > > // elevation difference meets the use_obs_thresh threshold.
> > > > // For the observations kept, when interpolating forecast data
to the
> > > > // observation location, only use forecast grid points where
the
> topo -
> > > > station
> > > > // difference meets the interp_fcst_thresh threshold.
> > > > // topo_mask.flag may be set separately in each "obs.field"
entry.
> > > > //
> > > > topo_mask = {
> > > > flag = FALSE;
> > > > file_name = [];
> > > > field = { name = "TOPO"; level = "L0"; }
> > > > regrid = { method = BILIN; width = 2; }
> > > > use_obs_thresh = ge-100&&le100;
> > > > interp_fcst_thresh = ge-50&&le50;
> > > > }
> > > >
> > > > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > > > Transaction: Ticket created by
> Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > Queue: met_help
> > > > > Subject: point stat questions
> > > > > Owner: Nobody
> > > > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> > > >
> > > > >
> > > > >
> > > > > Hi John,
> > > > >
> > > > >
> > > > >
> > > > > A few questions about point stat:
> > > > >
> > > > > 1. Are grid relative winds automatically converted
to Earth
> > > > > relative
> > > > > winds?
> > > > >
> > > > > 2. For observation that fall below the model surface,
are
> > model
> > > > > fields interpolated to observation location? If so, how?
> > > > >
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Liz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Elizabeth Satterfield
> > > > > Editor, Monthly Weather Review
> > > > >
> > > > > Chair, AMS Committee on Probability and Statistics
> > > > >
> > > > > Meteorologist, Code 7531
> > > > > Marine Meteorology Division
> > > > > U.S. Naval Research Laboratory
> > > > > T 831.656.4056 F 831.656.4026
> > > > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > > > >
> > > > >
__________________________________________________________________
> > > > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > > > Any misuse or unauthorized disclosure may result in both
civil and
> > > > criminal
> > > > > penalties.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: point stat questions
From: John Halley Gotway
Time: Fri Jan 17 14:11:01 2020
Glad to hear it!
John
On Fri, Jan 17, 2020 at 1:14 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
>
> Awesome,
>
> It is working now. Thanks a bunch John!
>
> Justin
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, January 17, 2020 12:11 PM
> To: Satterfield, Dr. Elizabeth
> Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
>
> Justin,
>
> Thanks for sending the config file. You currently have the
topography
> logic disabled in the config file:
>
> topo_mask = {
> flag = FALSE;
> ...
> }
>
> The flag must be set to TRUE to enable the application of this
logic:
> topo_mask = {
> flag = TRUE;
> ...
> }
>
> Thanks,
> John
>
> On Fri, Jan 17, 2020 at 1:04 PM Tsu, Mr. Justin via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> >
> > Ive put all the files you need to run (USE testConfig2) what I'm
running
> > with point-stat in "/incoming/irap/met_help/test"
> >
> > I am still getting 0 rejected topography, which I find hard to
believe
> > especially since I set my thresholds to 0.
> >
> > I do see the processing geography data, but I am not sure if it's
reading
> > in my model terrain height correctly
> >
> > Justin
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, January 17, 2020 11:25 AM
> > To: Satterfield, Dr. Elizabeth
> > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> >
> > Yeah definitely. You could try setting the thresholds that way.
FYI,
> you
> > should also see a "DEBUG 2" statement like this:
> > DEBUG 2: Processing geography data.
> >
> > Feel free to send along your latest Point-Stat config file and I
could
> look
> > for issues in your settings.
> >
> > Thanks,
> > John
> >
> > On Fri, Jan 17, 2020 at 12:15 PM Tsu, Mr. Justin via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > >
> > > Thanks for the quick reply. Okay, I set it to verbosity 4 and
got
> > nothing
> > > in the way of topography. I just saw the line for rejected
topography
> and
> > > see that it's 0:
> > >
> > >
> > > [tsu at lorenz 02_POINT_STAT_WORK]$ point_stat PYTHON_NUMPY
> > sfc_2019072100.nc
> > > testConfig -v 4
> > > DEBUG 1: Default Config File:
> > >
> >
> /software/depot/met-8.1a/met-
8.1a/share/met//config/PointStatConfig_default
> > > DEBUG 1: User Config File: testConfig
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_Python_Numpy".
> > > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of
> > > type "FileType_Python_Numpy".
> > > DEBUG 4: parse_conf_sid_exc() -> Station ID exclusion list
contains 0
> > > entries.
> > > GSL_RNG_TYPE=mt19937
> > > GSL_RNG_SEED=18446744071898014302
> > > DEBUG 1: Forecast File: PYTHON_NUMPY
> > > DEBUG 1: Observation File: sfc_2019072100.nc
> > > DEBUG 2:
> > > DEBUG 2:
> > >
> >
>
--------------------------------------------------------------------------------
> > > DEBUG 2:
> > > DEBUG 2: Reading data for uuwind/zht_000010.
> > > DEBUG 3: Use the observation grid.
> > > DEBUG 3: Grid Definition: Projection: Mercator Nx: 127 Ny: 127
> > > Lat_LL_radians: 0.0354 Lon_LL_radians: 1.9401 Lat_UR_radians:
0.8649
> > > Lon_UR_radians: 0.9768 Mx: 130.8075 My: 130.8067 Bx: 253.7739
By:
> -4.6309
> > > DEBUG 2: Processing masking regions.
> > > DEBUG 3: Processing grid mask: FULL
> > > DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
> > > DEBUG 2: Processing geography data.
> > > DEBUG 2: For uuwind/zht_000010 found 1 forecast levels, 0
climatology
> > mean
> > > levels, and 0 climatology standard deviation levels.
> > > DEBUG 2:
> > > DEBUG 2:
> > >
> >
>
--------------------------------------------------------------------------------
> > > DEBUG 2:
> > > DEBUG 2: Searching 18034 observations from 18034 messages.
> > > DEBUG 2:
> > > DEBUG 2:
> > >
> >
>
--------------------------------------------------------------------------------
> > > DEBUG 2:
> > > DEBUG 4: set FCST_UNITS:UNKNOWN
> > > DEBUG 4: set OBS_UNITS:NA
> > > DEBUG 2: Processing uuwind/zht_000010 versus wdir/Z2, for
observation
> > type
> > > SYNOP, over region FULL, for interpolation method
BEST_SQUARE(10000),
> > using
> > > 3238 pairs.
> > > DEBUG 3: Number of matched pairs = 3238
> > > DEBUG 3: Observations processed = 18034
> > > DEBUG 3: Rejected: SID exclusion = 0
> > > DEBUG 3: Rejected: obs type = 14718
> > > DEBUG 3: Rejected: valid time = 0
> > > DEBUG 3: Rejected: bad obs value = 0
> > > DEBUG 3: Rejected: off the grid = 78
> > > DEBUG 3: Rejected: topography = 0
> > > DEBUG 3: Rejected: level mismatch = 0
> > > DEBUG 3: Rejected: quality marker = 0
> > > DEBUG 3: Rejected: message type = 0
> > > DEBUG 3: Rejected: masking region = 0
> > > DEBUG 3: Rejected: bad fcst value = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > > DEBUG 2: Computing Continuous Statistics.
> > >
> > > Would a hail mary way to see if this works to set my
use_obs_thresh and
> > > interp_fcst_thresh to equal 0? That way the ob has to be exactly
at the
> > > vertical level to pass the topography mask? Ultimately I just
want to
> > see
> > > if I am implementing the topo mask correctly
> > >
> > > Justin
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, January 17, 2020 11:06 AM
> > > To: Satterfield, Dr. Elizabeth
> > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > >
> > > Justin,
> > >
> > > Running Point-stat and verbosity level 3 causes these types of
log
> > messages
> > > to be printed:
> > >
> > > DEBUG 2: Processing TMP/Z2 versus TMP/Z2, for observation type
ADPSFC,
> > over
> > > region FULL, for interpolation method GEOG_MATCH_SQUARE(9),
using 3779
> > > matched pairs.
> > > DEBUG 3: Number of matched pairs = 3779
> > > DEBUG 3: Observations processed = 116861
> > > DEBUG 3: Rejected: SID exclusion = 0
> > > DEBUG 3: Rejected: obs type = 103809
> > > DEBUG 3: Rejected: valid time = 0
> > > DEBUG 3: Rejected: bad obs value = 0
> > > DEBUG 3: Rejected: off the grid = 21
> > > DEBUG 3: Rejected: topography = 548
> > > DEBUG 3: Rejected: level mismatch = 7675
> > > DEBUG 3: Rejected: quality marker = 0
> > > DEBUG 3: Rejected: message type = 809
> > > DEBUG 3: Rejected: masking region = 0
> > > DEBUG 3: Rejected: bad fcst value = 220
> > > DEBUG 3: Rejected: duplicates = 0
> > >
> > > The "topography" line indicates the number of point observations
which
> > were
> > > skipped due to the topography logic. And they are skipped
when...
> > > (1) the model topo value or point obs elevation value are bad
data.
> > > (2) their difference exceeds the "use_obs_thresh" threshold
> > >
> > > And we also write a DEBUG(4) log message about it:
> > >
> > > DEBUG 4: Skipping observation for topography difference since
119.598
> is
> > > not >=-100&&<=100 for [msg_typ:sid:lat:lon:elevation] =
> > > [ADPSFC:71069:55.28:114.78:581] and model topography = 700.598.
> > >
> > > So please try rerunning with verbosity level 4 (-v 4) and check
for log
> > > messages about "topography" or "geography".
> > >
> > > Your next question...
> > > Also, how does point_stat know what netCDF variable to check
terrht
> > > against?
> > >
> > > Point-Stat is comparing the model "terrht" against the
observation
> > station
> > > elevation, which is the 6th column of the 11-column point
observation
> > > format:
> > >
> > > The "met_point" ASCII format consists of 11 columns:
> > > Message_Type Station_ID
Valid_Time(YYYYMMDD_HHMMSS)
> > > Lat(Deg North) Lon(Deg East) Elevation(msl)
> > > Var_Name(or GRIB_Code) Level Height(msl or agl)
> > > QC_String Observation_Value
> > >
> > > This logic is used to discard point obs with large differences
from the
> > > model elevation.
> > >
> > > John
> > >
> > > On Fri, Jan 17, 2020 at 11:26 AM Tsu, Mr. Justin via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
>
> > > >
> > > > Thanks John,
> > > >
> > > > I just implemented your changes for a simple point_stat case
> verifying
> > > > wind direction. The point_stat debugging output looks
exactly the
> > same
> > > as
> > > > if I hadn't used topo_mask... how do I know if it actually
worked and
> > > > terrain height got into point_stat?
> > > >
> > > > Also, how does point_stat know what netCDF variable to check
terrht
> > > > against?
> > > > Justin
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, January 9, 2020 10:27 AM
> > > > To: Satterfield, Dr. Elizabeth
> > > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > > >
> > > > Justin,
> > > >
> > > > Sorry for losing this in the shuffle. It's a hectic time of
year.
> > > >
> > > > So your question is whether or not Python-embedding will work
when
> > > setting
> > > > the "topo_mask" dictionary in the Point-Stat config file. I
just
> > tested
> > > > this out (albeit with the development version of the code) and
it
> seems
> > > to
> > > > have worked fine, just as expected. Remember though that for
> > > > python-embedding you set the file_name to the constant string
> > > > "PYTHON_NUMPY" and the field "name" to the python command you
want to
> > > > execute to read the data.
> > > >
> > > > It looks like you were setting file_name = python command, and
that
> > won't
> > > > work.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > //
> > > > // Topography
> > > > //
> > > > topo_mask = {
> > > > flag = FALSE;
> > > > file_name = [ "PYTHON_NUMPY" ];
> > > > field = { name = "read_NRL_binary.py
> > > >
trpres_sfc_0000.0_0000.0_glob360x181_2013022712_00000000_fcstfld"; }
> > > > regrid = { method = BILIN; width = 2; }
> > > > use_obs_thresh = ge-100&&le100;
> > > > interp_fcst_thresh = ge-50&&le50;
> > > > }
> > > >
> > > >
> > > >
> > > > On Wed, Jan 8, 2020 at 2:32 PM Tsu, Mr. Justin via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > This email probably got lost during the holidays. Let me
know if
> you
> > > can
> > > > > get around to helping me out with this problem!
> > > > >
> > > > > Thanks and happy new year!
> > > > > Justin
> > > > >
> > > > > -----Original Message-----
> > > > > From: Tsu, Mr. Justin
> > > > > Sent: Friday, December 20, 2019 4:18 PM
> > > > > To: 'met_help at ucar.edu'
> > > > > Subject: RE: [rt.rap.ucar.edu #93532] point stat questions
> > > > >
> > > > > Thanks John,
> > > > >
> > > > > When using topo_mask, will point_Stat understand a python
embedded
> > > file?
> > > > > The files we use are named '
> > > > >
> > > >
> > >
> >
>
/DATA/tsu/repos/flds/terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld'.
> > > > > These are projection gridded data of terrain heights.
Would this
> > look
> > > > > something like this?
> > > > >
> > > > > topo_mask = {
> > > > > flag = FALSE;
> > > > > file_name = ["./read_NRL_binary
> > > > >
terrht_sfc_000000_000000_4a0331x0331_2019072800_00000000_fcstfld"];
> > > > > field = { name = "TOPO"; level = "L0"; }
> > > > > regrid = { method = BILIN; width = 2; }
> > > > > use_obs_thresh = ge-100&&le100;
> > > > > interp_fcst_thresh = ge-50&&le50;
> > > > > }
> > > > > And how does point_stat know which HDF5 varible to use as
the point
> > of
> > > > > comparison to the model terrain height?
> > > > >
> > > > > Happy holidays!
> > > > > Justin
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Thursday, December 19, 2019 6:05 AM
> > > > > To: Satterfield, Dr. Elizabeth
> > > > > Cc: Nachamkin, Dr. Jason; Tsu, Mr. Justin
> > > > > Subject: Re: [rt.rap.ucar.edu #93532] point stat questions
> > > > >
> > > > > Liz,
> > > > >
> > > > > (1) When reading data from GRIB1 and GRIB2 files, MET
automatically
> > > > checks
> > > > > the flag in the header indicating whether winds are defined
as grid
> > or
> > > > > earth-relative. If grid relative, it searches the file for
the
> > > > correspond
> > > > > U or V pair and uses both U and V to rotate them to be
> > earth-relative.
> > > > >
> > > > > That logic does NOT work for python embedding. I have some
ideas
> > about
> > > > how
> > > > > to make it work but those changes are not slated for the
met-9.0
> > > release.
> > > > >
> > > > > I assume you’d like to see this enhancement? And perhaps
we’ve
> > already
> > > > > talked about this?
> > > > >
> > > > > (2) Config options were added in met-8.1 for the Air Force
which
> > > address
> > > > > this issue. By default this logic is disabled but can be
enabled
> > using
> > > > the
> > > > > “topo_mask” config file dictionary. FYI, the “land_mask”
> dictionary
> > > was
> > > > > also added at the same time. Below is a link to the default
> > point-stat
> > > > > config file followed by an excerpt from it.
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/NCAR/MET/blob/master_v8.1/met/data/config/PointStatConfig_default
> > > > >
> > > > > // Topography
> > > > > // For SURFACE message types, only use observations where
the topo
> -
> > > > > station
> > > > > // elevation difference meets the use_obs_thresh threshold.
> > > > > // For the observations kept, when interpolating forecast
data to
> the
> > > > > // observation location, only use forecast grid points where
the
> > topo -
> > > > > station
> > > > > // difference meets the interp_fcst_thresh threshold.
> > > > > // topo_mask.flag may be set separately in each "obs.field"
entry.
> > > > > //
> > > > > topo_mask = {
> > > > > flag = FALSE;
> > > > > file_name = [];
> > > > > field = { name = "TOPO"; level = "L0"; }
> > > > > regrid = { method = BILIN; width = 2; }
> > > > > use_obs_thresh = ge-100&&le100;
> > > > > interp_fcst_thresh = ge-50&&le50;
> > > > > }
> > > > >
> > > > > On Wed, Dec 18, 2019 at 6:37 PM Satterfield, Dr. Elizabeth
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Dec 18 18:37:22 2019: Request 93532 was acted upon.
> > > > > > Transaction: Ticket created by
> > Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > > Queue: met_help
> > > > > > Subject: point stat questions
> > > > > > Owner: Nobody
> > > > > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93532
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > >
> > > > > >
> > > > > > A few questions about point stat:
> > > > > >
> > > > > > 1. Are grid relative winds automatically converted
to
> Earth
> > > > > > relative
> > > > > > winds?
> > > > > >
> > > > > > 2. For observation that fall below the model
surface, are
> > > model
> > > > > > fields interpolated to observation location? If so, how?
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Liz
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Elizabeth Satterfield
> > > > > > Editor, Monthly Weather Review
> > > > > >
> > > > > > Chair, AMS Committee on Probability and Statistics
> > > > > >
> > > > > > Meteorologist, Code 7531
> > > > > > Marine Meteorology Division
> > > > > > U.S. Naval Research Laboratory
> > > > > > T 831.656.4056 F 831.656.4026
> > > > > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > > > > >
> > > > > >
> __________________________________________________________________
> > > > > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > > > > Any misuse or unauthorized disclosure may result in both
civil
> and
> > > > > criminal
> > > > > > penalties.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list