[Met_help] [rt.rap.ucar.edu #88386] History for GRIB2 Probabilities in Grid-Stat
John Halley Gotway via RT
met_help at ucar.edu
Fri Jan 11 14:34:28 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
A new issue for John or anyone else is a problem where I am degribbing a GRIB 2 file of forecast probabilities. I am using the notation below in the grid stat configuration file for get 500mb HGT probabilities.
fcst = {
field = [
{ name = "PROB"; level = "P500";
prob = { name = "HGT"; thresh_hi = 5400; } },
{ name = "PROB"; level = "P700";
prob = { name = "HGT"; thresh_hi = 3000; } },
{ name = "PROB"; level = "P500";
prob = { name = "HGT"; thresh_lo = 5800; } }
];
cat_thresh=[==0.1];
}
In the grib2 file, the table of contents is listed below, there are multiple 500MB HGT entries which include the probability I am after. The // are data I left out for clarity. I highlighted the record I am after.
1:0:d=2018112600:PRES:surface:6 hour fcst:mean all members
2:552511:d=2018112600:PRES:surface:6 hour fcst:std dev (cluster mean)
3:975062:d=2018112600:HGT:200 mb:6 hour fcst:mean all members
4:1365123:d=2018112600:HGT:250 mb:6 hour fcst:mean all members
5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
6:2112755:d=2018112600:HGT:700 mb:6 hour fcst:mean all members
7:2437836:d=2018112600:HGT:850 mb:6 hour fcst:mean all members
8:2762917:d=2018112600:HGT:925 mb:6 hour fcst:mean all members
//
12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster mean)
//
148:29723303:d=2018112600:TMP:2 m above ground:24 hour fcst:prob <273.15:probability forecast
149:29950925:d=2018112600:TMP:2 m above ground:24 hour fcst:prob >305.37:probability forecast
150:30178547:d=2018112600:HGT:500 mb:24 hour fcst:prob >5800:probability forecast
151:30406169:d=2018112600:GEOLAT:surface:24 hour fcst:
152:30666268:d=2018112600:GEOLON:surface:24 hour fcst:
153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob <5400:probability forecast
154:31186479:d=2018112600:HGT:700 mb:24 hour fcst:prob <3000:probability forecast
155:31414101:d=2018112600:PRMSL:mean sea level:24 hour fcst:prob >102500:probability forecast
Below is the MET out listing for this:
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Gb2".
DEBUG 3:
DEBUG 3: Multiple GRIB2 table entries match lookup criteria (parm_name = HGT):
DEBUG 3: parm_name: HGT, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3: parm_name: HGT, index_a = 0, grib2_mtab = 14, grib2_cntr = 57, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3: Using the first match found: parm_name: HGT, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3:
DEBUG 3:
DEBUG 3: Multiple GRIB2 table entries match lookup criteria (parm_name = HGT):
DEBUG 3: parm_name: HGT, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3: parm_name: HGT, index_a = 0, grib2_mtab = 14, grib2_cntr = 57, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3: Using the first match found: parm_name: HGT, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 3, index_c = 5
DEBUG 3:
DEBUG 3: Use the forecast grid.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361 lat_ll: -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=249568092
DEBUG 1: Forecast File: /h/data/global/ENSEMBLE/data/post/GEPS/global/2018112600/grib2.2018112600.0024.EPOSTOUT
DEBUG 1: Observation File: /h/data/global/WXQC/data/galwem/1811270000
DEBUG 3: Reading forecast data for PROB(HGT<5400.000)/P500/PROB.
WARNING:
WARNING: MetGrib2DataFile::data_plane() - Multiple matching records found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using 5
WARNING:
DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 5 field 1 of GRIB2 file '/h/data/global/ENSEMBLE/data/post/GEPS/global/2018112600/grib2.2018112600.0024.EPOSTOUT'
Notice it finds multiple records and chooses the first on it sees (5) instead of the correct one (153). Record 5 and 12 are not probability forecasts so why is it considering them? If I pull using the record number I will get the correct record but this can’t be done for automated runs.
Thanks for your help.
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: John Halley Gotway
Time: Thu Jan 03 14:35:03 2019
Hi Bob,
I saw you asked a new question on an older ticket. For clarity, I
pulled that question out and used it to make this new ticket.
I see that Grid-Stat is not selecting out the records you intend.
Based on the log messages, it finds 4 matches:
WARNING: MetGrib2DataFile::data_plane() - Multiple matching records
found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using 5
5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster mean)
153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
<5400:probability forecast
213: ?
I agree that it should only be considering the PROB forecast types.
Are you able to send me one of these GRIB2 files to debug further? I
like using the plot_data_plane tool to debug these sort of things.
Thanks,
John
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: robert.craig.2 at us.af.mil
Time: Fri Jan 04 07:26:58 2019
John, Thanks for your other response. Hope you had a good vacation.
I sent the GRIB2 file to you using ARL SAFE. Let me know if you
don't get it. I am also including the GRIB2 config file I point MET
to. What I don’t understand and I don't think the users manual
explains this well is the role this GRIB2 Config file plays in MET
reading a GRIB 2 file. When is this file used and does it play a
role in this problem.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 3, 2019 3:35 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2 Probabilities
in Grid-Stat
Hi Bob,
I saw you asked a new question on an older ticket. For clarity, I
pulled that question out and used it to make this new ticket.
I see that Grid-Stat is not selecting out the records you intend.
Based on the log messages, it finds 4 matches:
WARNING: MetGrib2DataFile::data_plane() - Multiple matching records
found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using 5
5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster mean)
153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
<5400:probability forecast
213: ?
I agree that it should only be considering the PROB forecast types.
Are you able to send me one of these GRIB2 files to debug further? I
like using the plot_data_plane tool to debug these sort of things.
Thanks,
John
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: robert.craig.2 at us.af.mil
Time: Fri Jan 04 07:26:58 2019
GRIB2
0 1 0 255 57 1 6 13 "CEIL" "Probability of ceiling < 500ft" "%"
0 14 0 255 57 0 1 33 "CRAIN" "Categorical Rain" "-"
0 14 0 255 57 0 1 36 "CSNOW" "Categorical Snow" "-"
0 14 0 255 57 0 1 34 "CFRZR" "Categorical Freezing Rain" "-"
0 14 0 255 57 0 19 0 "VIS" "Visibility" "m"
0 10 0 255 57 1 19 161 "PWS15" "10 m above ground" "%"
0 1 0 255 57 1 2 22 "PWS25" "10 m above ground" "%"
0 10 0 255 57 1 19 164 "PWS35" "Probabilty of 10 m winds above
35kts" "%"
0 10 0 255 57 1 19 165 "PWS50" "Probabilty of 10 m winds above
50kts" "%"
0 10 0 255 57 1 19 182 "PSN1" "Probability of snow accumulation > 1"
"%"
0 10 0 255 57 1 19 183 "PSN2" "Probability of snow accumulation > 2"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .10"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .25"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .50"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .75"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > 1.0"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > 5.0"
"%"
0 10 0 255 57 1 19 211 "DUST" "Dust visiblity" "m"
0 10 0 255 57 1 19 212 "PT273" "Probability of 2m temperature > 273"
"%"
0 10 0 255 57 1 19 213 "PT305" "Probability of 2m temperature > 305"
"%"
0 10 0 255 57 1 19 207 "PHT58" "Probability of 500mb HGT > 5800 m"
"%"
0 10 0 255 57 1 19 203 "PVIS1" "Probability of surface visiblity < 1
mi" "%"
0 10 0 255 57 1 19 202 "PVIS3" "Probability of surface visibility <
3 mi" "%"
0 10 0 255 57 1 19 201 "PVIS5" "Probability of surface visibility <
5 mi" "%"
0 10 0 255 57 1 19 224 "PC20" "Probabilty of Total Cloud < 20 %"
"%"
0 10 0 255 57 1 19 225 "PC80" "Probablity of Total Cloud > 80 %"
"%"
0 14 0 255 57 0 1 8 "APCP" "Total accumulated precipitation"
"kilogramsPerMeterSquared"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .01"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .02"
"%"
0 1 0 255 57 1 1 8 "APCP" "Probability of precip accumulation > .05"
"%"
0 10 0 255 57 1 19 176 "QP010" "Probability of precip accumulation >
.01" "%"
0 14 0 255 57 1 1 191 "GEOLAT" "Latitude" "deg"
0 14 0 255 57 1 6 11 "CEIL" "entire atmosphere" "m"
0 1 0 255 57 1 6 13 "CEIL" "prob <500" "%"
0 1 0 255 57 1 3 5 "HGT1" "prob <5400" "%"
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: John Halley Gotway
Time: Fri Jan 04 14:31:48 2019
Bob,
Thanks for sending that GRIB2 file. I was able to retrieve it. And I
am
able to replicate the behavior you describe... getting erroneous
matches.
I suspect this issue hasn't arisen before since the probability data
we
process has typically been stored in different files from the
deterministic
forecasts. To test, I pulled some ensemble data from NCEP... separate
files for the ensemble means and probabilities. plot_data_plane is
able to
extract data fine from each file separately. But when I concatenate
the
ensemble means and probabilities into a single file, plot_data_plane
demonstrates the same behavior you're seeing with "multiple matches".
In the long run, we should enhance the GRIB2 library code in MET to
better
distinguish between these deterministic and probabilistic fields.
But for now, let's identify additional filtering option we can use to
differentiate between these records. Looking in
met-6.1/share/met/config/README, you'll find the following GRIB2
filtering
criteria that are available:
// - The "GRIB2_disc" entry specifies the GRIB2 discipline code.
// - The "GRIB2_parm_cat" entry specifies the parameter category
code.
// - The "GRIB2_parm" entry specifies the parameter code.
// - The "GRIB2_pdt" entry specifies the product definition
template
(Table 4.0).
// - The "GRIB2_process" entry specifies the generating process
(Table 4.3).
// - The "GRIB2_cntr" entry specifies the originating center.
// - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
4.6).
// - The "GRIB2_der_type" entry specifies the derived product
type
(Table 4.7).
// - The "GRIB2_stat_type" entry specifies the statistical
processing
type (Table 4.10).
Some trial and error led to using the "GRIB2_pdt" option... for the
Product
Definition Template.
Using wgrib2 with the -pdt option, I see that the pdt = 2 for record 5
and
pdt = 5 for record 153:
And looking at this table, I see that 5 is for "probabilities":
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
0.shtml
> wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
153:30958857:pdt=5
So I added that to my plot_data_plane command like this:
*met-6.1/bin/plot_data_plane \*
*grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
*'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
GRIB2_pdt=5;' -v 3*
And I matches record 153:
*DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of GRIB2
file
'grib2.2018112600.0024.EPOSTOUT'*
And the resulting plot is attached.
Hope that helps.
Thanks,
John
On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>
> John, Thanks for your other response. Hope you had a good
vacation.
>
> I sent the GRIB2 file to you using ARL SAFE. Let me know if you
don't
> get it. I am also including the GRIB2 config file I point MET to.
What I
> don’t understand and I don't think the users manual explains this
well is
> the role this GRIB2 Config file plays in MET reading a GRIB 2 file.
When
> is this file used and does it play a role in this problem.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 3, 2019 3:35 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities in
> Grid-Stat
>
> Hi Bob,
>
> I saw you asked a new question on an older ticket. For clarity, I
pulled
> that question out and used it to make this new ticket.
>
> I see that Grid-Stat is not selecting out the records you intend.
Based
> on the log messages, it finds 4 matches:
> WARNING: MetGrib2DataFile::data_plane() - Multiple matching
records
> found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using 5
> 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
> 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
mean)
> 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
<5400:probability
> forecast
> 213: ?
>
> I agree that it should only be considering the PROB forecast types.
Are
> you able to send me one of these GRIB2 files to debug further? I
like
> using the plot_data_plane tool to debug these sort of things.
>
> Thanks,
> John
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2 Probabilities in Grid-Stat
From: robert.craig.2 at us.af.mil
Time: Fri Jan 04 15:49:33 2019
Thanks John I will give that a try. I am still confused at the role
of my custom grib table (which I attached). When is it used? It
doesn't seem to play a role in this retrieval at all. So I am never
sure what I should have in it or not.
Thanks Again
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, January 4, 2019 3:32 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities in Grid-Stat
Bob,
Thanks for sending that GRIB2 file. I was able to retrieve it. And I
am able to replicate the behavior you describe... getting erroneous
matches.
I suspect this issue hasn't arisen before since the probability data
we process has typically been stored in different files from the
deterministic forecasts. To test, I pulled some ensemble data from
NCEP... separate files for the ensemble means and probabilities.
plot_data_plane is able to extract data fine from each file
separately. But when I concatenate the ensemble means and
probabilities into a single file, plot_data_plane demonstrates the
same behavior you're seeing with "multiple matches".
In the long run, we should enhance the GRIB2 library code in MET to
better distinguish between these deterministic and probabilistic
fields.
But for now, let's identify additional filtering option we can use to
differentiate between these records. Looking in met-
6.1/share/met/config/README, you'll find the following GRIB2 filtering
criteria that are available:
// - The "GRIB2_disc" entry specifies the GRIB2 discipline code.
// - The "GRIB2_parm_cat" entry specifies the parameter category
code.
// - The "GRIB2_parm" entry specifies the parameter code.
// - The "GRIB2_pdt" entry specifies the product definition
template
(Table 4.0).
// - The "GRIB2_process" entry specifies the generating process
(Table 4.3).
// - The "GRIB2_cntr" entry specifies the originating center.
// - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
4.6).
// - The "GRIB2_der_type" entry specifies the derived product
type
(Table 4.7).
// - The "GRIB2_stat_type" entry specifies the statistical
processing
type (Table 4.10).
Some trial and error led to using the "GRIB2_pdt" option... for the
Product Definition Template.
Using wgrib2 with the -pdt option, I see that the pdt = 2 for record 5
and pdt = 5 for record 153:
And looking at this table, I see that 5 is for "probabilities":
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
0.shtml
> wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
153:30958857:pdt=5
So I added that to my plot_data_plane command like this:
*met-6.1/bin/plot_data_plane \*
*grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
*'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
GRIB2_pdt=5;' -v 3*
And I matches record 153:
*DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of GRIB2
file
'grib2.2018112600.0024.EPOSTOUT'*
And the resulting plot is attached.
Hope that helps.
Thanks,
John
On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>
> John, Thanks for your other response. Hope you had a good
vacation.
>
> I sent the GRIB2 file to you using ARL SAFE. Let me know if you
don't
> get it. I am also including the GRIB2 config file I point MET to.
What I
> don’t understand and I don't think the users manual explains this
well is
> the role this GRIB2 Config file plays in MET reading a GRIB 2 file.
When
> is this file used and does it play a role in this problem.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 3, 2019 3:35 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities
> in Grid-Stat
>
> Hi Bob,
>
> I saw you asked a new question on an older ticket. For clarity, I
> pulled that question out and used it to make this new ticket.
>
> I see that Grid-Stat is not selecting out the records you intend.
> Based on the log messages, it finds 4 matches:
> WARNING: MetGrib2DataFile::data_plane() - Multiple matching
records
> found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using 5
> 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
> 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
mean)
> 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
> <5400:probability forecast
> 213: ?
>
> I agree that it should only be considering the PROB forecast types.
> Are you able to send me one of these GRIB2 files to debug further?
I
> like using the plot_data_plane tool to debug these sort of things.
>
> Thanks,
> John
>
>
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: John Halley Gotway
Time: Fri Jan 04 16:14:04 2019
Bob,
MET contains some standard GRIB1 and GRIB2 tables that are read at
runtime
from met-6.1/share/met/table_files.
If the MET_GRIB_TABLES environment variable it reads those tables
first
before reading the standard ones. Here's a selection from the user's
guide
about it:
*The MET_GRIB_TABLES environment variable can be set to specify the
location ofcustom GRIB tables. It can either be set to a specific
file
name or to adirectory containing custom GRIB tables files. These file
names must begin witha "grib1" or "grib2" prefix and end with a ".txt"
suffix. Their format mustmatch the format used by the default MET
GRIB
table files, described below.The custom GRIB tables are read prior to
the
default tables and their settingstake precedence.*
Since the custom GRIB tables are read first, they can be used to
override
the default settings.
When MET_GRIB_TABLES is set, you should see a DEBUG level 1 log
message
tell you what table(s) are being read. For example, I set
MET_GRIB_TABLES
to point to the one you sent and re-ran plot_data_plane:
*setenv MET_GRIB_TABLES grib2_geps.txt*
*met-6.1/bin/plot_data_plane grib2.2018112600.0024.EPOSTOUT plot.ps
<http://plot.ps> 'name="PROB"; level="P500"; prob={name="HGT";
thresh_hi=5400;} GRIB2_pdt=5;' -v 3*
And here's the first line of log messages confirming that my custom
GRIB
table is being read:
*DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
grib2_geps.txt*
Thanks,
John
On Fri, Jan 4, 2019 at 3:50 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>
> Thanks John I will give that a try. I am still confused at the role
of my
> custom grib table (which I attached). When is it used? It doesn't
seem
> to play a role in this retrieval at all. So I am never sure what I
should
> have in it or not.
>
> Thanks Again
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, January 4, 2019 3:32 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> Probabilities in Grid-Stat
>
> Bob,
>
> Thanks for sending that GRIB2 file. I was able to retrieve it. And
I am
> able to replicate the behavior you describe... getting erroneous
matches.
>
> I suspect this issue hasn't arisen before since the probability data
we
> process has typically been stored in different files from the
deterministic
> forecasts. To test, I pulled some ensemble data from NCEP...
separate
> files for the ensemble means and probabilities. plot_data_plane is
able to
> extract data fine from each file separately. But when I concatenate
the
> ensemble means and probabilities into a single file, plot_data_plane
> demonstrates the same behavior you're seeing with "multiple
matches".
>
> In the long run, we should enhance the GRIB2 library code in MET to
better
> distinguish between these deterministic and probabilistic fields.
>
> But for now, let's identify additional filtering option we can use
to
> differentiate between these records. Looking in
> met-6.1/share/met/config/README, you'll find the following GRIB2
filtering
> criteria that are available:
> // - The "GRIB2_disc" entry specifies the GRIB2 discipline
code.
> // - The "GRIB2_parm_cat" entry specifies the parameter
category
> code.
> // - The "GRIB2_parm" entry specifies the parameter code.
> // - The "GRIB2_pdt" entry specifies the product definition
template
> (Table 4.0).
> // - The "GRIB2_process" entry specifies the generating
process
> (Table 4.3).
> // - The "GRIB2_cntr" entry specifies the originating center.
> // - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
> 4.6).
> // - The "GRIB2_der_type" entry specifies the derived product
type
> (Table 4.7).
> // - The "GRIB2_stat_type" entry specifies the statistical
processing
> type (Table 4.10).
>
> Some trial and error led to using the "GRIB2_pdt" option... for the
> Product Definition Template.
> Using wgrib2 with the -pdt option, I see that the pdt = 2 for record
5 and
> pdt = 5 for record 153:
>
> And looking at this table, I see that 5 is for "probabilities":
>
> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
0.shtml
>
> > wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
> 153:30958857:pdt=5
>
> So I added that to my plot_data_plane command like this:
>
> *met-6.1/bin/plot_data_plane \*
> *grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
> *'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
> GRIB2_pdt=5;' -v 3*
>
> And I matches record 153:
>
> *DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of GRIB2
file
> 'grib2.2018112600.0024.EPOSTOUT'*
>
> And the resulting plot is attached.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
> >
> > John, Thanks for your other response. Hope you had a good
vacation.
> >
> > I sent the GRIB2 file to you using ARL SAFE. Let me know if you
don't
> > get it. I am also including the GRIB2 config file I point MET to.
> What I
> > don’t understand and I don't think the users manual explains this
well is
> > the role this GRIB2 Config file plays in MET reading a GRIB 2
file.
> When
> > is this file used and does it play a role in this problem.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 3, 2019 3:35 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities
> > in Grid-Stat
> >
> > Hi Bob,
> >
> > I saw you asked a new question on an older ticket. For clarity, I
> > pulled that question out and used it to make this new ticket.
> >
> > I see that Grid-Stat is not selecting out the records you intend.
> > Based on the log messages, it finds 4 matches:
> > WARNING: MetGrib2DataFile::data_plane() - Multiple matching
records
> > found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213 - using
5
> > 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
> > 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
mean)
> > 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
> > <5400:probability forecast
> > 213: ?
> >
> > I agree that it should only be considering the PROB forecast
types.
> > Are you able to send me one of these GRIB2 files to debug further?
I
> > like using the plot_data_plane tool to debug these sort of things.
> >
> > Thanks,
> > John
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2 Probabilities in Grid-Stat
From: robert.craig.2 at us.af.mil
Time: Mon Jan 07 06:55:08 2019
Thanks John, I am aware how the tables are selected for use but I am
still fuzzy on what MET expects to find in these tables. In dealing
with the probability GRIB2 fields, does every field have to be in
these tables? If I am degribbing records with the same name and
parameter codes but have different thresholds, would I just have one
entry in the GRIB table? Does MET care what is in the description or
units fields in the GRIB table?
Thanks again
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, January 4, 2019 5:14 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities in Grid-Stat
Bob,
MET contains some standard GRIB1 and GRIB2 tables that are read at
runtime from met-6.1/share/met/table_files.
If the MET_GRIB_TABLES environment variable it reads those tables
first before reading the standard ones. Here's a selection from the
user's guide about it:
*The MET_GRIB_TABLES environment variable can be set to specify the
location ofcustom GRIB tables. It can either be set to a specific
file name or to adirectory containing custom GRIB tables files. These
file names must begin witha "grib1" or "grib2" prefix and end with a
".txt"
suffix. Their format mustmatch the format used by the default MET
GRIB table files, described below.The custom GRIB tables are read
prior to the default tables and their settingstake precedence.*
Since the custom GRIB tables are read first, they can be used to
override the default settings.
When MET_GRIB_TABLES is set, you should see a DEBUG level 1 log
message tell you what table(s) are being read. For example, I set
MET_GRIB_TABLES to point to the one you sent and re-ran
plot_data_plane:
*setenv MET_GRIB_TABLES grib2_geps.txt*
*met-6.1/bin/plot_data_plane grib2.2018112600.0024.EPOSTOUT plot.ps
<http://plot.ps> 'name="PROB"; level="P500"; prob={name="HGT";
thresh_hi=5400;} GRIB2_pdt=5;' -v 3*
And here's the first line of log messages confirming that my custom
GRIB table is being read:
*DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
grib2_geps.txt*
Thanks,
John
On Fri, Jan 4, 2019 at 3:50 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>
> Thanks John I will give that a try. I am still confused at the role
of my
> custom grib table (which I attached). When is it used? It doesn't
seem
> to play a role in this retrieval at all. So I am never sure what I
> should have in it or not.
>
> Thanks Again
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, January 4, 2019 3:32 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> Probabilities in Grid-Stat
>
> Bob,
>
> Thanks for sending that GRIB2 file. I was able to retrieve it. And
I
> am able to replicate the behavior you describe... getting erroneous
matches.
>
> I suspect this issue hasn't arisen before since the probability data
> we process has typically been stored in different files from the
> deterministic forecasts. To test, I pulled some ensemble data from
> NCEP... separate files for the ensemble means and probabilities.
> plot_data_plane is able to extract data fine from each file
> separately. But when I concatenate the ensemble means and
> probabilities into a single file, plot_data_plane demonstrates the
same behavior you're seeing with "multiple matches".
>
> In the long run, we should enhance the GRIB2 library code in MET to
> better distinguish between these deterministic and probabilistic
fields.
>
> But for now, let's identify additional filtering option we can use
to
> differentiate between these records. Looking in
> met-6.1/share/met/config/README, you'll find the following GRIB2
> filtering criteria that are available:
> // - The "GRIB2_disc" entry specifies the GRIB2 discipline
code.
> // - The "GRIB2_parm_cat" entry specifies the parameter
category
> code.
> // - The "GRIB2_parm" entry specifies the parameter code.
> // - The "GRIB2_pdt" entry specifies the product definition
template
> (Table 4.0).
> // - The "GRIB2_process" entry specifies the generating
process
> (Table 4.3).
> // - The "GRIB2_cntr" entry specifies the originating center.
> // - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
> 4.6).
> // - The "GRIB2_der_type" entry specifies the derived product
type
> (Table 4.7).
> // - The "GRIB2_stat_type" entry specifies the statistical
processing
> type (Table 4.10).
>
> Some trial and error led to using the "GRIB2_pdt" option... for the
> Product Definition Template.
> Using wgrib2 with the -pdt option, I see that the pdt = 2 for record
5
> and pdt = 5 for record 153:
>
> And looking at this table, I see that 5 is for "probabilities":
>
> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
0.s
> html
>
> > wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
> 153:30958857:pdt=5
>
> So I added that to my plot_data_plane command like this:
>
> *met-6.1/bin/plot_data_plane \*
> *grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
> *'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
> GRIB2_pdt=5;' -v 3*
>
> And I matches record 153:
>
> *DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of GRIB2
> file
> 'grib2.2018112600.0024.EPOSTOUT'*
>
> And the resulting plot is attached.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
> >
> > John, Thanks for your other response. Hope you had a good
vacation.
> >
> > I sent the GRIB2 file to you using ARL SAFE. Let me know if you
don't
> > get it. I am also including the GRIB2 config file I point MET to.
> What I
> > don’t understand and I don't think the users manual explains this
> > well is the role this GRIB2 Config file plays in MET reading a
GRIB 2 file.
> When
> > is this file used and does it play a role in this problem.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 3, 2019 3:35 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> > Probabilities in Grid-Stat
> >
> > Hi Bob,
> >
> > I saw you asked a new question on an older ticket. For clarity, I
> > pulled that question out and used it to make this new ticket.
> >
> > I see that Grid-Stat is not selecting out the records you intend.
> > Based on the log messages, it finds 4 matches:
> > WARNING: MetGrib2DataFile::data_plane() - Multiple matching
> > records found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153, 213
-
> > using 5
> > 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
> > 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
> > mean)
> > 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
> > <5400:probability forecast
> > 213: ?
> >
> > I agree that it should only be considering the PROB forecast
types.
> > Are you able to send me one of these GRIB2 files to debug further?
> > I like using the plot_data_plane tool to debug these sort of
things.
> >
> > Thanks,
> > John
> >
> >
>
>
>
>
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: John Halley Gotway
Time: Mon Jan 07 10:35:05 2019
John,
You only need to define a custom GRIB table when your local usage
differs
from the default tables. You only need one GRIB table entry for each
unique combination of GRIB codes... not separate ones for each
threshold.
Instead, the threshold information is handled using the "thresh_lo"
and
"thresh_hi" setting(s) in the MET config file. MET doesn't really
care
about the description and units strings... it just reads them in and
writes
them out as variables attributes to the NetCDF files it creates.
However, I do recall getting a segfault in the past when the unit
string
was missing since and MET was trying to write an empty string to a
NetCDF
file. I believe that issue is resolved in the current release, but am
not
sure if you'll encounter that in met-6.1.
Thanks,
John
On Mon, Jan 7, 2019 at 6:55 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>
> Thanks John, I am aware how the tables are selected for use but I am
still
> fuzzy on what MET expects to find in these tables. In dealing with
the
> probability GRIB2 fields, does every field have to be in these
tables?
> If I am degribbing records with the same name and parameter codes
but have
> different thresholds, would I just have one entry in the GRIB
table? Does
> MET care what is in the description or units fields in the GRIB
table?
>
> Thanks again
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, January 4, 2019 5:14 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> Probabilities in Grid-Stat
>
> Bob,
>
> MET contains some standard GRIB1 and GRIB2 tables that are read at
runtime
> from met-6.1/share/met/table_files.
>
> If the MET_GRIB_TABLES environment variable it reads those tables
first
> before reading the standard ones. Here's a selection from the
user's guide
> about it:
>
>
>
>
>
>
>
> *The MET_GRIB_TABLES environment variable can be set to specify the
> location ofcustom GRIB tables. It can either be set to a specific
file
> name or to adirectory containing custom GRIB tables files. These
file
> names must begin witha "grib1" or "grib2" prefix and end with a
".txt"
> suffix. Their format mustmatch the format used by the default MET
GRIB
> table files, described below.The custom GRIB tables are read prior
to the
> default tables and their settingstake precedence.*
>
> Since the custom GRIB tables are read first, they can be used to
override
> the default settings.
>
> When MET_GRIB_TABLES is set, you should see a DEBUG level 1 log
message
> tell you what table(s) are being read. For example, I set
MET_GRIB_TABLES
> to point to the one you sent and re-ran plot_data_plane:
>
>
> *setenv MET_GRIB_TABLES grib2_geps.txt*
> *met-6.1/bin/plot_data_plane grib2.2018112600.0024.EPOSTOUT plot.ps
<
> http://plot.ps> 'name="PROB"; level="P500"; prob={name="HGT";
> thresh_hi=5400;} GRIB2_pdt=5;' -v 3*
>
> And here's the first line of log messages confirming that my custom
GRIB
> table is being read:
> *DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
grib2_geps.txt*
>
> Thanks,
> John
>
>
>
> On Fri, Jan 4, 2019 at 3:50 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
> >
> > Thanks John I will give that a try. I am still confused at the
role of
> my
> > custom grib table (which I attached). When is it used? It
doesn't seem
> > to play a role in this retrieval at all. So I am never sure what
I
> > should have in it or not.
> >
> > Thanks Again
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Friday, January 4, 2019 3:32 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> > Probabilities in Grid-Stat
> >
> > Bob,
> >
> > Thanks for sending that GRIB2 file. I was able to retrieve it.
And I
> > am able to replicate the behavior you describe... getting
erroneous
> matches.
> >
> > I suspect this issue hasn't arisen before since the probability
data
> > we process has typically been stored in different files from the
> > deterministic forecasts. To test, I pulled some ensemble data
from
> > NCEP... separate files for the ensemble means and probabilities.
> > plot_data_plane is able to extract data fine from each file
> > separately. But when I concatenate the ensemble means and
> > probabilities into a single file, plot_data_plane demonstrates the
same
> behavior you're seeing with "multiple matches".
> >
> > In the long run, we should enhance the GRIB2 library code in MET
to
> > better distinguish between these deterministic and probabilistic
fields.
> >
> > But for now, let's identify additional filtering option we can use
to
> > differentiate between these records. Looking in
> > met-6.1/share/met/config/README, you'll find the following GRIB2
> > filtering criteria that are available:
> > // - The "GRIB2_disc" entry specifies the GRIB2 discipline
code.
> > // - The "GRIB2_parm_cat" entry specifies the parameter
category
> > code.
> > // - The "GRIB2_parm" entry specifies the parameter code.
> > // - The "GRIB2_pdt" entry specifies the product definition
> template
> > (Table 4.0).
> > // - The "GRIB2_process" entry specifies the generating
process
> > (Table 4.3).
> > // - The "GRIB2_cntr" entry specifies the originating
center.
> > // - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
> > 4.6).
> > // - The "GRIB2_der_type" entry specifies the derived
product type
> > (Table 4.7).
> > // - The "GRIB2_stat_type" entry specifies the statistical
> processing
> > type (Table 4.10).
> >
> > Some trial and error led to using the "GRIB2_pdt" option... for
the
> > Product Definition Template.
> > Using wgrib2 with the -pdt option, I see that the pdt = 2 for
record 5
> > and pdt = 5 for record 153:
> >
> > And looking at this table, I see that 5 is for "probabilities":
> >
> >
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-0.s
> > html
> >
> > > wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
> > 153:30958857:pdt=5
> >
> > So I added that to my plot_data_plane command like this:
> >
> > *met-6.1/bin/plot_data_plane \*
> > *grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
> > *'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
> > GRIB2_pdt=5;' -v 3*
> >
> > And I matches record 153:
> >
> > *DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
> > 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of
GRIB2
> > file
> > 'grib2.2018112600.0024.EPOSTOUT'*
> >
> > And the resulting plot is attached.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
> > >
> > > John, Thanks for your other response. Hope you had a good
vacation.
> > >
> > > I sent the GRIB2 file to you using ARL SAFE. Let me know if
you don't
> > > get it. I am also including the GRIB2 config file I point MET
to.
> > What I
> > > don’t understand and I don't think the users manual explains
this
> > > well is the role this GRIB2 Config file plays in MET reading a
GRIB 2
> file.
> > When
> > > is this file used and does it play a role in this problem.
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 3, 2019 3:35 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
> > > Probabilities in Grid-Stat
> > >
> > > Hi Bob,
> > >
> > > I saw you asked a new question on an older ticket. For clarity,
I
> > > pulled that question out and used it to make this new ticket.
> > >
> > > I see that Grid-Stat is not selecting out the records you
intend.
> > > Based on the log messages, it finds 4 matches:
> > > WARNING: MetGrib2DataFile::data_plane() - Multiple matching
> > > records found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153,
213 -
> > > using 5
> > > 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
> > > 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
> > > mean)
> > > 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
> > > <5400:probability forecast
> > > 213: ?
> > >
> > > I agree that it should only be considering the PROB forecast
types.
> > > Are you able to send me one of these GRIB2 files to debug
further?
> > > I like using the plot_data_plane tool to debug these sort of
things.
> > >
> > > Thanks,
> > > John
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: GRIB2 Probabilities in Grid-Stat
From: John Halley Gotway
Time: Mon Jan 07 10:36:01 2019
Bob,
My apologies... I called you "John" in the email I just sent. Time
for
another cup of coffee!
John
On Mon, Jan 7, 2019 at 10:34 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> You only need to define a custom GRIB table when your local usage
differs
> from the default tables. You only need one GRIB table entry for
each
> unique combination of GRIB codes... not separate ones for each
threshold.
> Instead, the threshold information is handled using the "thresh_lo"
and
> "thresh_hi" setting(s) in the MET config file. MET doesn't really
care
> about the description and units strings... it just reads them in and
writes
> them out as variables attributes to the NetCDF files it creates.
>
> However, I do recall getting a segfault in the past when the unit
string
> was missing since and MET was trying to write an empty string to a
NetCDF
> file. I believe that issue is resolved in the current release, but
am not
> sure if you'll encounter that in met-6.1.
>
> Thanks,
> John
>
> On Mon, Jan 7, 2019 at 6:55 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>>
>> Thanks John, I am aware how the tables are selected for use but I
am
>> still fuzzy on what MET expects to find in these tables. In
dealing with
>> the probability GRIB2 fields, does every field have to be in
these
>> tables? If I am degribbing records with the same name and
parameter codes
>> but have different thresholds, would I just have one entry in the
GRIB
>> table? Does MET care what is in the description or units fields in
the
>> GRIB table?
>>
>> Thanks again
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Friday, January 4, 2019 5:14 PM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> Probabilities in Grid-Stat
>>
>> Bob,
>>
>> MET contains some standard GRIB1 and GRIB2 tables that are read at
>> runtime from met-6.1/share/met/table_files.
>>
>> If the MET_GRIB_TABLES environment variable it reads those tables
first
>> before reading the standard ones. Here's a selection from the
user's guide
>> about it:
>>
>>
>>
>>
>>
>>
>>
>> *The MET_GRIB_TABLES environment variable can be set to specify the
>> location ofcustom GRIB tables. It can either be set to a specific
file
>> name or to adirectory containing custom GRIB tables files. These
file
>> names must begin witha "grib1" or "grib2" prefix and end with a
".txt"
>> suffix. Their format mustmatch the format used by the default MET
GRIB
>> table files, described below.The custom GRIB tables are read prior
to the
>> default tables and their settingstake precedence.*
>>
>> Since the custom GRIB tables are read first, they can be used to
override
>> the default settings.
>>
>> When MET_GRIB_TABLES is set, you should see a DEBUG level 1 log
message
>> tell you what table(s) are being read. For example, I set
MET_GRIB_TABLES
>> to point to the one you sent and re-ran plot_data_plane:
>>
>>
>> *setenv MET_GRIB_TABLES grib2_geps.txt*
>> *met-6.1/bin/plot_data_plane grib2.2018112600.0024.EPOSTOUT plot.ps
<
>> http://plot.ps> 'name="PROB"; level="P500"; prob={name="HGT";
>> thresh_hi=5400;} GRIB2_pdt=5;' -v 3*
>>
>> And here's the first line of log messages confirming that my custom
GRIB
>> table is being read:
>> *DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
grib2_geps.txt*
>>
>> Thanks,
>> John
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:50 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>> >
>> > Thanks John I will give that a try. I am still confused at the
role of
>> my
>> > custom grib table (which I attached). When is it used? It
doesn't
>> seem
>> > to play a role in this retrieval at all. So I am never sure what
I
>> > should have in it or not.
>> >
>> > Thanks Again
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: Friday, January 4, 2019 3:32 PM
>> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > <robert.craig.2 at us.af.mil>
>> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> > Probabilities in Grid-Stat
>> >
>> > Bob,
>> >
>> > Thanks for sending that GRIB2 file. I was able to retrieve it.
And I
>> > am able to replicate the behavior you describe... getting
erroneous
>> matches.
>> >
>> > I suspect this issue hasn't arisen before since the probability
data
>> > we process has typically been stored in different files from the
>> > deterministic forecasts. To test, I pulled some ensemble data
from
>> > NCEP... separate files for the ensemble means and probabilities.
>> > plot_data_plane is able to extract data fine from each file
>> > separately. But when I concatenate the ensemble means and
>> > probabilities into a single file, plot_data_plane demonstrates
the same
>> behavior you're seeing with "multiple matches".
>> >
>> > In the long run, we should enhance the GRIB2 library code in MET
to
>> > better distinguish between these deterministic and probabilistic
fields.
>> >
>> > But for now, let's identify additional filtering option we can
use to
>> > differentiate between these records. Looking in
>> > met-6.1/share/met/config/README, you'll find the following GRIB2
>> > filtering criteria that are available:
>> > // - The "GRIB2_disc" entry specifies the GRIB2 discipline
code.
>> > // - The "GRIB2_parm_cat" entry specifies the parameter
category
>> > code.
>> > // - The "GRIB2_parm" entry specifies the parameter code.
>> > // - The "GRIB2_pdt" entry specifies the product definition
>> template
>> > (Table 4.0).
>> > // - The "GRIB2_process" entry specifies the generating
process
>> > (Table 4.3).
>> > // - The "GRIB2_cntr" entry specifies the originating
center.
>> > // - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
>> > 4.6).
>> > // - The "GRIB2_der_type" entry specifies the derived
product type
>> > (Table 4.7).
>> > // - The "GRIB2_stat_type" entry specifies the statistical
>> processing
>> > type (Table 4.10).
>> >
>> > Some trial and error led to using the "GRIB2_pdt" option... for
the
>> > Product Definition Template.
>> > Using wgrib2 with the -pdt option, I see that the pdt = 2 for
record 5
>> > and pdt = 5 for record 153:
>> >
>> > And looking at this table, I see that 5 is for "probabilities":
>> >
>> >
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-0.s
>> > html
>> >
>> > > wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
>> > 153:30958857:pdt=5
>> >
>> > So I added that to my plot_data_plane command like this:
>> >
>> > *met-6.1/bin/plot_data_plane \*
>> > *grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
>> > *'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
>> > GRIB2_pdt=5;' -v 3*
>> >
>> > And I matches record 153:
>> >
>> > *DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
>> > 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of
GRIB2
>> > file
>> > 'grib2.2018112600.0024.EPOSTOUT'*
>> >
>> > And the resulting plot is attached.
>> >
>> > Hope that helps.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>> > >
>> > > John, Thanks for your other response. Hope you had a good
vacation.
>> > >
>> > > I sent the GRIB2 file to you using ARL SAFE. Let me know if
you
>> don't
>> > > get it. I am also including the GRIB2 config file I point MET
to.
>> > What I
>> > > don’t understand and I don't think the users manual explains
this
>> > > well is the role this GRIB2 Config file plays in MET reading a
GRIB 2
>> file.
>> > When
>> > > is this file used and does it play a role in this problem.
>> > >
>> > > Thanks
>> > > Bob
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > Sent: Thursday, January 3, 2019 3:35 PM
>> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > <robert.craig.2 at us.af.mil>
>> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> > > Probabilities in Grid-Stat
>> > >
>> > > Hi Bob,
>> > >
>> > > I saw you asked a new question on an older ticket. For
clarity, I
>> > > pulled that question out and used it to make this new ticket.
>> > >
>> > > I see that Grid-Stat is not selecting out the records you
intend.
>> > > Based on the log messages, it finds 4 matches:
>> > > WARNING: MetGrib2DataFile::data_plane() - Multiple matching
>> > > records found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153,
213 -
>> > > using 5
>> > > 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
>> > > 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
>> > > mean)
>> > > 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
>> > > <5400:probability forecast
>> > > 213: ?
>> > >
>> > > I agree that it should only be considering the PROB forecast
types.
>> > > Are you able to send me one of these GRIB2 files to debug
further?
>> > > I like using the plot_data_plane tool to debug these sort of
things.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2 Probabilities in Grid-Stat
From: robert.craig.2 at us.af.mil
Time: Mon Jan 07 11:41:34 2019
That’s okay bob:)
Thanks for the added information.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, January 7, 2019 11:36 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
Probabilities in Grid-Stat
Bob,
My apologies... I called you "John" in the email I just sent. Time
for another cup of coffee!
John
On Mon, Jan 7, 2019 at 10:34 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> You only need to define a custom GRIB table when your local usage
> differs from the default tables. You only need one GRIB table entry
> for each unique combination of GRIB codes... not separate ones for
each threshold.
> Instead, the threshold information is handled using the "thresh_lo"
> and "thresh_hi" setting(s) in the MET config file. MET doesn't
really
> care about the description and units strings... it just reads them
in
> and writes them out as variables attributes to the NetCDF files it
creates.
>
> However, I do recall getting a segfault in the past when the unit
> string was missing since and MET was trying to write an empty string
> to a NetCDF file. I believe that issue is resolved in the current
> release, but am not sure if you'll encounter that in met-6.1.
>
> Thanks,
> John
>
> On Mon, Jan 7, 2019 at 6:55 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>>
>> Thanks John, I am aware how the tables are selected for use but I
am
>> still fuzzy on what MET expects to find in these tables. In
dealing with
>> the probability GRIB2 fields, does every field have to be in
these
>> tables? If I am degribbing records with the same name and
parameter
>> codes but have different thresholds, would I just have one entry in
>> the GRIB table? Does MET care what is in the description or units
>> fields in the GRIB table?
>>
>> Thanks again
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Friday, January 4, 2019 5:14 PM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> <robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> Probabilities in Grid-Stat
>>
>> Bob,
>>
>> MET contains some standard GRIB1 and GRIB2 tables that are read at
>> runtime from met-6.1/share/met/table_files.
>>
>> If the MET_GRIB_TABLES environment variable it reads those tables
>> first before reading the standard ones. Here's a selection from
the
>> user's guide about it:
>>
>>
>>
>>
>>
>>
>>
>> *The MET_GRIB_TABLES environment variable can be set to specify the
>> location ofcustom GRIB tables. It can either be set to a specific
>> file name or to adirectory containing custom GRIB tables files.
>> These file names must begin witha "grib1" or "grib2" prefix and end
with a ".txt"
>> suffix. Their format mustmatch the format used by the default MET
>> GRIB table files, described below.The custom GRIB tables are read
>> prior to the default tables and their settingstake precedence.*
>>
>> Since the custom GRIB tables are read first, they can be used to
>> override the default settings.
>>
>> When MET_GRIB_TABLES is set, you should see a DEBUG level 1 log
>> message tell you what table(s) are being read. For example, I set
>> MET_GRIB_TABLES to point to the one you sent and re-ran
plot_data_plane:
>>
>>
>> *setenv MET_GRIB_TABLES grib2_geps.txt* *met-
6.1/bin/plot_data_plane
>> grib2.2018112600.0024.EPOSTOUT plot.ps < http://plot.ps>
>> 'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
>> GRIB2_pdt=5;' -v 3*
>>
>> And here's the first line of log messages confirming that my custom
>> GRIB table is being read:
>> *DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
>> grib2_geps.txt*
>>
>> Thanks,
>> John
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:50 PM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>> >
>> > Thanks John I will give that a try. I am still confused at the
>> > role of
>> my
>> > custom grib table (which I attached). When is it used? It
doesn't
>> seem
>> > to play a role in this retrieval at all. So I am never sure what
I
>> > should have in it or not.
>> >
>> > Thanks Again
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: Friday, January 4, 2019 3:32 PM
>> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > <robert.craig.2 at us.af.mil>
>> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> > Probabilities in Grid-Stat
>> >
>> > Bob,
>> >
>> > Thanks for sending that GRIB2 file. I was able to retrieve it.
>> > And I am able to replicate the behavior you describe... getting
>> > erroneous
>> matches.
>> >
>> > I suspect this issue hasn't arisen before since the probability
>> > data we process has typically been stored in different files from
>> > the deterministic forecasts. To test, I pulled some ensemble
data
>> > from NCEP... separate files for the ensemble means and
probabilities.
>> > plot_data_plane is able to extract data fine from each file
>> > separately. But when I concatenate the ensemble means and
>> > probabilities into a single file, plot_data_plane demonstrates
the
>> > same
>> behavior you're seeing with "multiple matches".
>> >
>> > In the long run, we should enhance the GRIB2 library code in MET
to
>> > better distinguish between these deterministic and probabilistic
fields.
>> >
>> > But for now, let's identify additional filtering option we can
use
>> > to differentiate between these records. Looking in
>> > met-6.1/share/met/config/README, you'll find the following GRIB2
>> > filtering criteria that are available:
>> > // - The "GRIB2_disc" entry specifies the GRIB2 discipline
code.
>> > // - The "GRIB2_parm_cat" entry specifies the parameter
category
>> > code.
>> > // - The "GRIB2_parm" entry specifies the parameter code.
>> > // - The "GRIB2_pdt" entry specifies the product definition
>> template
>> > (Table 4.0).
>> > // - The "GRIB2_process" entry specifies the generating
process
>> > (Table 4.3).
>> > // - The "GRIB2_cntr" entry specifies the originating
center.
>> > // - The "GRIB2_ens_type" entry specifies the ensemble type
(Table
>> > 4.6).
>> > // - The "GRIB2_der_type" entry specifies the derived
product type
>> > (Table 4.7).
>> > // - The "GRIB2_stat_type" entry specifies the statistical
>> processing
>> > type (Table 4.10).
>> >
>> > Some trial and error led to using the "GRIB2_pdt" option... for
the
>> > Product Definition Template.
>> > Using wgrib2 with the -pdt option, I see that the pdt = 2 for
>> > record 5 and pdt = 5 for record 153:
>> >
>> > And looking at this table, I see that 5 is for "probabilities":
>> >
>> >
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
>> > 0.s
>> > html
>> >
>> > > wgrib2 -d 153 -pdt grib2.2018112600.0024.EPOSTOUT
>> > 153:30958857:pdt=5
>> >
>> > So I added that to my plot_data_plane command like this:
>> >
>> > *met-6.1/bin/plot_data_plane \*
>> > *grib2.2018112600.0024.EPOSTOUT plot.ps <http://plot.ps> \*
>> > *'name="PROB"; level="P500"; prob={name="HGT"; thresh_hi=5400;}
>> > GRIB2_pdt=5;' -v 3*
>> >
>> > And I matches record 153:
>> >
>> > *DEBUG 3: MetGrib2DataFile::data_plane() - Found exact match for
>> > 'PROB(HGT<5400.000)/P500/PROB' in GRIB2 record 153 field 1 of
GRIB2
>> > file
>> > 'grib2.2018112600.0024.EPOSTOUT'*
>> >
>> > And the resulting plot is attached.
>> >
>> > Hope that helps.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Fri, Jan 4, 2019 at 7:27 AM robert.craig.2 at us.af.mil via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88386 >
>> > >
>> > > John, Thanks for your other response. Hope you had a good
vacation.
>> > >
>> > > I sent the GRIB2 file to you using ARL SAFE. Let me know if
you
>> don't
>> > > get it. I am also including the GRIB2 config file I point MET
to.
>> > What I
>> > > don’t understand and I don't think the users manual explains
this
>> > > well is the role this GRIB2 Config file plays in MET reading a
>> > > GRIB 2
>> file.
>> > When
>> > > is this file used and does it play a role in this problem.
>> > >
>> > > Thanks
>> > > Bob
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > Sent: Thursday, January 3, 2019 3:35 PM
>> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > <robert.craig.2 at us.af.mil>
>> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #88386] GRIB2
>> > > Probabilities in Grid-Stat
>> > >
>> > > Hi Bob,
>> > >
>> > > I saw you asked a new question on an older ticket. For
clarity,
>> > > I pulled that question out and used it to make this new ticket.
>> > >
>> > > I see that Grid-Stat is not selecting out the records you
intend.
>> > > Based on the log messages, it finds 4 matches:
>> > > WARNING: MetGrib2DataFile::data_plane() - Multiple matching
>> > > records found for 'PROB(HGT<5400.000)/P500/PROB' - 5, 12, 153,
>> > > 213 - using 5
>> > > 5:1755184:d=2018112600:HGT:500 mb:6 hour fcst:mean all members
>> > > 12:3803321:d=2018112600:HGT:500 mb:6 hour fcst:std dev (cluster
>> > > mean)
>> > > 153:30958857:d=2018112600:HGT:500 mb:24 hour fcst:prob
>> > > <5400:probability forecast
>> > > 213: ?
>> > >
>> > > I agree that it should only be considering the PROB forecast
types.
>> > > Are you able to send me one of these GRIB2 files to debug
further?
>> > > I like using the plot_data_plane tool to debug these sort of
things.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list