[Met_help] [rt.rap.ucar.edu #97769] History for urgent question for reading ECMWF ensmeble data

John Halley Gotway via RT met_help at ucar.edu
Wed Feb 10 09:05:14 MST 2021


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I have an urgent question for how to read ECMWF ensemble grib2 data.
The GRIB2 tables by EC are quite different from NCEP/WMO.
In VSDB we can read it by using getgb2 function. But In MET, I have no idea
to set up in the ensemble_stat and grid-stat config files. 
If using wgrib2 for one member file,  the output is following: 

1:0:d=2020120100:var discipline=255 master_table=2 parmcat=3 parm=5:500 mb:12 hour fcst:ENS=-25
2:29547:d=2020120100:var discipline=255 master_table=2 parmcat=3 parm=5:1000 mb:12 hour fcst:ENS=-25
3:61542:d=2020120100:var discipline=255 master_table=2 parmcat=0 parm=0:850 mb:12 hour fcst:ENS=-25
4:78706:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=2:850 mb:12 hour fcst:ENS=-25
5:100088:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=3:850 mb:12 hour fcst:ENS=-25
6:123772:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=2:250 mb:12 hour fcst:ENS=-25
7:146448:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=3:250 mb:12 hour fcst:ENS=-25
8:171304:d=2020120100:var discipline=255 master_table=2 parmcat=0 parm=0:2 m above ground:12 hour fcst:ENS=-25
9:193892:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=2:10 m above ground:12 hour fcst:ENS=-25
10:217559:d=2020120100:var discipline=255 master_table=2 parmcat=2 parm=3:10 m above ground:12 hour fcst:ENS=-25

These 10 records are fixed,   500mb HGT, 1000mb HGT, 850 T, 850mb U and V, 250mb U and V, 2m T and 10m U and V.
The record number is not changed for all members and not change at all  times. 
The problem is there is no field name  string  in this file.  Only the record number is known to match each field. In VSDB, 
getgb2 subroutine  uses 3 product IDs:  parmcat=3 parm=5:1000 mb  to get them correctly. But I don't know how to set 
up fcst entry in  config files. I look at the MET instruction, saying can use level="Rxxx", but name has to be specified? 
or can use white card "*" ?

Thanks!

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Fri Dec 04 09:57:10 2020

Binbin,
You can do something similar with MET. Please go to this
link:https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html

And then search for "GRIB2_mtab", and you'll find the content
copied/pasted below. You can specify these INSTEAD OF specifying a
requested name.

//       - The "GRIB2_mtab" is an integer specifying the master table
number.
//       - The "GRIB2_ltab" is an integer specifying the local table
number.
//       - The "GRIB2_disc" is an integer specifying the GRIB2
discipline code.
//       - The "GRIB2_parm_cat" is an integer specifying the parameter
category
//         code.
//       - The "GRIB2_parm" is an integer specifying the parameter
code.
//       - The "GRIB2_pdt" is an integer specifying the product
definition
//         template (Table 4.0).
//       - The "GRIB2_process" is an integer specifying the generating
process
//         (Table 4.3).
//       - The "GRIB2_cntr" is an integer specifying the originating
center.
//       - The "GRIB2_ens_type" is an integer specifying the ensemble
type
//         (Table 4.6).
//       - The "GRIB2_der_type" is an integer specifying the derived
product
//         type (Table 4.7).
//       - The "GRIB2_stat_type" is an integer specifying the
statistical
//         processing type (Table 4.10).
//       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val" entries
are arrays
//         of integers which specify the product description template
values to
//         be used. The indices are 0-based. For example, use the
following to
//         request a GRIB2 record whose 9-th and 27-th product
description
//         template values are 1 and 2, respectively:
//            GRIB2_ipdtmpl_index=[8, 26]; GRIB2_ipdtmpl_val=[1, 2];

While I don't have any ECMWF data on hand for testing, I do have a
file from the Air Force from which wgrib2 prints a similar result:

549:687031550:d=2017061000:var discipline=0 center=57 local_table=1
parmcat=1 parm=192:surface:18 hour fcst:

So I used that for testing. I ran the following plot_data_plane
command WITHOUT specifying a requested name:


plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps 'GRIB2_disc=0;
GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
level="Z0";' -v 3

And then I looked closely at the log message to confirm that it used
record 549, as expected:

DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for
'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'

So MET used these GRIB2 numbers I defined to do a table lookup and
find a match in the MET GRIB
tables:https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files

And it determined that that field should be called "CTSTM". Hopefully,
the ECMWF GRIB2 tables are already defined there in MET. If not, we
can always add them.

Please try running plot_data_plane at verbosity level 3 on one of
these files first. And make sure it runs without error and the log
message prints out the "name" it found for that data. If so, you can
just skip using all those integers and revert back to using the name
MET knows. If not, then we may need to define a custom GRIB table for
this ECMWF GRIB2 data.

Thanks,
John

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Fri Dec 04 10:07:41 2020

John,

   You can go to WCOSS/Venus my directory
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
 ecme.ens01.t00z.grd3.f12.grib2 is one sample file

Binbin

On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
> You can do something similar with MET. Please go to this link:
> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>
> And then search for "GRIB2_mtab", and you'll find the content
> copied/pasted below. You can specify these INSTEAD OF specifying a
> requested name.
>
> //       - The "GRIB2_mtab" is an integer specifying the master
table
> number.
> //       - The "GRIB2_ltab" is an integer specifying the local table
> number.
> //       - The "GRIB2_disc" is an integer specifying the GRIB2
discipline
> code.
> //       - The "GRIB2_parm_cat" is an integer specifying the
parameter
> category
> //         code.
> //       - The "GRIB2_parm" is an integer specifying the parameter
code.
> //       - The "GRIB2_pdt" is an integer specifying the product
definition
> //         template (Table 4.0).
> //       - The "GRIB2_process" is an integer specifying the
generating
> process
> //         (Table 4.3).
> //       - The "GRIB2_cntr" is an integer specifying the originating
> center.
> //       - The "GRIB2_ens_type" is an integer specifying the
ensemble type
> //         (Table 4.6).
> //       - The "GRIB2_der_type" is an integer specifying the derived
> product
> //         type (Table 4.7).
> //       - The "GRIB2_stat_type" is an integer specifying the
statistical
> //         processing type (Table 4.10).
> //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val" entries
are
> arrays
> //         of integers which specify the product description
template
> values to
> //         be used. The indices are 0-based. For example, use the
> following to
> //         request a GRIB2 record whose 9-th and 27-th product
description
> //         template values are 1 and 2, respectively:
> //            GRIB2_ipdtmpl_index=[8, 26]; GRIB2_ipdtmpl_val=[1, 2];
>
> While I don't have any ECMWF data on hand for testing, I do have a
file
> from the Air Force from which wgrib2 prints a similar result:
>
> 549:687031550:d=2017061000:var discipline=0 center=57 local_table=1
> parmcat=1 parm=192:surface:18 hour fcst:
>
> So I used that for testing. I ran the following plot_data_plane
command
> WITHOUT specifying a requested name:
>
>
> plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps 'GRIB2_disc=0;
> GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
level="Z0";'
> -v 3
>
> And then I looked closely at the log message to confirm that it used
> record 549, as expected:
>
> DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for
> 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>
> So MET used these GRIB2 numbers I defined to do a table lookup and
find a
> match in the MET GRIB tables:
> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>
> And it determined that that field should be called "CTSTM".
Hopefully, the
> ECMWF GRIB2 tables are already defined there in MET. If not, we can
always
> add them.
>
> Please try running plot_data_plane at verbosity level 3 on one of
these
> files first. And make sure it runs without error and the log message
prints
> out the "name" it found for that data. If so, you can just skip
using all
> those integers and revert back to using the name MET knows. If not,
then we
> may need to define a custom GRIB table for this ECMWF GRIB2 data.
>
> Thanks,
> John
>


--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Fri Dec 04 11:05:32 2020

John,

  I tested using following command:

 plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3

ERROR  :
ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
GRIB2_parm_cat and GRIB2_parm must be specified in field information

 plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v
3

Still got error
ERROR  :
ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
GRIB2_parm_cat and GRIB2_parm must be specified in field information

Looks like discipline=255 is not recognized

Binbin


On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
> You can do something similar with MET. Please go to this link:
> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>
> And then search for "GRIB2_mtab", and you'll find the content
> copied/pasted below. You can specify these INSTEAD OF specifying a
> requested name.
>
> //       - The "GRIB2_mtab" is an integer specifying the master
table
> number.
> //       - The "GRIB2_ltab" is an integer specifying the local table
> number.
> //       - The "GRIB2_disc" is an integer specifying the GRIB2
discipline
> code.
> //       - The "GRIB2_parm_cat" is an integer specifying the
parameter
> category
> //         code.
> //       - The "GRIB2_parm" is an integer specifying the parameter
code.
> //       - The "GRIB2_pdt" is an integer specifying the product
definition
> //         template (Table 4.0).
> //       - The "GRIB2_process" is an integer specifying the
generating
> process
> //         (Table 4.3).
> //       - The "GRIB2_cntr" is an integer specifying the originating
> center.
> //       - The "GRIB2_ens_type" is an integer specifying the
ensemble type
> //         (Table 4.6).
> //       - The "GRIB2_der_type" is an integer specifying the derived
> product
> //         type (Table 4.7).
> //       - The "GRIB2_stat_type" is an integer specifying the
statistical
> //         processing type (Table 4.10).
> //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val" entries
are
> arrays
> //         of integers which specify the product description
template
> values to
> //         be used. The indices are 0-based. For example, use the
> following to
> //         request a GRIB2 record whose 9-th and 27-th product
description
> //         template values are 1 and 2, respectively:
> //            GRIB2_ipdtmpl_index=[8, 26]; GRIB2_ipdtmpl_val=[1, 2];
>
> While I don't have any ECMWF data on hand for testing, I do have a
file
> from the Air Force from which wgrib2 prints a similar result:
>
> 549:687031550:d=2017061000:var discipline=0 center=57 local_table=1
> parmcat=1 parm=192:surface:18 hour fcst:
>
> So I used that for testing. I ran the following plot_data_plane
command
> WITHOUT specifying a requested name:
>
>
> plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps 'GRIB2_disc=0;
> GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
level="Z0";'
> -v 3
>
> And then I looked closely at the log message to confirm that it used
> record 549, as expected:
>
> DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for
> 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>
> So MET used these GRIB2 numbers I defined to do a table lookup and
find a
> match in the MET GRIB tables:
> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>
> And it determined that that field should be called "CTSTM".
Hopefully, the
> ECMWF GRIB2 tables are already defined there in MET. If not, we can
always
> add them.
>
> Please try running plot_data_plane at verbosity level 3 on one of
these
> files first. And make sure it runs without error and the log message
prints
> out the "name" it found for that data. If so, you can just skip
using all
> those integers and revert back to using the name MET knows. If not,
then we
> may need to define a custom GRIB table for this ECMWF GRIB2 data.
>
> Thanks,
> John
>


--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Fri Dec 04 15:31:01 2020

We might already have them. I'm not sure.

But there's an additional required piece of the puzzle. Binbin runs
"wgrib2" to figure out what "name" he should use in the MET config
files.
If wgrib2 doesn't tell him the right "name", the problem will remain.
wgrib2 can be configured to define additional tables as well, but for
this
to be easy for Binbin, that'd mean updating the script that's run for
"module load grib_utils" on WCOSS. And I'm guessing that's not very
easy.

Yes, I do agree that the best solution is getting the ECMWF GRIB
tables and
adding them for both wgrib2 and MET.

John

On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu> wrote:

> John,
>
> It seems to me that it would be worthwhile to just ask ECMWF for
their
> Grib2 Table for their Ensemble files.  And other operational centers
for
> that matter.  Do you agree this would be a good solution?
>
> Thanks, Tara
>
> ---------- Forwarded message ---------
> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> Date: Fri, Dec 4, 2020 at 11:06 AM
> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for reading
ECMWF
> ensmeble data
> To:
>
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   I tested using following command:
>
>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3
>
> ERROR  :
> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> GRIB2_parm_cat and GRIB2_parm must be specified in field information
>
>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v
> 3
>
> Still got error
> ERROR  :
> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> GRIB2_parm_cat and GRIB2_parm must be specified in field information
>
> Looks like discipline=255 is not recognized
>
> Binbin
>
>
> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binbin,
> > You can do something similar with MET. Please go to this link:
> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> >
> > And then search for "GRIB2_mtab", and you'll find the content
> > copied/pasted below. You can specify these INSTEAD OF specifying a
> > requested name.
> >
> > //       - The "GRIB2_mtab" is an integer specifying the master
table
> > number.
> > //       - The "GRIB2_ltab" is an integer specifying the local
table
> > number.
> > //       - The "GRIB2_disc" is an integer specifying the GRIB2
discipline
> > code.
> > //       - The "GRIB2_parm_cat" is an integer specifying the
parameter
> > category
> > //         code.
> > //       - The "GRIB2_parm" is an integer specifying the parameter
code.
> > //       - The "GRIB2_pdt" is an integer specifying the product
> definition
> > //         template (Table 4.0).
> > //       - The "GRIB2_process" is an integer specifying the
generating
> > process
> > //         (Table 4.3).
> > //       - The "GRIB2_cntr" is an integer specifying the
originating
> > center.
> > //       - The "GRIB2_ens_type" is an integer specifying the
ensemble
> type
> > //         (Table 4.6).
> > //       - The "GRIB2_der_type" is an integer specifying the
derived
> > product
> > //         type (Table 4.7).
> > //       - The "GRIB2_stat_type" is an integer specifying the
statistical
> > //         processing type (Table 4.10).
> > //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val"
entries are
> > arrays
> > //         of integers which specify the product description
template
> > values to
> > //         be used. The indices are 0-based. For example, use the
> > following to
> > //         request a GRIB2 record whose 9-th and 27-th product
> description
> > //         template values are 1 and 2, respectively:
> > //            GRIB2_ipdtmpl_index=[8, 26]; GRIB2_ipdtmpl_val=[1,
2];
> >
> > While I don't have any ECMWF data on hand for testing, I do have a
file
> > from the Air Force from which wgrib2 prints a similar result:
> >
> > 549:687031550:d=2017061000:var discipline=0 center=57
local_table=1
> > parmcat=1 parm=192:surface:18 hour fcst:
> >
> > So I used that for testing. I ran the following plot_data_plane
command
> > WITHOUT specifying a requested name:
> >
> >
> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
'GRIB2_disc=0;
> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
> level="Z0";'
> > -v 3
> >
> > And then I looked closely at the log message to confirm that it
used
> > record 549, as expected:
> >
> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for
> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> >
> > So MET used these GRIB2 numbers I defined to do a table lookup and
find a
> > match in the MET GRIB tables:
> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> >
> > And it determined that that field should be called "CTSTM".
Hopefully,
> the
> > ECMWF GRIB2 tables are already defined there in MET. If not, we
can
> always
> > add them.
> >
> > Please try running plot_data_plane at verbosity level 3 on one of
these
> > files first. And make sure it runs without error and the log
message
> prints
> > out the "name" it found for that data. If so, you can just skip
using all
> > those integers and revert back to using the name MET knows. If
not, then
> we
> > may need to define a custom GRIB table for this ECMWF GRIB2 data.
> >
> > Thanks,
> > John
> >
>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Tara Jensen
> Project Manager II
> NCAR RAL and DTC
> PO Box 3000, Boulder, Colorado 80307 USA
> +1 303-497-8479          jensen at ucar.edu
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Fri Dec 04 15:48:11 2020

Binbin,

OK, working on Venus.

You have a typo in the command you sent.
 plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v
3

You have "RIB2_parm_cat" that should be "GRIB2_parm_cat". After
correcting
that I did still get an error but it's a very useful one!

module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
module load met/9.1
module load grib_util/1.1.1
cd
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
$WGRIB2 ensposte.t00z.z500.grd3.grib2
plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'GRIB2_disc=255;
GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'



*ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter found with
matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5). Use the
MET_GRIB_TABLES environment variable to define custom GRIB
tables.ERROR  : *

So this table is not yet defined in MET. I checked their website:
https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2

But I don't see an entry defined for GRIB2_parm=5;

3 things...

(1) As a crude temporary solution, you could refer to it by it's
record
number. Let's assume you know that record 1 has temperature data:

plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps 'name="TMP";
level="R1"; set_attr_level="P500";' -v 5

Here I'm reading record 1, naming it TMP and the using set_attr_level
write
the output level as P500 instead of R1.

(2) You could create a custom GRIB table yourself locally, and then
set the
MET_GRIB_TABLES environment variable to point MET to it.

(3) The best solution is contacting ECMWF and requesting they send
over the
GRIB2 table we should be using to decode this data.

Thanks,
John

On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:

> We might already have them. I'm not sure.
>
> But there's an additional required piece of the puzzle. Binbin runs
> "wgrib2" to figure out what "name" he should use in the MET config
files.
> If wgrib2 doesn't tell him the right "name", the problem will
remain.
> wgrib2 can be configured to define additional tables as well, but
for this
> to be easy for Binbin, that'd mean updating the script that's run
for
> "module load grib_utils" on WCOSS. And I'm guessing that's not very
easy.
>
> Yes, I do agree that the best solution is getting the ECMWF GRIB
tables
> and adding them for both wgrib2 and MET.
>
> John
>
> On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu> wrote:
>
>> John,
>>
>> It seems to me that it would be worthwhile to just ask ECMWF for
their
>> Grib2 Table for their Ensemble files.  And other operational
centers for
>> that matter.  Do you agree this would be a good solution?
>>
>> Thanks, Tara
>>
>> ---------- Forwarded message ---------
>> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
>> Date: Fri, Dec 4, 2020 at 11:06 AM
>> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for reading
ECMWF
>> ensmeble data
>> To:
>>
>>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>>
>> John,
>>
>>   I tested using following command:
>>
>>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3
>>
>> ERROR  :
>> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
>> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
>>
>>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> -v
>> 3
>>
>> Still got error
>> ERROR  :
>> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
>> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
>>
>> Looks like discipline=255 is not recognized
>>
>> Binbin
>>
>>
>> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Binbin,
>> > You can do something similar with MET. Please go to this link:
>> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>> >
>> > And then search for "GRIB2_mtab", and you'll find the content
>> > copied/pasted below. You can specify these INSTEAD OF specifying
a
>> > requested name.
>> >
>> > //       - The "GRIB2_mtab" is an integer specifying the master
table
>> > number.
>> > //       - The "GRIB2_ltab" is an integer specifying the local
table
>> > number.
>> > //       - The "GRIB2_disc" is an integer specifying the GRIB2
>> discipline
>> > code.
>> > //       - The "GRIB2_parm_cat" is an integer specifying the
parameter
>> > category
>> > //         code.
>> > //       - The "GRIB2_parm" is an integer specifying the
parameter code.
>> > //       - The "GRIB2_pdt" is an integer specifying the product
>> definition
>> > //         template (Table 4.0).
>> > //       - The "GRIB2_process" is an integer specifying the
generating
>> > process
>> > //         (Table 4.3).
>> > //       - The "GRIB2_cntr" is an integer specifying the
originating
>> > center.
>> > //       - The "GRIB2_ens_type" is an integer specifying the
ensemble
>> type
>> > //         (Table 4.6).
>> > //       - The "GRIB2_der_type" is an integer specifying the
derived
>> > product
>> > //         type (Table 4.7).
>> > //       - The "GRIB2_stat_type" is an integer specifying the
>> statistical
>> > //         processing type (Table 4.10).
>> > //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val"
entries are
>> > arrays
>> > //         of integers which specify the product description
template
>> > values to
>> > //         be used. The indices are 0-based. For example, use the
>> > following to
>> > //         request a GRIB2 record whose 9-th and 27-th product
>> description
>> > //         template values are 1 and 2, respectively:
>> > //            GRIB2_ipdtmpl_index=[8, 26]; GRIB2_ipdtmpl_val=[1,
2];
>> >
>> > While I don't have any ECMWF data on hand for testing, I do have
a file
>> > from the Air Force from which wgrib2 prints a similar result:
>> >
>> > 549:687031550:d=2017061000:var discipline=0 center=57
local_table=1
>> > parmcat=1 parm=192:surface:18 hour fcst:
>> >
>> > So I used that for testing. I ran the following plot_data_plane
command
>> > WITHOUT specifying a requested name:
>> >
>> >
>> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
'GRIB2_disc=0;
>> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
>> level="Z0";'
>> > -v 3
>> >
>> > And then I looked closely at the log message to confirm that it
used
>> > record 549, as expected:
>> >
>> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for
>> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
>> >
>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>> >
>> > So MET used these GRIB2 numbers I defined to do a table lookup
and find
>> a
>> > match in the MET GRIB tables:
>> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>> >
>> > And it determined that that field should be called "CTSTM".
Hopefully,
>> the
>> > ECMWF GRIB2 tables are already defined there in MET. If not, we
can
>> always
>> > add them.
>> >
>> > Please try running plot_data_plane at verbosity level 3 on one of
these
>> > files first. And make sure it runs without error and the log
message
>> prints
>> > out the "name" it found for that data. If so, you can just skip
using
>> all
>> > those integers and revert back to using the name MET knows. If
not,
>> then we
>> > may need to define a custom GRIB table for this ECMWF GRIB2 data.
>> >
>> > Thanks,
>> > John
>> >
>>
>>
>> --
>>
>> Binbin Zhou
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct.
>>
>> College Park, MD 20740
>>
>> Binbin.Zhou at noaa.gov
>>
>> 301-683-3683
>>
>>
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Tara Jensen
>> Project Manager II
>> NCAR RAL and DTC
>> PO Box 3000, Boulder, Colorado 80307 USA
>> +1 303-497-8479          jensen at ucar.edu
>>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Mon Dec 07 14:49:00 2020

John,

  Our ECMWF grib2 ensemble data files are converted from its grib1
format
data.
Before conversion, the parameter names are there, but after
conversion, the
names
are gone. It is because of unknown tables. The grib1 data is still
original. The original
grib1 data files are very strange data format we can not read. So we
have
to use
a Frotran code to re-arrange the fields and save them as NCEP-like
grib1
data files.
So after several processes, some important info like tables could be
lost.

   I'll test your first suggestion since the field record order is
fixed.
Currently we still
obtained grib1 data from ECWMF, eventually, we will get grib2 data
directly. So before
we can get original grib2 files from ECMWF, we just use this temporal
solution.
'
Thanks!


On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
>
> OK, working on Venus.
>
> You have a typo in the command you sent.
>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v
> 3
>
> You have "RIB2_parm_cat" that should be "GRIB2_parm_cat". After
correcting
> that I did still get an error but it's a very useful one!
>
> module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> module load met/9.1
> module load grib_util/1.1.1
> cd
>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> $WGRIB2 ensposte.t00z.z500.grd3.grib2
> plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'GRIB2_disc=255;
> GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'
>
>
>
> *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter found
with
> matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5). Use the
> MET_GRIB_TABLES environment variable to define custom GRIB
tables.ERROR  :
> *
>
> So this table is not yet defined in MET. I checked their website:
>
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>
> But I don't see an entry defined for GRIB2_parm=5;
>
> 3 things...
>
> (1) As a crude temporary solution, you could refer to it by it's
record
> number. Let's assume you know that record 1 has temperature data:
>
> plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'name="TMP";
> level="R1"; set_attr_level="P500";' -v 5
>
> Here I'm reading record 1, naming it TMP and the using
set_attr_level write
> the output level as P500 instead of R1.
>
> (2) You could create a custom GRIB table yourself locally, and then
set the
> MET_GRIB_TABLES environment variable to point MET to it.
>
> (3) The best solution is contacting ECMWF and requesting they send
over the
> GRIB2 table we should be using to decode this data.
>
> Thanks,
> John
>
> On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > We might already have them. I'm not sure.
> >
> > But there's an additional required piece of the puzzle. Binbin
runs
> > "wgrib2" to figure out what "name" he should use in the MET config
files.
> > If wgrib2 doesn't tell him the right "name", the problem will
remain.
> > wgrib2 can be configured to define additional tables as well, but
for
> this
> > to be easy for Binbin, that'd mean updating the script that's run
for
> > "module load grib_utils" on WCOSS. And I'm guessing that's not
very easy.
> >
> > Yes, I do agree that the best solution is getting the ECMWF GRIB
tables
> > and adding them for both wgrib2 and MET.
> >
> > John
> >
> > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu>
wrote:
> >
> >> John,
> >>
> >> It seems to me that it would be worthwhile to just ask ECMWF for
their
> >> Grib2 Table for their Ensemble files.  And other operational
centers for
> >> that matter.  Do you agree this would be a good solution?
> >>
> >> Thanks, Tara
> >>
> >> ---------- Forwarded message ---------
> >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> >> Date: Fri, Dec 4, 2020 at 11:06 AM
> >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for reading
ECMWF
> >> ensmeble data
> >> To:
> >>
> >>
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >>
> >> John,
> >>
> >>   I tested using following command:
> >>
> >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3
> >>
> >> ERROR  :
> >> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> >>
> >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> -v
> >> 3
> >>
> >> Still got error
> >> ERROR  :
> >> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> >>
> >> Looks like discipline=255 is not recognized
> >>
> >> Binbin
> >>
> >>
> >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Binbin,
> >> > You can do something similar with MET. Please go to this link:
> >> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> >> >
> >> > And then search for "GRIB2_mtab", and you'll find the content
> >> > copied/pasted below. You can specify these INSTEAD OF
specifying a
> >> > requested name.
> >> >
> >> > //       - The "GRIB2_mtab" is an integer specifying the master
table
> >> > number.
> >> > //       - The "GRIB2_ltab" is an integer specifying the local
table
> >> > number.
> >> > //       - The "GRIB2_disc" is an integer specifying the GRIB2
> >> discipline
> >> > code.
> >> > //       - The "GRIB2_parm_cat" is an integer specifying the
parameter
> >> > category
> >> > //         code.
> >> > //       - The "GRIB2_parm" is an integer specifying the
parameter
> code.
> >> > //       - The "GRIB2_pdt" is an integer specifying the product
> >> definition
> >> > //         template (Table 4.0).
> >> > //       - The "GRIB2_process" is an integer specifying the
generating
> >> > process
> >> > //         (Table 4.3).
> >> > //       - The "GRIB2_cntr" is an integer specifying the
originating
> >> > center.
> >> > //       - The "GRIB2_ens_type" is an integer specifying the
ensemble
> >> type
> >> > //         (Table 4.6).
> >> > //       - The "GRIB2_der_type" is an integer specifying the
derived
> >> > product
> >> > //         type (Table 4.7).
> >> > //       - The "GRIB2_stat_type" is an integer specifying the
> >> statistical
> >> > //         processing type (Table 4.10).
> >> > //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val"
entries
> are
> >> > arrays
> >> > //         of integers which specify the product description
template
> >> > values to
> >> > //         be used. The indices are 0-based. For example, use
the
> >> > following to
> >> > //         request a GRIB2 record whose 9-th and 27-th product
> >> description
> >> > //         template values are 1 and 2, respectively:
> >> > //            GRIB2_ipdtmpl_index=[8, 26];
GRIB2_ipdtmpl_val=[1, 2];
> >> >
> >> > While I don't have any ECMWF data on hand for testing, I do
have a
> file
> >> > from the Air Force from which wgrib2 prints a similar result:
> >> >
> >> > 549:687031550:d=2017061000:var discipline=0 center=57
local_table=1
> >> > parmcat=1 parm=192:surface:18 hour fcst:
> >> >
> >> > So I used that for testing. I ran the following plot_data_plane
> command
> >> > WITHOUT specifying a requested name:
> >> >
> >> >
> >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
'GRIB2_disc=0;
> >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1; GRIB2_parm=192;
> >> level="Z0";'
> >> > -v 3
> >> >
> >> > And then I looked closely at the log message to confirm that it
used
> >> > record 549, as expected:
> >> >
> >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match
for
> >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> >> >
> >>
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> >> >
> >> > So MET used these GRIB2 numbers I defined to do a table lookup
and
> find
> >> a
> >> > match in the MET GRIB tables:
> >> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> >> >
> >> > And it determined that that field should be called "CTSTM".
Hopefully,
> >> the
> >> > ECMWF GRIB2 tables are already defined there in MET. If not, we
can
> >> always
> >> > add them.
> >> >
> >> > Please try running plot_data_plane at verbosity level 3 on one
of
> these
> >> > files first. And make sure it runs without error and the log
message
> >> prints
> >> > out the "name" it found for that data. If so, you can just skip
using
> >> all
> >> > those integers and revert back to using the name MET knows. If
not,
> >> then we
> >> > may need to define a custom GRIB table for this ECMWF GRIB2
data.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >>
> >>
> >> --
> >>
> >> Binbin Zhou
> >>
> >> IMSG at NOAA/NWS/NCEP/EMC
> >>
> >> 5830 University Research Ct.
> >>
> >> College Park, MD 20740
> >>
> >> Binbin.Zhou at noaa.gov
> >>
> >> 301-683-3683
> >>
> >>
> >>
> >> --
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> Tara Jensen
> >> Project Manager II
> >> NCAR RAL and DTC
> >> PO Box 3000, Boulder, Colorado 80307 USA
> >> +1 303-497-8479          jensen at ucar.edu
> >>
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Tue Dec 08 13:56:51 2020

Binbin,

Sounds like there's a lot of moving parts there. How would you like to
proceed?

Should we check to see if MET can read those ECMWF GRIB1 files
directly
without needing to reformat?
Or do you just want to create a custom GRIB table for the result of
your
pre-processing and conversion steps?

John

On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   Our ECMWF grib2 ensemble data files are converted from its grib1
format
> data.
> Before conversion, the parameter names are there, but after
conversion, the
> names
> are gone. It is because of unknown tables. The grib1 data is still
> original. The original
> grib1 data files are very strange data format we can not read. So we
have
> to use
> a Frotran code to re-arrange the fields and save them as NCEP-like
grib1
> data files.
> So after several processes, some important info like tables could be
lost.
>
>    I'll test your first suggestion since the field record order is
fixed.
> Currently we still
> obtained grib1 data from ECWMF, eventually, we will get grib2 data
> directly. So before
> we can get original grib2 files from ECMWF, we just use this
temporal
> solution.
> '
> Thanks!
>
>
> On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binbin,
> >
> > OK, working on Venus.
> >
> > You have a typo in the command you sent.
> >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> -v
> > 3
> >
> > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat". After
> correcting
> > that I did still get an error but it's a very useful one!
> >
> > module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > module load met/9.1
> > module load grib_util/1.1.1
> > cd
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> 'GRIB2_disc=255;
> > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'
> >
> >
> >
> > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter found
with
> > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5). Use
the
> > MET_GRIB_TABLES environment variable to define custom GRIB
tables.ERROR
> :
> > *
> >
> > So this table is not yet defined in MET. I checked their website:
> >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> >
> > But I don't see an entry defined for GRIB2_parm=5;
> >
> > 3 things...
> >
> > (1) As a crude temporary solution, you could refer to it by it's
record
> > number. Let's assume you know that record 1 has temperature data:
> >
> > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'name="TMP";
> > level="R1"; set_attr_level="P500";' -v 5
> >
> > Here I'm reading record 1, naming it TMP and the using
set_attr_level
> write
> > the output level as P500 instead of R1.
> >
> > (2) You could create a custom GRIB table yourself locally, and
then set
> the
> > MET_GRIB_TABLES environment variable to point MET to it.
> >
> > (3) The best solution is contacting ECMWF and requesting they send
over
> the
> > GRIB2 table we should be using to decode this data.
> >
> > Thanks,
> > John
> >
> > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > We might already have them. I'm not sure.
> > >
> > > But there's an additional required piece of the puzzle. Binbin
runs
> > > "wgrib2" to figure out what "name" he should use in the MET
config
> files.
> > > If wgrib2 doesn't tell him the right "name", the problem will
remain.
> > > wgrib2 can be configured to define additional tables as well,
but for
> > this
> > > to be easy for Binbin, that'd mean updating the script that's
run for
> > > "module load grib_utils" on WCOSS. And I'm guessing that's not
very
> easy.
> > >
> > > Yes, I do agree that the best solution is getting the ECMWF GRIB
tables
> > > and adding them for both wgrib2 and MET.
> > >
> > > John
> > >
> > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu>
wrote:
> > >
> > >> John,
> > >>
> > >> It seems to me that it would be worthwhile to just ask ECMWF
for their
> > >> Grib2 Table for their Ensemble files.  And other operational
centers
> for
> > >> that matter.  Do you agree this would be a good solution?
> > >>
> > >> Thanks, Tara
> > >>
> > >> ---------- Forwarded message ---------
> > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for
reading
> ECMWF
> > >> ensmeble data
> > >> To:
> > >>
> > >>
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >>
> > >> John,
> > >>
> > >>   I tested using following command:
> > >>
> > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3
> > >>
> > >> ERROR  :
> > >> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> > >>
> > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >>
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> -v
> > >> 3
> > >>
> > >> Still got error
> > >> ERROR  :
> > >> ERROR  : VarInfoGrib2::set_dict() -> either name or GRIB2_disc,
> > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> > >>
> > >> Looks like discipline=255 is not recognized
> > >>
> > >> Binbin
> > >>
> > >>
> > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Binbin,
> > >> > You can do something similar with MET. Please go to this
link:
> > >> >
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > >> >
> > >> > And then search for "GRIB2_mtab", and you'll find the content
> > >> > copied/pasted below. You can specify these INSTEAD OF
specifying a
> > >> > requested name.
> > >> >
> > >> > //       - The "GRIB2_mtab" is an integer specifying the
master
> table
> > >> > number.
> > >> > //       - The "GRIB2_ltab" is an integer specifying the
local table
> > >> > number.
> > >> > //       - The "GRIB2_disc" is an integer specifying the
GRIB2
> > >> discipline
> > >> > code.
> > >> > //       - The "GRIB2_parm_cat" is an integer specifying the
> parameter
> > >> > category
> > >> > //         code.
> > >> > //       - The "GRIB2_parm" is an integer specifying the
parameter
> > code.
> > >> > //       - The "GRIB2_pdt" is an integer specifying the
product
> > >> definition
> > >> > //         template (Table 4.0).
> > >> > //       - The "GRIB2_process" is an integer specifying the
> generating
> > >> > process
> > >> > //         (Table 4.3).
> > >> > //       - The "GRIB2_cntr" is an integer specifying the
originating
> > >> > center.
> > >> > //       - The "GRIB2_ens_type" is an integer specifying the
> ensemble
> > >> type
> > >> > //         (Table 4.6).
> > >> > //       - The "GRIB2_der_type" is an integer specifying the
derived
> > >> > product
> > >> > //         type (Table 4.7).
> > >> > //       - The "GRIB2_stat_type" is an integer specifying the
> > >> statistical
> > >> > //         processing type (Table 4.10).
> > >> > //       - The "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val"
entries
> > are
> > >> > arrays
> > >> > //         of integers which specify the product description
> template
> > >> > values to
> > >> > //         be used. The indices are 0-based. For example, use
the
> > >> > following to
> > >> > //         request a GRIB2 record whose 9-th and 27-th
product
> > >> description
> > >> > //         template values are 1 and 2, respectively:
> > >> > //            GRIB2_ipdtmpl_index=[8, 26];
GRIB2_ipdtmpl_val=[1, 2];
> > >> >
> > >> > While I don't have any ECMWF data on hand for testing, I do
have a
> > file
> > >> > from the Air Force from which wgrib2 prints a similar result:
> > >> >
> > >> > 549:687031550:d=2017061000:var discipline=0 center=57
local_table=1
> > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > >> >
> > >> > So I used that for testing. I ran the following
plot_data_plane
> > command
> > >> > WITHOUT specifying a requested name:
> > >> >
> > >> >
> > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
> 'GRIB2_disc=0;
> > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
GRIB2_parm=192;
> > >> level="Z0";'
> > >> > -v 3
> > >> >
> > >> > And then I looked closely at the log message to confirm that
it used
> > >> > record 549, as expected:
> > >> >
> > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match
for
> > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> > >> >
> > >>
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > >> >
> > >> > So MET used these GRIB2 numbers I defined to do a table
lookup and
> > find
> > >> a
> > >> > match in the MET GRIB tables:
> > >> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > >> >
> > >> > And it determined that that field should be called "CTSTM".
> Hopefully,
> > >> the
> > >> > ECMWF GRIB2 tables are already defined there in MET. If not,
we can
> > >> always
> > >> > add them.
> > >> >
> > >> > Please try running plot_data_plane at verbosity level 3 on
one of
> > these
> > >> > files first. And make sure it runs without error and the log
message
> > >> prints
> > >> > out the "name" it found for that data. If so, you can just
skip
> using
> > >> all
> > >> > those integers and revert back to using the name MET knows.
If not,
> > >> then we
> > >> > may need to define a custom GRIB table for this ECMWF GRIB2
data.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Binbin Zhou
> > >>
> > >> IMSG at NOAA/NWS/NCEP/EMC
> > >>
> > >> 5830 University Research Ct.
> > >>
> > >> College Park, MD 20740
> > >>
> > >> Binbin.Zhou at noaa.gov
> > >>
> > >> 301-683-3683
> > >>
> > >>
> > >>
> > >> --
> > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >> Tara Jensen
> > >> Project Manager II
> > >> NCAR RAL and DTC
> > >> PO Box 3000, Boulder, Colorado 80307 USA
> > >> +1 303-497-8479          jensen at ucar.edu
> > >>
> > >
> >
> >
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Tue Dec 08 14:37:07 2020

John,

  Grib1 data is too large, another issue is the low-res control
analysis
data can not be retrieved
from the grib1 ecmwf data files. If using wgrib, we can see 3 analysis
fields, but don't know
which is low-res control. After convert to grib2, wgrib2 can show
which
record is low-res control.
Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.

Binbin

On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
>
> Sounds like there's a lot of moving parts there. How would you like
to
> proceed?
>
> Should we check to see if MET can read those ECMWF GRIB1 files
directly
> without needing to reformat?
> Or do you just want to create a custom GRIB table for the result of
your
> pre-processing and conversion steps?
>
> John
>
> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > John,
> >
> >   Our ECMWF grib2 ensemble data files are converted from its grib1
format
> > data.
> > Before conversion, the parameter names are there, but after
conversion,
> the
> > names
> > are gone. It is because of unknown tables. The grib1 data is still
> > original. The original
> > grib1 data files are very strange data format we can not read. So
we have
> > to use
> > a Frotran code to re-arrange the fields and save them as NCEP-like
grib1
> > data files.
> > So after several processes, some important info like tables could
be
> lost.
> >
> >    I'll test your first suggestion since the field record order is
fixed.
> > Currently we still
> > obtained grib1 data from ECWMF, eventually, we will get grib2 data
> > directly. So before
> > we can get original grib2 files from ECMWF, we just use this
temporal
> > solution.
> > '
> > Thanks!
> >
> >
> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binbin,
> > >
> > > OK, working on Venus.
> > >
> > > You have a typo in the command you sent.
> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > -v
> > > 3
> > >
> > > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat". After
> > correcting
> > > that I did still get an error but it's a very useful one!
> > >
> > > module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > module load met/9.1
> > > module load grib_util/1.1.1
> > > cd
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> > 'GRIB2_disc=255;
> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'
> > >
> > >
> > >
> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter
found with
> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5). Use
the
> > > MET_GRIB_TABLES environment variable to define custom GRIB
tables.ERROR
> > :
> > > *
> > >
> > > So this table is not yet defined in MET. I checked their
website:
> > >
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > >
> > > But I don't see an entry defined for GRIB2_parm=5;
> > >
> > > 3 things...
> > >
> > > (1) As a crude temporary solution, you could refer to it by it's
record
> > > number. Let's assume you know that record 1 has temperature
data:
> > >
> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'name="TMP";
> > > level="R1"; set_attr_level="P500";' -v 5
> > >
> > > Here I'm reading record 1, naming it TMP and the using
set_attr_level
> > write
> > > the output level as P500 instead of R1.
> > >
> > > (2) You could create a custom GRIB table yourself locally, and
then set
> > the
> > > MET_GRIB_TABLES environment variable to point MET to it.
> > >
> > > (3) The best solution is contacting ECMWF and requesting they
send over
> > the
> > > GRIB2 table we should be using to decode this data.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > We might already have them. I'm not sure.
> > > >
> > > > But there's an additional required piece of the puzzle. Binbin
runs
> > > > "wgrib2" to figure out what "name" he should use in the MET
config
> > files.
> > > > If wgrib2 doesn't tell him the right "name", the problem will
remain.
> > > > wgrib2 can be configured to define additional tables as well,
but for
> > > this
> > > > to be easy for Binbin, that'd mean updating the script that's
run for
> > > > "module load grib_utils" on WCOSS. And I'm guessing that's not
very
> > easy.
> > > >
> > > > Yes, I do agree that the best solution is getting the ECMWF
GRIB
> tables
> > > > and adding them for both wgrib2 and MET.
> > > >
> > > > John
> > > >
> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu>
wrote:
> > > >
> > > >> John,
> > > >>
> > > >> It seems to me that it would be worthwhile to just ask ECMWF
for
> their
> > > >> Grib2 Table for their Ensemble files.  And other operational
centers
> > for
> > > >> that matter.  Do you agree this would be a good solution?
> > > >>
> > > >> Thanks, Tara
> > > >>
> > > >> ---------- Forwarded message ---------
> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for
reading
> > ECMWF
> > > >> ensmeble data
> > > >> To:
> > > >>
> > > >>
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > > >>
> > > >> John,
> > > >>
> > > >>   I tested using following command:
> > > >>
> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > > >> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v3
> > > >>
> > > >> ERROR  :
> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> > > >>
> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > > >>
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >> -v
> > > >> 3
> > > >>
> > > >> Still got error
> > > >> ERROR  :
> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
information
> > > >>
> > > >> Looks like discipline=255 is not recognized
> > > >>
> > > >> Binbin
> > > >>
> > > >>
> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
> > > >> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> > Binbin,
> > > >> > You can do something similar with MET. Please go to this
link:
> > > >> >
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > >> >
> > > >> > And then search for "GRIB2_mtab", and you'll find the
content
> > > >> > copied/pasted below. You can specify these INSTEAD OF
specifying a
> > > >> > requested name.
> > > >> >
> > > >> > //       - The "GRIB2_mtab" is an integer specifying the
master
> > table
> > > >> > number.
> > > >> > //       - The "GRIB2_ltab" is an integer specifying the
local
> table
> > > >> > number.
> > > >> > //       - The "GRIB2_disc" is an integer specifying the
GRIB2
> > > >> discipline
> > > >> > code.
> > > >> > //       - The "GRIB2_parm_cat" is an integer specifying
the
> > parameter
> > > >> > category
> > > >> > //         code.
> > > >> > //       - The "GRIB2_parm" is an integer specifying the
parameter
> > > code.
> > > >> > //       - The "GRIB2_pdt" is an integer specifying the
product
> > > >> definition
> > > >> > //         template (Table 4.0).
> > > >> > //       - The "GRIB2_process" is an integer specifying the
> > generating
> > > >> > process
> > > >> > //         (Table 4.3).
> > > >> > //       - The "GRIB2_cntr" is an integer specifying the
> originating
> > > >> > center.
> > > >> > //       - The "GRIB2_ens_type" is an integer specifying
the
> > ensemble
> > > >> type
> > > >> > //         (Table 4.6).
> > > >> > //       - The "GRIB2_der_type" is an integer specifying
the
> derived
> > > >> > product
> > > >> > //         type (Table 4.7).
> > > >> > //       - The "GRIB2_stat_type" is an integer specifying
the
> > > >> statistical
> > > >> > //         processing type (Table 4.10).
> > > >> > //       - The "GRIB2_ipdtmpl_index" and
"GRIB2_ipdtmpl_val"
> entries
> > > are
> > > >> > arrays
> > > >> > //         of integers which specify the product
description
> > template
> > > >> > values to
> > > >> > //         be used. The indices are 0-based. For example,
use the
> > > >> > following to
> > > >> > //         request a GRIB2 record whose 9-th and 27-th
product
> > > >> description
> > > >> > //         template values are 1 and 2, respectively:
> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
GRIB2_ipdtmpl_val=[1,
> 2];
> > > >> >
> > > >> > While I don't have any ECMWF data on hand for testing, I do
have a
> > > file
> > > >> > from the Air Force from which wgrib2 prints a similar
result:
> > > >> >
> > > >> > 549:687031550:d=2017061000:var discipline=0 center=57
> local_table=1
> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > > >> >
> > > >> > So I used that for testing. I ran the following
plot_data_plane
> > > command
> > > >> > WITHOUT specifying a requested name:
> > > >> >
> > > >> >
> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
> > 'GRIB2_disc=0;
> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
GRIB2_parm=192;
> > > >> level="Z0";'
> > > >> > -v 3
> > > >> >
> > > >> > And then I looked closely at the log message to confirm
that it
> used
> > > >> > record 549, as expected:
> > > >> >
> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact
match for
> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> > > >> >
> > > >>
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > >> >
> > > >> > So MET used these GRIB2 numbers I defined to do a table
lookup and
> > > find
> > > >> a
> > > >> > match in the MET GRIB tables:
> > > >> >
> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > >> >
> > > >> > And it determined that that field should be called "CTSTM".
> > Hopefully,
> > > >> the
> > > >> > ECMWF GRIB2 tables are already defined there in MET. If
not, we
> can
> > > >> always
> > > >> > add them.
> > > >> >
> > > >> > Please try running plot_data_plane at verbosity level 3 on
one of
> > > these
> > > >> > files first. And make sure it runs without error and the
log
> message
> > > >> prints
> > > >> > out the "name" it found for that data. If so, you can just
skip
> > using
> > > >> all
> > > >> > those integers and revert back to using the name MET knows.
If
> not,
> > > >> then we
> > > >> > may need to define a custom GRIB table for this ECMWF GRIB2
data.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Binbin Zhou
> > > >>
> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > >>
> > > >> 5830 University Research Ct.
> > > >>
> > > >> College Park, MD 20740
> > > >>
> > > >> Binbin.Zhou at noaa.gov
> > > >>
> > > >> 301-683-3683
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >> Tara Jensen
> > > >> Project Manager II
> > > >> NCAR RAL and DTC
> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > > >> +1 303-497-8479          jensen at ucar.edu
> > > >>
> > > >
> > >
> > >
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Tue Dec 08 14:51:26 2020

John,

  Look at my
directory /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
The grib1 file is ensposte.t00z.z500hr, this is only for 500 HGT.
wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
I get 3 records:

11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
mb:anl:type=analysis:NAve=0
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
mb:anl:type=analysis:NAve=0
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
mb:anl:type=analysis:NAve=0

>From its grib2 file, we know only  record#26 is analysis low-res
control.
If you have some way to determine which record is analysis low-res
control
from the grib1 file,
then we can use grib1 directly (don't consider the size issue).

Binbin

On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
binbin.zhou at noaa.gov> wrote:

> John,
>
>   Grib1 data is too large, another issue is the low-res control
analysis
> data can not be retrieved
> from the grib1 ecmwf data files. If using wgrib, we can see 3
analysis
> fields, but don't know
> which is low-res control. After convert to grib2, wgrib2 can show
which
> record is low-res control.
> Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
>
> Binbin
>
> On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binbin,
>>
>> Sounds like there's a lot of moving parts there. How would you like
to
>> proceed?
>>
>> Should we check to see if MET can read those ECMWF GRIB1 files
directly
>> without needing to reformat?
>> Or do you just want to create a custom GRIB table for the result of
your
>> pre-processing and conversion steps?
>>
>> John
>>
>> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> >
>> > John,
>> >
>> >   Our ECMWF grib2 ensemble data files are converted from its
grib1
>> format
>> > data.
>> > Before conversion, the parameter names are there, but after
conversion,
>> the
>> > names
>> > are gone. It is because of unknown tables. The grib1 data is
still
>> > original. The original
>> > grib1 data files are very strange data format we can not read. So
we
>> have
>> > to use
>> > a Frotran code to re-arrange the fields and save them as NCEP-
like grib1
>> > data files.
>> > So after several processes, some important info like tables could
be
>> lost.
>> >
>> >    I'll test your first suggestion since the field record order
is
>> fixed.
>> > Currently we still
>> > obtained grib1 data from ECWMF, eventually, we will get grib2
data
>> > directly. So before
>> > we can get original grib2 files from ECMWF, we just use this
temporal
>> > solution.
>> > '
>> > Thanks!
>> >
>> >
>> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Binbin,
>> > >
>> > > OK, working on Venus.
>> > >
>> > > You have a typo in the command you sent.
>> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > >
>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > -v
>> > > 3
>> > >
>> > > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat". After
>> > correcting
>> > > that I did still get an error but it's a very useful one!
>> > >
>> > > module use
>> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>> > > module load met/9.1
>> > > module load grib_util/1.1.1
>> > > cd
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
>> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
>> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
>> > 'GRIB2_disc=255;
>> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'
>> > >
>> > >
>> > >
>> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter
found with
>> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5).
Use the
>> > > MET_GRIB_TABLES environment variable to define custom GRIB
>> tables.ERROR
>> > :
>> > > *
>> > >
>> > > So this table is not yet defined in MET. I checked their
website:
>> > >
>> > >
>> >
>> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>> > >
>> > > But I don't see an entry defined for GRIB2_parm=5;
>> > >
>> > > 3 things...
>> > >
>> > > (1) As a crude temporary solution, you could refer to it by
it's
>> record
>> > > number. Let's assume you know that record 1 has temperature
data:
>> > >
>> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
'name="TMP";
>> > > level="R1"; set_attr_level="P500";' -v 5
>> > >
>> > > Here I'm reading record 1, naming it TMP and the using
set_attr_level
>> > write
>> > > the output level as P500 instead of R1.
>> > >
>> > > (2) You could create a custom GRIB table yourself locally, and
then
>> set
>> > the
>> > > MET_GRIB_TABLES environment variable to point MET to it.
>> > >
>> > > (3) The best solution is contacting ECMWF and requesting they
send
>> over
>> > the
>> > > GRIB2 table we should be using to decode this data.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<johnhg at ucar.edu>
>> > wrote:
>> > >
>> > > > We might already have them. I'm not sure.
>> > > >
>> > > > But there's an additional required piece of the puzzle.
Binbin runs
>> > > > "wgrib2" to figure out what "name" he should use in the MET
config
>> > files.
>> > > > If wgrib2 doesn't tell him the right "name", the problem will
>> remain.
>> > > > wgrib2 can be configured to define additional tables as well,
but
>> for
>> > > this
>> > > > to be easy for Binbin, that'd mean updating the script that's
run
>> for
>> > > > "module load grib_utils" on WCOSS. And I'm guessing that's
not very
>> > easy.
>> > > >
>> > > > Yes, I do agree that the best solution is getting the ECMWF
GRIB
>> tables
>> > > > and adding them for both wgrib2 and MET.
>> > > >
>> > > > John
>> > > >
>> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <jensen at ucar.edu>
wrote:
>> > > >
>> > > >> John,
>> > > >>
>> > > >> It seems to me that it would be worthwhile to just ask ECMWF
for
>> their
>> > > >> Grib2 Table for their Ensemble files.  And other operational
>> centers
>> > for
>> > > >> that matter.  Do you agree this would be a good solution?
>> > > >>
>> > > >> Thanks, Tara
>> > > >>
>> > > >> ---------- Forwarded message ---------
>> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
>> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
>> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for
reading
>> > ECMWF
>> > > >> ensmeble data
>> > > >> To:
>> > > >>
>> > > >>
>> > > >>
>> > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > > >>
>> > > >> John,
>> > > >>
>> > > >>   I tested using following command:
>> > > >>
>> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > > >> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v3
>> > > >>
>> > > >> ERROR  :
>> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
>> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
>> information
>> > > >>
>> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > > >>
>> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > >> -v
>> > > >> 3
>> > > >>
>> > > >> Still got error
>> > > >> ERROR  :
>> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
>> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
>> information
>> > > >>
>> > > >> Looks like discipline=255 is not recognized
>> > > >>
>> > > >> Binbin
>> > > >>
>> > > >>
>> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT <
>> > > >> met_help at ucar.edu>
>> > > >> wrote:
>> > > >>
>> > > >> > Binbin,
>> > > >> > You can do something similar with MET. Please go to this
link:
>> > > >> >
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>> > > >> >
>> > > >> > And then search for "GRIB2_mtab", and you'll find the
content
>> > > >> > copied/pasted below. You can specify these INSTEAD OF
specifying
>> a
>> > > >> > requested name.
>> > > >> >
>> > > >> > //       - The "GRIB2_mtab" is an integer specifying the
master
>> > table
>> > > >> > number.
>> > > >> > //       - The "GRIB2_ltab" is an integer specifying the
local
>> table
>> > > >> > number.
>> > > >> > //       - The "GRIB2_disc" is an integer specifying the
GRIB2
>> > > >> discipline
>> > > >> > code.
>> > > >> > //       - The "GRIB2_parm_cat" is an integer specifying
the
>> > parameter
>> > > >> > category
>> > > >> > //         code.
>> > > >> > //       - The "GRIB2_parm" is an integer specifying the
>> parameter
>> > > code.
>> > > >> > //       - The "GRIB2_pdt" is an integer specifying the
product
>> > > >> definition
>> > > >> > //         template (Table 4.0).
>> > > >> > //       - The "GRIB2_process" is an integer specifying
the
>> > generating
>> > > >> > process
>> > > >> > //         (Table 4.3).
>> > > >> > //       - The "GRIB2_cntr" is an integer specifying the
>> originating
>> > > >> > center.
>> > > >> > //       - The "GRIB2_ens_type" is an integer specifying
the
>> > ensemble
>> > > >> type
>> > > >> > //         (Table 4.6).
>> > > >> > //       - The "GRIB2_der_type" is an integer specifying
the
>> derived
>> > > >> > product
>> > > >> > //         type (Table 4.7).
>> > > >> > //       - The "GRIB2_stat_type" is an integer specifying
the
>> > > >> statistical
>> > > >> > //         processing type (Table 4.10).
>> > > >> > //       - The "GRIB2_ipdtmpl_index" and
"GRIB2_ipdtmpl_val"
>> entries
>> > > are
>> > > >> > arrays
>> > > >> > //         of integers which specify the product
description
>> > template
>> > > >> > values to
>> > > >> > //         be used. The indices are 0-based. For example,
use the
>> > > >> > following to
>> > > >> > //         request a GRIB2 record whose 9-th and 27-th
product
>> > > >> description
>> > > >> > //         template values are 1 and 2, respectively:
>> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
GRIB2_ipdtmpl_val=[1,
>> 2];
>> > > >> >
>> > > >> > While I don't have any ECMWF data on hand for testing, I
do have
>> a
>> > > file
>> > > >> > from the Air Force from which wgrib2 prints a similar
result:
>> > > >> >
>> > > >> > 549:687031550:d=2017061000:var discipline=0 center=57
>> local_table=1
>> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
>> > > >> >
>> > > >> > So I used that for testing. I ran the following
plot_data_plane
>> > > command
>> > > >> > WITHOUT specifying a requested name:
>> > > >> >
>> > > >> >
>> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
>> > 'GRIB2_disc=0;
>> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
GRIB2_parm=192;
>> > > >> level="Z0";'
>> > > >> > -v 3
>> > > >> >
>> > > >> > And then I looked closely at the log message to confirm
that it
>> used
>> > > >> > record 549, as expected:
>> > > >> >
>> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact
match for
>> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
>> > > >> >
>> > > >>
>> > >
>> >
>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>> > > >> >
>> > > >> > So MET used these GRIB2 numbers I defined to do a table
lookup
>> and
>> > > find
>> > > >> a
>> > > >> > match in the MET GRIB tables:
>> > > >> >
>> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>> > > >> >
>> > > >> > And it determined that that field should be called
"CTSTM".
>> > Hopefully,
>> > > >> the
>> > > >> > ECMWF GRIB2 tables are already defined there in MET. If
not, we
>> can
>> > > >> always
>> > > >> > add them.
>> > > >> >
>> > > >> > Please try running plot_data_plane at verbosity level 3 on
one of
>> > > these
>> > > >> > files first. And make sure it runs without error and the
log
>> message
>> > > >> prints
>> > > >> > out the "name" it found for that data. If so, you can just
skip
>> > using
>> > > >> all
>> > > >> > those integers and revert back to using the name MET
knows. If
>> not,
>> > > >> then we
>> > > >> > may need to define a custom GRIB table for this ECMWF
GRIB2 data.
>> > > >> >
>> > > >> > Thanks,
>> > > >> > John
>> > > >> >
>> > > >>
>> > > >>
>> > > >> --
>> > > >>
>> > > >> Binbin Zhou
>> > > >>
>> > > >> IMSG at NOAA/NWS/NCEP/EMC
>> > > >>
>> > > >> 5830 University Research Ct.
>> > > >>
>> > > >> College Park, MD 20740
>> > > >>
>> > > >> Binbin.Zhou at noaa.gov
>> > > >>
>> > > >> 301-683-3683
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > > >> Tara Jensen
>> > > >> Project Manager II
>> > > >> NCAR RAL and DTC
>> > > >> PO Box 3000, Boulder, Colorado 80307 USA
>> > > >> +1 303-497-8479          jensen at ucar.edu
>> > > >>
>> > > >
>> > >
>> > >
>> >
>> > --
>> >
>> > Binbin Zhou
>> >
>> > IMSG at NOAA/NWS/NCEP/EMC
>> >
>> > 5830 University Research Ct.
>> >
>> > College Park, MD 20740
>> >
>> > Binbin.Zhou at noaa.gov
>> >
>> > 301-683-3683
>> >
>> >
>>
>>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>


--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Tue Dec 08 15:43:19 2020

Binbin,

I used the "-PDS10" option for wgrib to dump those records:

[John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
| egrep  "^11:|^26:|^43:"
11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0 0
0 21
2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0 0
0 21
2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0 0
0 21
2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255

Looking VERY closely, there is a difference in the 42-nd column which
is
the 4-th from the last... records 11, 26, and 43 have values of 3, 1,
and 2
respectively.

On this page
(https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html),
byte 42 = 1 means "unperturbed control forecast".

So we *should* be able to use the "GRIB_ens" config option to
differentiate
between those records. BUT when I try, it doesn't work. And that's
because
of this line:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89

We only apply those filtering options to GRIB files created by NCEP.
If
you'd like, I could try removing that condition to see if we can pick
out
the single record of interest.

John

On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   Look at my
> directory /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> The grib1 file is ensposte.t00z.z500hr, this is only for 500 HGT.
> wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
> I get 3 records:
>
>
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> mb:anl:type=analysis:NAve=0
>
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> mb:anl:type=analysis:NAve=0
>
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> mb:anl:type=analysis:NAve=0
>
> From its grib2 file, we know only  record#26 is analysis low-res
control.
> If you have some way to determine which record is analysis low-res
control
> from the grib1 file,
> then we can use grib1 directly (don't consider the size issue).
>
> Binbin
>
> On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
> binbin.zhou at noaa.gov> wrote:
>
> > John,
> >
> >   Grib1 data is too large, another issue is the low-res control
analysis
> > data can not be retrieved
> > from the grib1 ecmwf data files. If using wgrib, we can see 3
analysis
> > fields, but don't know
> > which is low-res control. After convert to grib2, wgrib2 can show
which
> > record is low-res control.
> > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
> >
> > Binbin
> >
> > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Binbin,
> >>
> >> Sounds like there's a lot of moving parts there. How would you
like to
> >> proceed?
> >>
> >> Should we check to see if MET can read those ECMWF GRIB1 files
directly
> >> without needing to reformat?
> >> Or do you just want to create a custom GRIB table for the result
of your
> >> pre-processing and conversion steps?
> >>
> >> John
> >>
> >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> >
> >> > John,
> >> >
> >> >   Our ECMWF grib2 ensemble data files are converted from its
grib1
> >> format
> >> > data.
> >> > Before conversion, the parameter names are there, but after
> conversion,
> >> the
> >> > names
> >> > are gone. It is because of unknown tables. The grib1 data is
still
> >> > original. The original
> >> > grib1 data files are very strange data format we can not read.
So we
> >> have
> >> > to use
> >> > a Frotran code to re-arrange the fields and save them as NCEP-
like
> grib1
> >> > data files.
> >> > So after several processes, some important info like tables
could be
> >> lost.
> >> >
> >> >    I'll test your first suggestion since the field record order
is
> >> fixed.
> >> > Currently we still
> >> > obtained grib1 data from ECWMF, eventually, we will get grib2
data
> >> > directly. So before
> >> > we can get original grib2 files from ECMWF, we just use this
temporal
> >> > solution.
> >> > '
> >> > Thanks!
> >> >
> >> >
> >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Binbin,
> >> > >
> >> > > OK, working on Venus.
> >> > >
> >> > > You have a typo in the command you sent.
> >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > >
> >>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> > -v
> >> > > 3
> >> > >
> >> > > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat".
After
> >> > correcting
> >> > > that I did still get an error but it's a very useful one!
> >> > >
> >> > > module use
> >> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> >> > > module load met/9.1
> >> > > module load grib_util/1.1.1
> >> > > cd
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> >> > 'GRIB2_disc=255;
> >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5; level="P500";'
> >> > >
> >> > >
> >> > >
> >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter
found
> with
> >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm (5).
Use the
> >> > > MET_GRIB_TABLES environment variable to define custom GRIB
> >> tables.ERROR
> >> > :
> >> > > *
> >> > >
> >> > > So this table is not yet defined in MET. I checked their
website:
> >> > >
> >> > >
> >> >
> >>
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> >> > >
> >> > > But I don't see an entry defined for GRIB2_parm=5;
> >> > >
> >> > > 3 things...
> >> > >
> >> > > (1) As a crude temporary solution, you could refer to it by
it's
> >> record
> >> > > number. Let's assume you know that record 1 has temperature
data:
> >> > >
> >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> 'name="TMP";
> >> > > level="R1"; set_attr_level="P500";' -v 5
> >> > >
> >> > > Here I'm reading record 1, naming it TMP and the using
> set_attr_level
> >> > write
> >> > > the output level as P500 instead of R1.
> >> > >
> >> > > (2) You could create a custom GRIB table yourself locally,
and then
> >> set
> >> > the
> >> > > MET_GRIB_TABLES environment variable to point MET to it.
> >> > >
> >> > > (3) The best solution is contacting ECMWF and requesting they
send
> >> over
> >> > the
> >> > > GRIB2 table we should be using to decode this data.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<johnhg at ucar.edu>
> >> > wrote:
> >> > >
> >> > > > We might already have them. I'm not sure.
> >> > > >
> >> > > > But there's an additional required piece of the puzzle.
Binbin
> runs
> >> > > > "wgrib2" to figure out what "name" he should use in the MET
config
> >> > files.
> >> > > > If wgrib2 doesn't tell him the right "name", the problem
will
> >> remain.
> >> > > > wgrib2 can be configured to define additional tables as
well, but
> >> for
> >> > > this
> >> > > > to be easy for Binbin, that'd mean updating the script
that's run
> >> for
> >> > > > "module load grib_utils" on WCOSS. And I'm guessing that's
not
> very
> >> > easy.
> >> > > >
> >> > > > Yes, I do agree that the best solution is getting the ECMWF
GRIB
> >> tables
> >> > > > and adding them for both wgrib2 and MET.
> >> > > >
> >> > > > John
> >> > > >
> >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<jensen at ucar.edu>
> wrote:
> >> > > >
> >> > > >> John,
> >> > > >>
> >> > > >> It seems to me that it would be worthwhile to just ask
ECMWF for
> >> their
> >> > > >> Grib2 Table for their Ensemble files.  And other
operational
> >> centers
> >> > for
> >> > > >> that matter.  Do you agree this would be a good solution?
> >> > > >>
> >> > > >> Thanks, Tara
> >> > > >>
> >> > > >> ---------- Forwarded message ---------
> >> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question for
> reading
> >> > ECMWF
> >> > > >> ensmeble data
> >> > > >> To:
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> > > >>
> >> > > >> John,
> >> > > >>
> >> > > >>   I tested using following command:
> >> > > >>
> >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > > >> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
-v3
> >> > > >>
> >> > > >> ERROR  :
> >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
> >> information
> >> > > >>
> >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > > >>
> >> >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> > > >> -v
> >> > > >> 3
> >> > > >>
> >> > > >> Still got error
> >> > > >> ERROR  :
> >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
> >> information
> >> > > >>
> >> > > >> Looks like discipline=255 is not recognized
> >> > > >>
> >> > > >> Binbin
> >> > > >>
> >> > > >>
> >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via RT
<
> >> > > >> met_help at ucar.edu>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Binbin,
> >> > > >> > You can do something similar with MET. Please go to this
link:
> >> > > >> >
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> >> > > >> >
> >> > > >> > And then search for "GRIB2_mtab", and you'll find the
content
> >> > > >> > copied/pasted below. You can specify these INSTEAD OF
> specifying
> >> a
> >> > > >> > requested name.
> >> > > >> >
> >> > > >> > //       - The "GRIB2_mtab" is an integer specifying the
master
> >> > table
> >> > > >> > number.
> >> > > >> > //       - The "GRIB2_ltab" is an integer specifying the
local
> >> table
> >> > > >> > number.
> >> > > >> > //       - The "GRIB2_disc" is an integer specifying the
GRIB2
> >> > > >> discipline
> >> > > >> > code.
> >> > > >> > //       - The "GRIB2_parm_cat" is an integer specifying
the
> >> > parameter
> >> > > >> > category
> >> > > >> > //         code.
> >> > > >> > //       - The "GRIB2_parm" is an integer specifying the
> >> parameter
> >> > > code.
> >> > > >> > //       - The "GRIB2_pdt" is an integer specifying the
product
> >> > > >> definition
> >> > > >> > //         template (Table 4.0).
> >> > > >> > //       - The "GRIB2_process" is an integer specifying
the
> >> > generating
> >> > > >> > process
> >> > > >> > //         (Table 4.3).
> >> > > >> > //       - The "GRIB2_cntr" is an integer specifying the
> >> originating
> >> > > >> > center.
> >> > > >> > //       - The "GRIB2_ens_type" is an integer specifying
the
> >> > ensemble
> >> > > >> type
> >> > > >> > //         (Table 4.6).
> >> > > >> > //       - The "GRIB2_der_type" is an integer specifying
the
> >> derived
> >> > > >> > product
> >> > > >> > //         type (Table 4.7).
> >> > > >> > //       - The "GRIB2_stat_type" is an integer
specifying the
> >> > > >> statistical
> >> > > >> > //         processing type (Table 4.10).
> >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
"GRIB2_ipdtmpl_val"
> >> entries
> >> > > are
> >> > > >> > arrays
> >> > > >> > //         of integers which specify the product
description
> >> > template
> >> > > >> > values to
> >> > > >> > //         be used. The indices are 0-based. For
example, use
> the
> >> > > >> > following to
> >> > > >> > //         request a GRIB2 record whose 9-th and 27-th
product
> >> > > >> description
> >> > > >> > //         template values are 1 and 2, respectively:
> >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> GRIB2_ipdtmpl_val=[1,
> >> 2];
> >> > > >> >
> >> > > >> > While I don't have any ECMWF data on hand for testing, I
do
> have
> >> a
> >> > > file
> >> > > >> > from the Air Force from which wgrib2 prints a similar
result:
> >> > > >> >
> >> > > >> > 549:687031550:d=2017061000:var discipline=0 center=57
> >> local_table=1
> >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> >> > > >> >
> >> > > >> > So I used that for testing. I ran the following
plot_data_plane
> >> > > command
> >> > > >> > WITHOUT specifying a requested name:
> >> > > >> >
> >> > > >> >
> >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
> >> > 'GRIB2_disc=0;
> >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
GRIB2_parm=192;
> >> > > >> level="Z0";'
> >> > > >> > -v 3
> >> > > >> >
> >> > > >> > And then I looked closely at the log message to confirm
that it
> >> used
> >> > > >> > record 549, as expected:
> >> > > >> >
> >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact
match
> for
> >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> >> > > >> >
> >> > > >> > So MET used these GRIB2 numbers I defined to do a table
lookup
> >> and
> >> > > find
> >> > > >> a
> >> > > >> > match in the MET GRIB tables:
> >> > > >> >
> >>
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> >> > > >> >
> >> > > >> > And it determined that that field should be called
"CTSTM".
> >> > Hopefully,
> >> > > >> the
> >> > > >> > ECMWF GRIB2 tables are already defined there in MET. If
not, we
> >> can
> >> > > >> always
> >> > > >> > add them.
> >> > > >> >
> >> > > >> > Please try running plot_data_plane at verbosity level 3
on one
> of
> >> > > these
> >> > > >> > files first. And make sure it runs without error and the
log
> >> message
> >> > > >> prints
> >> > > >> > out the "name" it found for that data. If so, you can
just skip
> >> > using
> >> > > >> all
> >> > > >> > those integers and revert back to using the name MET
knows. If
> >> not,
> >> > > >> then we
> >> > > >> > may need to define a custom GRIB table for this ECMWF
GRIB2
> data.
> >> > > >> >
> >> > > >> > Thanks,
> >> > > >> > John
> >> > > >> >
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >>
> >> > > >> Binbin Zhou
> >> > > >>
> >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> >> > > >>
> >> > > >> 5830 University Research Ct.
> >> > > >>
> >> > > >> College Park, MD 20740
> >> > > >>
> >> > > >> Binbin.Zhou at noaa.gov
> >> > > >>
> >> > > >> 301-683-3683
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> > > >> Tara Jensen
> >> > > >> Project Manager II
> >> > > >> NCAR RAL and DTC
> >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> >> > > >> +1 303-497-8479          jensen at ucar.edu
> >> > > >>
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> >
> >> > Binbin Zhou
> >> >
> >> > IMSG at NOAA/NWS/NCEP/EMC
> >> >
> >> > 5830 University Research Ct.
> >> >
> >> > College Park, MD 20740
> >> >
> >> > Binbin.Zhou at noaa.gov
> >> >
> >> > 301-683-3683
> >> >
> >> >
> >>
> >>
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Tue Dec 08 16:46:20 2020

John,

  Yes, please try to use this approach.

Thanks!

Binbin

Binbin

On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
>
> I used the "-PDS10" option for wgrib to dump those records:
>
> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> | egrep  "^11:|^26:|^43:"
> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
>
> Looking VERY closely, there is a difference in the 42-nd column
which is
> the 4-th from the last... records 11, 26, and 43 have values of 3,
1, and 2
> respectively.
>
> On this page
(https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> ),
> byte 42 = 1 means "unperturbed control forecast".
>
> So we *should* be able to use the "GRIB_ens" config option to
differentiate
> between those records. BUT when I try, it doesn't work. And that's
because
> of this line:
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
>
> We only apply those filtering options to GRIB files created by NCEP.
If
> you'd like, I could try removing that condition to see if we can
pick out
> the single record of interest.
>
> John
>
> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > John,
> >
> >   Look at my
> > directory /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > The grib1 file is ensposte.t00z.z500hr, this is only for 500 HGT.
> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
> > I get 3 records:
> >
> >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > mb:anl:type=analysis:NAve=0
> >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > mb:anl:type=analysis:NAve=0
> >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > mb:anl:type=analysis:NAve=0
> >
> > From its grib2 file, we know only  record#26 is analysis low-res
control.
> > If you have some way to determine which record is analysis low-res
> control
> > from the grib1 file,
> > then we can use grib1 directly (don't consider the size issue).
> >
> > Binbin
> >
> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
> > binbin.zhou at noaa.gov> wrote:
> >
> > > John,
> > >
> > >   Grib1 data is too large, another issue is the low-res control
> analysis
> > > data can not be retrieved
> > > from the grib1 ecmwf data files. If using wgrib, we can see 3
analysis
> > > fields, but don't know
> > > which is low-res control. After convert to grib2, wgrib2 can
show which
> > > record is low-res control.
> > > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
> > >
> > > Binbin
> > >
> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Binbin,
> > >>
> > >> Sounds like there's a lot of moving parts there. How would you
like to
> > >> proceed?
> > >>
> > >> Should we check to see if MET can read those ECMWF GRIB1 files
> directly
> > >> without needing to reformat?
> > >> Or do you just want to create a custom GRIB table for the
result of
> your
> > >> pre-processing and conversion steps?
> > >>
> > >> John
> > >>
> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > >> >
> > >> > John,
> > >> >
> > >> >   Our ECMWF grib2 ensemble data files are converted from its
grib1
> > >> format
> > >> > data.
> > >> > Before conversion, the parameter names are there, but after
> > conversion,
> > >> the
> > >> > names
> > >> > are gone. It is because of unknown tables. The grib1 data is
still
> > >> > original. The original
> > >> > grib1 data files are very strange data format we can not
read. So we
> > >> have
> > >> > to use
> > >> > a Frotran code to re-arrange the fields and save them as
NCEP-like
> > grib1
> > >> > data files.
> > >> > So after several processes, some important info like tables
could be
> > >> lost.
> > >> >
> > >> >    I'll test your first suggestion since the field record
order is
> > >> fixed.
> > >> > Currently we still
> > >> > obtained grib1 data from ECWMF, eventually, we will get grib2
data
> > >> > directly. So before
> > >> > we can get original grib2 files from ECMWF, we just use this
> temporal
> > >> > solution.
> > >> > '
> > >> > Thanks!
> > >> >
> > >> >
> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
> > >> > met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Binbin,
> > >> > >
> > >> > > OK, working on Venus.
> > >> > >
> > >> > > You have a typo in the command you sent.
> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >> > >
> > >>
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> > -v
> > >> > > 3
> > >> > >
> > >> > > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat".
After
> > >> > correcting
> > >> > > that I did still get an error but it's a very useful one!
> > >> > >
> > >> > > module use
> > >> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > >> > > module load met/9.1
> > >> > > module load grib_util/1.1.1
> > >> > > cd
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> > >> > 'GRIB2_disc=255;
> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
level="P500";'
> > >> > >
> > >> > >
> > >> > >
> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no parameter
found
> > with
> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm
(5). Use
> the
> > >> > > MET_GRIB_TABLES environment variable to define custom GRIB
> > >> tables.ERROR
> > >> > :
> > >> > > *
> > >> > >
> > >> > > So this table is not yet defined in MET. I checked their
website:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > >> > >
> > >> > > But I don't see an entry defined for GRIB2_parm=5;
> > >> > >
> > >> > > 3 things...
> > >> > >
> > >> > > (1) As a crude temporary solution, you could refer to it by
it's
> > >> record
> > >> > > number. Let's assume you know that record 1 has temperature
data:
> > >> > >
> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> > 'name="TMP";
> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > >> > >
> > >> > > Here I'm reading record 1, naming it TMP and the using
> > set_attr_level
> > >> > write
> > >> > > the output level as P500 instead of R1.
> > >> > >
> > >> > > (2) You could create a custom GRIB table yourself locally,
and
> then
> > >> set
> > >> > the
> > >> > > MET_GRIB_TABLES environment variable to point MET to it.
> > >> > >
> > >> > > (3) The best solution is contacting ECMWF and requesting
they send
> > >> over
> > >> > the
> > >> > > GRIB2 table we should be using to decode this data.
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
> johnhg at ucar.edu>
> > >> > wrote:
> > >> > >
> > >> > > > We might already have them. I'm not sure.
> > >> > > >
> > >> > > > But there's an additional required piece of the puzzle.
Binbin
> > runs
> > >> > > > "wgrib2" to figure out what "name" he should use in the
MET
> config
> > >> > files.
> > >> > > > If wgrib2 doesn't tell him the right "name", the problem
will
> > >> remain.
> > >> > > > wgrib2 can be configured to define additional tables as
well,
> but
> > >> for
> > >> > > this
> > >> > > > to be easy for Binbin, that'd mean updating the script
that's
> run
> > >> for
> > >> > > > "module load grib_utils" on WCOSS. And I'm guessing
that's not
> > very
> > >> > easy.
> > >> > > >
> > >> > > > Yes, I do agree that the best solution is getting the
ECMWF GRIB
> > >> tables
> > >> > > > and adding them for both wgrib2 and MET.
> > >> > > >
> > >> > > > John
> > >> > > >
> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<jensen at ucar.edu>
> > wrote:
> > >> > > >
> > >> > > >> John,
> > >> > > >>
> > >> > > >> It seems to me that it would be worthwhile to just ask
ECMWF
> for
> > >> their
> > >> > > >> Grib2 Table for their Ensemble files.  And other
operational
> > >> centers
> > >> > for
> > >> > > >> that matter.  Do you agree this would be a good
solution?
> > >> > > >>
> > >> > > >> Thanks, Tara
> > >> > > >>
> > >> > > >> ---------- Forwarded message ---------
> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question
for
> > reading
> > >> > ECMWF
> > >> > > >> ensmeble data
> > >> > > >> To:
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> > >> > > >>
> > >> > > >> John,
> > >> > > >>
> > >> > > >>   I tested using following command:
> > >> > > >>
> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >> > > >>
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'  -v3
> > >> > > >>
> > >> > > >> ERROR  :
> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
> > >> information
> > >> > > >>
> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >> > > >>
> > >> >
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> > > >> -v
> > >> > > >> 3
> > >> > > >>
> > >> > > >> Still got error
> > >> > > >> ERROR  :
> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
GRIB2_disc,
> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in field
> > >> information
> > >> > > >>
> > >> > > >> Looks like discipline=255 is not recognized
> > >> > > >>
> > >> > > >> Binbin
> > >> > > >>
> > >> > > >>
> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via
RT <
> > >> > > >> met_help at ucar.edu>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > Binbin,
> > >> > > >> > You can do something similar with MET. Please go to
this
> link:
> > >> > > >> >
> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > >> > > >> >
> > >> > > >> > And then search for "GRIB2_mtab", and you'll find the
content
> > >> > > >> > copied/pasted below. You can specify these INSTEAD OF
> > specifying
> > >> a
> > >> > > >> > requested name.
> > >> > > >> >
> > >> > > >> > //       - The "GRIB2_mtab" is an integer specifying
the
> master
> > >> > table
> > >> > > >> > number.
> > >> > > >> > //       - The "GRIB2_ltab" is an integer specifying
the
> local
> > >> table
> > >> > > >> > number.
> > >> > > >> > //       - The "GRIB2_disc" is an integer specifying
the
> GRIB2
> > >> > > >> discipline
> > >> > > >> > code.
> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
specifying the
> > >> > parameter
> > >> > > >> > category
> > >> > > >> > //         code.
> > >> > > >> > //       - The "GRIB2_parm" is an integer specifying
the
> > >> parameter
> > >> > > code.
> > >> > > >> > //       - The "GRIB2_pdt" is an integer specifying
the
> product
> > >> > > >> definition
> > >> > > >> > //         template (Table 4.0).
> > >> > > >> > //       - The "GRIB2_process" is an integer
specifying the
> > >> > generating
> > >> > > >> > process
> > >> > > >> > //         (Table 4.3).
> > >> > > >> > //       - The "GRIB2_cntr" is an integer specifying
the
> > >> originating
> > >> > > >> > center.
> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
specifying the
> > >> > ensemble
> > >> > > >> type
> > >> > > >> > //         (Table 4.6).
> > >> > > >> > //       - The "GRIB2_der_type" is an integer
specifying the
> > >> derived
> > >> > > >> > product
> > >> > > >> > //         type (Table 4.7).
> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
specifying the
> > >> > > >> statistical
> > >> > > >> > //         processing type (Table 4.10).
> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
"GRIB2_ipdtmpl_val"
> > >> entries
> > >> > > are
> > >> > > >> > arrays
> > >> > > >> > //         of integers which specify the product
description
> > >> > template
> > >> > > >> > values to
> > >> > > >> > //         be used. The indices are 0-based. For
example, use
> > the
> > >> > > >> > following to
> > >> > > >> > //         request a GRIB2 record whose 9-th and 27-th
> product
> > >> > > >> description
> > >> > > >> > //         template values are 1 and 2, respectively:
> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> > GRIB2_ipdtmpl_val=[1,
> > >> 2];
> > >> > > >> >
> > >> > > >> > While I don't have any ECMWF data on hand for testing,
I do
> > have
> > >> a
> > >> > > file
> > >> > > >> > from the Air Force from which wgrib2 prints a similar
result:
> > >> > > >> >
> > >> > > >> > 549:687031550:d=2017061000:var discipline=0 center=57
> > >> local_table=1
> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > >> > > >> >
> > >> > > >> > So I used that for testing. I ran the following
> plot_data_plane
> > >> > > command
> > >> > > >> > WITHOUT specifying a requested name:
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
> > >> > 'GRIB2_disc=0;
> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
> GRIB2_parm=192;
> > >> > > >> level="Z0";'
> > >> > > >> > -v 3
> > >> > > >> >
> > >> > > >> > And then I looked closely at the log message to
confirm that
> it
> > >> used
> > >> > > >> > record 549, as expected:
> > >> > > >> >
> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact
match
> > for
> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > >> > > >> >
> > >> > > >> > So MET used these GRIB2 numbers I defined to do a
table
> lookup
> > >> and
> > >> > > find
> > >> > > >> a
> > >> > > >> > match in the MET GRIB tables:
> > >> > > >> >
> > >>
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > >> > > >> >
> > >> > > >> > And it determined that that field should be called
"CTSTM".
> > >> > Hopefully,
> > >> > > >> the
> > >> > > >> > ECMWF GRIB2 tables are already defined there in MET.
If not,
> we
> > >> can
> > >> > > >> always
> > >> > > >> > add them.
> > >> > > >> >
> > >> > > >> > Please try running plot_data_plane at verbosity level
3 on
> one
> > of
> > >> > > these
> > >> > > >> > files first. And make sure it runs without error and
the log
> > >> message
> > >> > > >> prints
> > >> > > >> > out the "name" it found for that data. If so, you can
just
> skip
> > >> > using
> > >> > > >> all
> > >> > > >> > those integers and revert back to using the name MET
knows.
> If
> > >> not,
> > >> > > >> then we
> > >> > > >> > may need to define a custom GRIB table for this ECMWF
GRIB2
> > data.
> > >> > > >> >
> > >> > > >> > Thanks,
> > >> > > >> > John
> > >> > > >> >
> > >> > > >>
> > >> > > >>
> > >> > > >> --
> > >> > > >>
> > >> > > >> Binbin Zhou
> > >> > > >>
> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > >> > > >>
> > >> > > >> 5830 University Research Ct.
> > >> > > >>
> > >> > > >> College Park, MD 20740
> > >> > > >>
> > >> > > >> Binbin.Zhou at noaa.gov
> > >> > > >>
> > >> > > >> 301-683-3683
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> --
> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >> > > >> Tara Jensen
> > >> > > >> Project Manager II
> > >> > > >> NCAR RAL and DTC
> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > >> > > >>
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> >
> > >> > Binbin Zhou
> > >> >
> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >> >
> > >> > 5830 University Research Ct.
> > >> >
> > >> > College Park, MD 20740
> > >> >
> > >> > Binbin.Zhou at noaa.gov
> > >> >
> > >> > 301-683-3683
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > >
> > > Binbin Zhou
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct.
> > >
> > > College Park, MD 20740
> > >
> > > Binbin.Zhou at noaa.gov
> > >
> > > 301-683-3683
> > >
> >
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Tue Dec 08 17:27:23 2020

John,

  If this removing impacts other applications, or takes some time
to be effective, then won't try.  I will use another approach, such as
borrow GEFS's grib2 data structure as tank to save EC ensemble
data.

Binbin


On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
binbin.zhou at noaa.gov> wrote:

> John,
>
>   Yes, please try to use this approach.
>
> Thanks!
>
> Binbin
>
> Binbin
>
> On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binbin,
>>
>> I used the "-PDS10" option for wgrib to dump those records:
>>
>> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
>> | egrep  "^11:|^26:|^43:"
>> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
>> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
>> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
>> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
>> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1 0
0 0 21
>> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
>>
>> Looking VERY closely, there is a difference in the 42-nd column
which is
>> the 4-th from the last... records 11, 26, and 43 have values of 3,
1, and
>> 2
>> respectively.
>>
>> On this page
(https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
>> ),
>> byte 42 = 1 means "unperturbed control forecast".
>>
>> So we *should* be able to use the "GRIB_ens" config option to
>> differentiate
>> between those records. BUT when I try, it doesn't work. And that's
because
>> of this line:
>>
>>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
>>
>> We only apply those filtering options to GRIB files created by
NCEP. If
>> you'd like, I could try removing that condition to see if we can
pick out
>> the single record of interest.
>>
>> John
>>
>> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> >
>> > John,
>> >
>> >   Look at my
>> > directory
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
>> > The grib1 file is ensposte.t00z.z500hr, this is only for 500 HGT.
>> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
>> > I get 3 records:
>> >
>> >
>> >
>>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > mb:anl:type=analysis:NAve=0
>> >
>> >
>>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > mb:anl:type=analysis:NAve=0
>> >
>> >
>>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > mb:anl:type=analysis:NAve=0
>> >
>> > From its grib2 file, we know only  record#26 is analysis low-res
>> control.
>> > If you have some way to determine which record is analysis low-
res
>> control
>> > from the grib1 file,
>> > then we can use grib1 directly (don't consider the size issue).
>> >
>> > Binbin
>> >
>> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
>> > binbin.zhou at noaa.gov> wrote:
>> >
>> > > John,
>> > >
>> > >   Grib1 data is too large, another issue is the low-res control
>> analysis
>> > > data can not be retrieved
>> > > from the grib1 ecmwf data files. If using wgrib, we can see 3
analysis
>> > > fields, but don't know
>> > > which is low-res control. After convert to grib2, wgrib2 can
show
>> which
>> > > record is low-res control.
>> > > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
>> > >
>> > > Binbin
>> > >
>> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >> Binbin,
>> > >>
>> > >> Sounds like there's a lot of moving parts there. How would you
like
>> to
>> > >> proceed?
>> > >>
>> > >> Should we check to see if MET can read those ECMWF GRIB1 files
>> directly
>> > >> without needing to reformat?
>> > >> Or do you just want to create a custom GRIB table for the
result of
>> your
>> > >> pre-processing and conversion steps?
>> > >>
>> > >> John
>> > >>
>> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
>> > >> met_help at ucar.edu> wrote:
>> > >>
>> > >> >
>> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > >> >
>> > >> > John,
>> > >> >
>> > >> >   Our ECMWF grib2 ensemble data files are converted from its
grib1
>> > >> format
>> > >> > data.
>> > >> > Before conversion, the parameter names are there, but after
>> > conversion,
>> > >> the
>> > >> > names
>> > >> > are gone. It is because of unknown tables. The grib1 data is
still
>> > >> > original. The original
>> > >> > grib1 data files are very strange data format we can not
read. So
>> we
>> > >> have
>> > >> > to use
>> > >> > a Frotran code to re-arrange the fields and save them as
NCEP-like
>> > grib1
>> > >> > data files.
>> > >> > So after several processes, some important info like tables
could
>> be
>> > >> lost.
>> > >> >
>> > >> >    I'll test your first suggestion since the field record
order is
>> > >> fixed.
>> > >> > Currently we still
>> > >> > obtained grib1 data from ECWMF, eventually, we will get
grib2 data
>> > >> > directly. So before
>> > >> > we can get original grib2 files from ECMWF, we just use this
>> temporal
>> > >> > solution.
>> > >> > '
>> > >> > Thanks!
>> > >> >
>> > >> >
>> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
>> > >> > met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Binbin,
>> > >> > >
>> > >> > > OK, working on Venus.
>> > >> > >
>> > >> > > You have a typo in the command you sent.
>> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > >> > >
>> > >>
>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > >> > -v
>> > >> > > 3
>> > >> > >
>> > >> > > You have "RIB2_parm_cat" that should be "GRIB2_parm_cat".
After
>> > >> > correcting
>> > >> > > that I did still get an error but it's a very useful one!
>> > >> > >
>> > >> > > module use
>> > >> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>> > >> > > module load met/9.1
>> > >> > > module load grib_util/1.1.1
>> > >> > > cd
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
>> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
>> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
>> > >> > 'GRIB2_disc=255;
>> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
level="P500";'
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
parameter found
>> > with
>> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm
(5). Use
>> the
>> > >> > > MET_GRIB_TABLES environment variable to define custom GRIB
>> > >> tables.ERROR
>> > >> > :
>> > >> > > *
>> > >> > >
>> > >> > > So this table is not yet defined in MET. I checked their
website:
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>> > >> > >
>> > >> > > But I don't see an entry defined for GRIB2_parm=5;
>> > >> > >
>> > >> > > 3 things...
>> > >> > >
>> > >> > > (1) As a crude temporary solution, you could refer to it
by it's
>> > >> record
>> > >> > > number. Let's assume you know that record 1 has
temperature data:
>> > >> > >
>> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
>> > 'name="TMP";
>> > >> > > level="R1"; set_attr_level="P500";' -v 5
>> > >> > >
>> > >> > > Here I'm reading record 1, naming it TMP and the using
>> > set_attr_level
>> > >> > write
>> > >> > > the output level as P500 instead of R1.
>> > >> > >
>> > >> > > (2) You could create a custom GRIB table yourself locally,
and
>> then
>> > >> set
>> > >> > the
>> > >> > > MET_GRIB_TABLES environment variable to point MET to it.
>> > >> > >
>> > >> > > (3) The best solution is contacting ECMWF and requesting
they
>> send
>> > >> over
>> > >> > the
>> > >> > > GRIB2 table we should be using to decode this data.
>> > >> > >
>> > >> > > Thanks,
>> > >> > > John
>> > >> > >
>> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
>> johnhg at ucar.edu>
>> > >> > wrote:
>> > >> > >
>> > >> > > > We might already have them. I'm not sure.
>> > >> > > >
>> > >> > > > But there's an additional required piece of the puzzle.
Binbin
>> > runs
>> > >> > > > "wgrib2" to figure out what "name" he should use in the
MET
>> config
>> > >> > files.
>> > >> > > > If wgrib2 doesn't tell him the right "name", the problem
will
>> > >> remain.
>> > >> > > > wgrib2 can be configured to define additional tables as
well,
>> but
>> > >> for
>> > >> > > this
>> > >> > > > to be easy for Binbin, that'd mean updating the script
that's
>> run
>> > >> for
>> > >> > > > "module load grib_utils" on WCOSS. And I'm guessing
that's not
>> > very
>> > >> > easy.
>> > >> > > >
>> > >> > > > Yes, I do agree that the best solution is getting the
ECMWF
>> GRIB
>> > >> tables
>> > >> > > > and adding them for both wgrib2 and MET.
>> > >> > > >
>> > >> > > > John
>> > >> > > >
>> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<jensen at ucar.edu>
>> > wrote:
>> > >> > > >
>> > >> > > >> John,
>> > >> > > >>
>> > >> > > >> It seems to me that it would be worthwhile to just ask
ECMWF
>> for
>> > >> their
>> > >> > > >> Grib2 Table for their Ensemble files.  And other
operational
>> > >> centers
>> > >> > for
>> > >> > > >> that matter.  Do you agree this would be a good
solution?
>> > >> > > >>
>> > >> > > >> Thanks, Tara
>> > >> > > >>
>> > >> > > >> ---------- Forwarded message ---------
>> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
>> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
>> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question
for
>> > reading
>> > >> > ECMWF
>> > >> > > >> ensmeble data
>> > >> > > >> To:
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> >
>> > >> > > >>
>> > >> > > >> John,
>> > >> > > >>
>> > >> > > >>   I tested using following command:
>> > >> > > >>
>> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > >> > > >>
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> -v3
>> > >> > > >>
>> > >> > > >> ERROR  :
>> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
>> GRIB2_disc,
>> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
>> > >> information
>> > >> > > >>
>> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
>> > >> > > >>
>> > >> >
>> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > >> > > >> -v
>> > >> > > >> 3
>> > >> > > >>
>> > >> > > >> Still got error
>> > >> > > >> ERROR  :
>> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
>> GRIB2_disc,
>> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
>> > >> information
>> > >> > > >>
>> > >> > > >> Looks like discipline=255 is not recognized
>> > >> > > >>
>> > >> > > >> Binbin
>> > >> > > >>
>> > >> > > >>
>> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway via
RT <
>> > >> > > >> met_help at ucar.edu>
>> > >> > > >> wrote:
>> > >> > > >>
>> > >> > > >> > Binbin,
>> > >> > > >> > You can do something similar with MET. Please go to
this
>> link:
>> > >> > > >> >
>> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>> > >> > > >> >
>> > >> > > >> > And then search for "GRIB2_mtab", and you'll find the
>> content
>> > >> > > >> > copied/pasted below. You can specify these INSTEAD OF
>> > specifying
>> > >> a
>> > >> > > >> > requested name.
>> > >> > > >> >
>> > >> > > >> > //       - The "GRIB2_mtab" is an integer specifying
the
>> master
>> > >> > table
>> > >> > > >> > number.
>> > >> > > >> > //       - The "GRIB2_ltab" is an integer specifying
the
>> local
>> > >> table
>> > >> > > >> > number.
>> > >> > > >> > //       - The "GRIB2_disc" is an integer specifying
the
>> GRIB2
>> > >> > > >> discipline
>> > >> > > >> > code.
>> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
specifying the
>> > >> > parameter
>> > >> > > >> > category
>> > >> > > >> > //         code.
>> > >> > > >> > //       - The "GRIB2_parm" is an integer specifying
the
>> > >> parameter
>> > >> > > code.
>> > >> > > >> > //       - The "GRIB2_pdt" is an integer specifying
the
>> product
>> > >> > > >> definition
>> > >> > > >> > //         template (Table 4.0).
>> > >> > > >> > //       - The "GRIB2_process" is an integer
specifying the
>> > >> > generating
>> > >> > > >> > process
>> > >> > > >> > //         (Table 4.3).
>> > >> > > >> > //       - The "GRIB2_cntr" is an integer specifying
the
>> > >> originating
>> > >> > > >> > center.
>> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
specifying the
>> > >> > ensemble
>> > >> > > >> type
>> > >> > > >> > //         (Table 4.6).
>> > >> > > >> > //       - The "GRIB2_der_type" is an integer
specifying the
>> > >> derived
>> > >> > > >> > product
>> > >> > > >> > //         type (Table 4.7).
>> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
specifying
>> the
>> > >> > > >> statistical
>> > >> > > >> > //         processing type (Table 4.10).
>> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
"GRIB2_ipdtmpl_val"
>> > >> entries
>> > >> > > are
>> > >> > > >> > arrays
>> > >> > > >> > //         of integers which specify the product
description
>> > >> > template
>> > >> > > >> > values to
>> > >> > > >> > //         be used. The indices are 0-based. For
example,
>> use
>> > the
>> > >> > > >> > following to
>> > >> > > >> > //         request a GRIB2 record whose 9-th and 27-
th
>> product
>> > >> > > >> description
>> > >> > > >> > //         template values are 1 and 2, respectively:
>> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
>> > GRIB2_ipdtmpl_val=[1,
>> > >> 2];
>> > >> > > >> >
>> > >> > > >> > While I don't have any ECMWF data on hand for
testing, I do
>> > have
>> > >> a
>> > >> > > file
>> > >> > > >> > from the Air Force from which wgrib2 prints a similar
>> result:
>> > >> > > >> >
>> > >> > > >> > 549:687031550:d=2017061000:var discipline=0 center=57
>> > >> local_table=1
>> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
>> > >> > > >> >
>> > >> > > >> > So I used that for testing. I ran the following
>> plot_data_plane
>> > >> > > command
>> > >> > > >> > WITHOUT specifying a requested name:
>> > >> > > >> >
>> > >> > > >> >
>> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2 plot.ps
>> > >> > 'GRIB2_disc=0;
>> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
>> GRIB2_parm=192;
>> > >> > > >> level="Z0";'
>> > >> > > >> > -v 3
>> > >> > > >> >
>> > >> > > >> > And then I looked closely at the log message to
confirm
>> that it
>> > >> used
>> > >> > > >> > record 549, as expected:
>> > >> > > >> >
>> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found
exact match
>> > for
>> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2 file
>> > >> > > >> >
>> > >> > > >>
>> > >> > >
>> > >> >
>> > >>
>> >
>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>> > >> > > >> >
>> > >> > > >> > So MET used these GRIB2 numbers I defined to do a
table
>> lookup
>> > >> and
>> > >> > > find
>> > >> > > >> a
>> > >> > > >> > match in the MET GRIB tables:
>> > >> > > >> >
>> > >>
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>> > >> > > >> >
>> > >> > > >> > And it determined that that field should be called
"CTSTM".
>> > >> > Hopefully,
>> > >> > > >> the
>> > >> > > >> > ECMWF GRIB2 tables are already defined there in MET.
If
>> not, we
>> > >> can
>> > >> > > >> always
>> > >> > > >> > add them.
>> > >> > > >> >
>> > >> > > >> > Please try running plot_data_plane at verbosity level
3 on
>> one
>> > of
>> > >> > > these
>> > >> > > >> > files first. And make sure it runs without error and
the log
>> > >> message
>> > >> > > >> prints
>> > >> > > >> > out the "name" it found for that data. If so, you can
just
>> skip
>> > >> > using
>> > >> > > >> all
>> > >> > > >> > those integers and revert back to using the name MET
knows.
>> If
>> > >> not,
>> > >> > > >> then we
>> > >> > > >> > may need to define a custom GRIB table for this ECMWF
GRIB2
>> > data.
>> > >> > > >> >
>> > >> > > >> > Thanks,
>> > >> > > >> > John
>> > >> > > >> >
>> > >> > > >>
>> > >> > > >>
>> > >> > > >> --
>> > >> > > >>
>> > >> > > >> Binbin Zhou
>> > >> > > >>
>> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
>> > >> > > >>
>> > >> > > >> 5830 University Research Ct.
>> > >> > > >>
>> > >> > > >> College Park, MD 20740
>> > >> > > >>
>> > >> > > >> Binbin.Zhou at noaa.gov
>> > >> > > >>
>> > >> > > >> 301-683-3683
>> > >> > > >>
>> > >> > > >>
>> > >> > > >>
>> > >> > > >> --
>> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > >> > > >> Tara Jensen
>> > >> > > >> Project Manager II
>> > >> > > >> NCAR RAL and DTC
>> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
>> > >> > > >> +1 303-497-8479          jensen at ucar.edu
>> > >> > > >>
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >> > --
>> > >> >
>> > >> > Binbin Zhou
>> > >> >
>> > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > >> >
>> > >> > 5830 University Research Ct.
>> > >> >
>> > >> > College Park, MD 20740
>> > >> >
>> > >> > Binbin.Zhou at noaa.gov
>> > >> >
>> > >> > 301-683-3683
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > > --
>> > >
>> > > Binbin Zhou
>> > >
>> > > IMSG at NOAA/NWS/NCEP/EMC
>> > >
>> > > 5830 University Research Ct.
>> > >
>> > > College Park, MD 20740
>> > >
>> > > Binbin.Zhou at noaa.gov
>> > >
>> > > 301-683-3683
>> > >
>> >
>> >
>> > --
>> >
>> > Binbin Zhou
>> >
>> > IMSG at NOAA/NWS/NCEP/EMC
>> >
>> > 5830 University Research Ct.
>> >
>> > College Park, MD 20740
>> >
>> > Binbin.Zhou at noaa.gov
>> >
>> > 301-683-3683
>> >
>> >
>>
>>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>


--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Wed Dec 09 14:14:08 2020

Binbin,

Success! When I recompile with that one line commented out,
plot_data_plane
is able to filter down all those records to the single one that we
want.
Note in the example below that I used the "lead_time" and "GRIB_ens"
options to filter the input records down.

plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
level="P500";
lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
DEBUG 1: Opening data file: ensposte.t00z.z500hr
DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact match
for
VarInfo "HGT/P500" in GRIB record 26 of GRIB file
"ensposte.t00z.z500hr".*
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB records
matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
DEBUG 1: Creating postscript file: plot.ps

Also note that this selects out record number 26.

Shall I make this change as a "bugfix" for met-9.1...which could be
included in a met-9.1.2 bugfix release? Or shall I just include it in
development for the next release (planned for March 2021)?

Thanks,
John

On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   If this removing impacts other applications, or takes some time
> to be effective, then won't try.  I will use another approach, such
as
> borrow GEFS's grib2 data structure as tank to save EC ensemble
> data.
>
> Binbin
>
>
> On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
> binbin.zhou at noaa.gov> wrote:
>
> > John,
> >
> >   Yes, please try to use this approach.
> >
> > Thanks!
> >
> > Binbin
> >
> > Binbin
> >
> > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Binbin,
> >>
> >> I used the "-PDS10" option for wgrib to dump those records:
> >>
> >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> >> | egrep  "^11:|^26:|^43:"
> >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1
0 0 0
> 21
> >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1
0 0 0
> 21
> >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0 1
0 0 0
> 21
> >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> >>
> >> Looking VERY closely, there is a difference in the 42-nd column
which is
> >> the 4-th from the last... records 11, 26, and 43 have values of
3, 1,
> and
> >> 2
> >> respectively.
> >>
> >> On this page (
> https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> >> ),
> >> byte 42 = 1 means "unperturbed control forecast".
> >>
> >> So we *should* be able to use the "GRIB_ens" config option to
> >> differentiate
> >> between those records. BUT when I try, it doesn't work. And
that's
> because
> >> of this line:
> >>
> >>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> >>
> >> We only apply those filtering options to GRIB files created by
NCEP. If
> >> you'd like, I could try removing that condition to see if we can
pick
> out
> >> the single record of interest.
> >>
> >> John
> >>
> >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> >
> >> > John,
> >> >
> >> >   Look at my
> >> > directory
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> >> > The grib1 file is ensposte.t00z.z500hr, this is only for 500
HGT.
> >> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
> >> > I get 3 records:
> >> >
> >> >
> >> >
> >>
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > mb:anl:type=analysis:NAve=0
> >> >
> >> >
> >>
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > mb:anl:type=analysis:NAve=0
> >> >
> >> >
> >>
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > mb:anl:type=analysis:NAve=0
> >> >
> >> > From its grib2 file, we know only  record#26 is analysis low-
res
> >> control.
> >> > If you have some way to determine which record is analysis low-
res
> >> control
> >> > from the grib1 file,
> >> > then we can use grib1 directly (don't consider the size issue).
> >> >
> >> > Binbin
> >> >
> >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
> >> > binbin.zhou at noaa.gov> wrote:
> >> >
> >> > > John,
> >> > >
> >> > >   Grib1 data is too large, another issue is the low-res
control
> >> analysis
> >> > > data can not be retrieved
> >> > > from the grib1 ecmwf data files. If using wgrib, we can see 3
> analysis
> >> > > fields, but don't know
> >> > > which is low-res control. After convert to grib2, wgrib2 can
show
> >> which
> >> > > record is low-res control.
> >> > > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
> >> > >
> >> > > Binbin
> >> > >
> >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >> Binbin,
> >> > >>
> >> > >> Sounds like there's a lot of moving parts there. How would
you like
> >> to
> >> > >> proceed?
> >> > >>
> >> > >> Should we check to see if MET can read those ECMWF GRIB1
files
> >> directly
> >> > >> without needing to reformat?
> >> > >> Or do you just want to create a custom GRIB table for the
result of
> >> your
> >> > >> pre-processing and conversion steps?
> >> > >>
> >> > >> John
> >> > >>
> >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT <
> >> > >> met_help at ucar.edu> wrote:
> >> > >>
> >> > >> >
> >> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> > >> >
> >> > >> > John,
> >> > >> >
> >> > >> >   Our ECMWF grib2 ensemble data files are converted from
its
> grib1
> >> > >> format
> >> > >> > data.
> >> > >> > Before conversion, the parameter names are there, but
after
> >> > conversion,
> >> > >> the
> >> > >> > names
> >> > >> > are gone. It is because of unknown tables. The grib1 data
is
> still
> >> > >> > original. The original
> >> > >> > grib1 data files are very strange data format we can not
read. So
> >> we
> >> > >> have
> >> > >> > to use
> >> > >> > a Frotran code to re-arrange the fields and save them as
> NCEP-like
> >> > grib1
> >> > >> > data files.
> >> > >> > So after several processes, some important info like
tables could
> >> be
> >> > >> lost.
> >> > >> >
> >> > >> >    I'll test your first suggestion since the field record
order
> is
> >> > >> fixed.
> >> > >> > Currently we still
> >> > >> > obtained grib1 data from ECWMF, eventually, we will get
grib2
> data
> >> > >> > directly. So before
> >> > >> > we can get original grib2 files from ECMWF, we just use
this
> >> temporal
> >> > >> > solution.
> >> > >> > '
> >> > >> > Thanks!
> >> > >> >
> >> > >> >
> >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT <
> >> > >> > met_help at ucar.edu>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Binbin,
> >> > >> > >
> >> > >> > > OK, working on Venus.
> >> > >> > >
> >> > >> > > You have a typo in the command you sent.
> >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > >> > >
> >> > >>
> >>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> > >> > -v
> >> > >> > > 3
> >> > >> > >
> >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat". After
> >> > >> > correcting
> >> > >> > > that I did still get an error but it's a very useful
one!
> >> > >> > >
> >> > >> > > module use
> >> > >> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> >> > >> > > module load met/9.1
> >> > >> > > module load grib_util/1.1.1
> >> > >> > > cd
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> >> > >> > 'GRIB2_disc=255;
> >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
level="P500";'
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
parameter
> found
> >> > with
> >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3) GRIB2_parm
(5).
> Use
> >> the
> >> > >> > > MET_GRIB_TABLES environment variable to define custom
GRIB
> >> > >> tables.ERROR
> >> > >> > :
> >> > >> > > *
> >> > >> > >
> >> > >> > > So this table is not yet defined in MET. I checked their
> website:
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> >> > >> > >
> >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
> >> > >> > >
> >> > >> > > 3 things...
> >> > >> > >
> >> > >> > > (1) As a crude temporary solution, you could refer to it
by
> it's
> >> > >> record
> >> > >> > > number. Let's assume you know that record 1 has
temperature
> data:
> >> > >> > >
> >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 ~/plot.ps
> >> > 'name="TMP";
> >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> >> > >> > >
> >> > >> > > Here I'm reading record 1, naming it TMP and the using
> >> > set_attr_level
> >> > >> > write
> >> > >> > > the output level as P500 instead of R1.
> >> > >> > >
> >> > >> > > (2) You could create a custom GRIB table yourself
locally, and
> >> then
> >> > >> set
> >> > >> > the
> >> > >> > > MET_GRIB_TABLES environment variable to point MET to it.
> >> > >> > >
> >> > >> > > (3) The best solution is contacting ECMWF and requesting
they
> >> send
> >> > >> over
> >> > >> > the
> >> > >> > > GRIB2 table we should be using to decode this data.
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > John
> >> > >> > >
> >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
> >> johnhg at ucar.edu>
> >> > >> > wrote:
> >> > >> > >
> >> > >> > > > We might already have them. I'm not sure.
> >> > >> > > >
> >> > >> > > > But there's an additional required piece of the
puzzle.
> Binbin
> >> > runs
> >> > >> > > > "wgrib2" to figure out what "name" he should use in
the MET
> >> config
> >> > >> > files.
> >> > >> > > > If wgrib2 doesn't tell him the right "name", the
problem will
> >> > >> remain.
> >> > >> > > > wgrib2 can be configured to define additional tables
as well,
> >> but
> >> > >> for
> >> > >> > > this
> >> > >> > > > to be easy for Binbin, that'd mean updating the script
that's
> >> run
> >> > >> for
> >> > >> > > > "module load grib_utils" on WCOSS. And I'm guessing
that's
> not
> >> > very
> >> > >> > easy.
> >> > >> > > >
> >> > >> > > > Yes, I do agree that the best solution is getting the
ECMWF
> >> GRIB
> >> > >> tables
> >> > >> > > > and adding them for both wgrib2 and MET.
> >> > >> > > >
> >> > >> > > > John
> >> > >> > > >
> >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<jensen at ucar.edu>
> >> > wrote:
> >> > >> > > >
> >> > >> > > >> John,
> >> > >> > > >>
> >> > >> > > >> It seems to me that it would be worthwhile to just
ask ECMWF
> >> for
> >> > >> their
> >> > >> > > >> Grib2 Table for their Ensemble files.  And other
operational
> >> > >> centers
> >> > >> > for
> >> > >> > > >> that matter.  Do you agree this would be a good
solution?
> >> > >> > > >>
> >> > >> > > >> Thanks, Tara
> >> > >> > > >>
> >> > >> > > >> ---------- Forwarded message ---------
> >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <met_help at ucar.edu>
> >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent question
for
> >> > reading
> >> > >> > ECMWF
> >> > >> > > >> ensmeble data
> >> > >> > > >> To:
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >> >
> >> > >> > > >>
> >> > >> > > >> John,
> >> > >> > > >>
> >> > >> > > >>   I tested using following command:
> >> > >> > > >>
> >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > >> > > >>
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> -v3
> >> > >> > > >>
> >> > >> > > >> ERROR  :
> >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
> >> GRIB2_disc,
> >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
> >> > >> information
> >> > >> > > >>
> >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> >> > >> > > >>
> >> > >> >
> >> >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >> > >> > > >> -v
> >> > >> > > >> 3
> >> > >> > > >>
> >> > >> > > >> Still got error
> >> > >> > > >> ERROR  :
> >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
> >> GRIB2_disc,
> >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
> >> > >> information
> >> > >> > > >>
> >> > >> > > >> Looks like discipline=255 is not recognized
> >> > >> > > >>
> >> > >> > > >> Binbin
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway
via RT <
> >> > >> > > >> met_help at ucar.edu>
> >> > >> > > >> wrote:
> >> > >> > > >>
> >> > >> > > >> > Binbin,
> >> > >> > > >> > You can do something similar with MET. Please go to
this
> >> link:
> >> > >> > > >> >
> >> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> >> > >> > > >> >
> >> > >> > > >> > And then search for "GRIB2_mtab", and you'll find
the
> >> content
> >> > >> > > >> > copied/pasted below. You can specify these INSTEAD
OF
> >> > specifying
> >> > >> a
> >> > >> > > >> > requested name.
> >> > >> > > >> >
> >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
specifying the
> >> master
> >> > >> > table
> >> > >> > > >> > number.
> >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
specifying the
> >> local
> >> > >> table
> >> > >> > > >> > number.
> >> > >> > > >> > //       - The "GRIB2_disc" is an integer
specifying the
> >> GRIB2
> >> > >> > > >> discipline
> >> > >> > > >> > code.
> >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
specifying
> the
> >> > >> > parameter
> >> > >> > > >> > category
> >> > >> > > >> > //         code.
> >> > >> > > >> > //       - The "GRIB2_parm" is an integer
specifying the
> >> > >> parameter
> >> > >> > > code.
> >> > >> > > >> > //       - The "GRIB2_pdt" is an integer specifying
the
> >> product
> >> > >> > > >> definition
> >> > >> > > >> > //         template (Table 4.0).
> >> > >> > > >> > //       - The "GRIB2_process" is an integer
specifying
> the
> >> > >> > generating
> >> > >> > > >> > process
> >> > >> > > >> > //         (Table 4.3).
> >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
specifying the
> >> > >> originating
> >> > >> > > >> > center.
> >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
specifying
> the
> >> > >> > ensemble
> >> > >> > > >> type
> >> > >> > > >> > //         (Table 4.6).
> >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
specifying
> the
> >> > >> derived
> >> > >> > > >> > product
> >> > >> > > >> > //         type (Table 4.7).
> >> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
specifying
> >> the
> >> > >> > > >> statistical
> >> > >> > > >> > //         processing type (Table 4.10).
> >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> "GRIB2_ipdtmpl_val"
> >> > >> entries
> >> > >> > > are
> >> > >> > > >> > arrays
> >> > >> > > >> > //         of integers which specify the product
> description
> >> > >> > template
> >> > >> > > >> > values to
> >> > >> > > >> > //         be used. The indices are 0-based. For
example,
> >> use
> >> > the
> >> > >> > > >> > following to
> >> > >> > > >> > //         request a GRIB2 record whose 9-th and
27-th
> >> product
> >> > >> > > >> description
> >> > >> > > >> > //         template values are 1 and 2,
respectively:
> >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> >> > GRIB2_ipdtmpl_val=[1,
> >> > >> 2];
> >> > >> > > >> >
> >> > >> > > >> > While I don't have any ECMWF data on hand for
testing, I
> do
> >> > have
> >> > >> a
> >> > >> > > file
> >> > >> > > >> > from the Air Force from which wgrib2 prints a
similar
> >> result:
> >> > >> > > >> >
> >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
center=57
> >> > >> local_table=1
> >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> >> > >> > > >> >
> >> > >> > > >> > So I used that for testing. I ran the following
> >> plot_data_plane
> >> > >> > > command
> >> > >> > > >> > WITHOUT specifying a requested name:
> >> > >> > > >> >
> >> > >> > > >> >
> >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
plot.ps
> >> > >> > 'GRIB2_disc=0;
> >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
> >> GRIB2_parm=192;
> >> > >> > > >> level="Z0";'
> >> > >> > > >> > -v 3
> >> > >> > > >> >
> >> > >> > > >> > And then I looked closely at the log message to
confirm
> >> that it
> >> > >> used
> >> > >> > > >> > record 549, as expected:
> >> > >> > > >> >
> >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found
exact
> match
> >> > for
> >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2
file
> >> > >> > > >> >
> >> > >> > > >>
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> >> > >> > > >> >
> >> > >> > > >> > So MET used these GRIB2 numbers I defined to do a
table
> >> lookup
> >> > >> and
> >> > >> > > find
> >> > >> > > >> a
> >> > >> > > >> > match in the MET GRIB tables:
> >> > >> > > >> >
> >> > >>
> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> >> > >> > > >> >
> >> > >> > > >> > And it determined that that field should be called
> "CTSTM".
> >> > >> > Hopefully,
> >> > >> > > >> the
> >> > >> > > >> > ECMWF GRIB2 tables are already defined there in
MET. If
> >> not, we
> >> > >> can
> >> > >> > > >> always
> >> > >> > > >> > add them.
> >> > >> > > >> >
> >> > >> > > >> > Please try running plot_data_plane at verbosity
level 3 on
> >> one
> >> > of
> >> > >> > > these
> >> > >> > > >> > files first. And make sure it runs without error
and the
> log
> >> > >> message
> >> > >> > > >> prints
> >> > >> > > >> > out the "name" it found for that data. If so, you
can just
> >> skip
> >> > >> > using
> >> > >> > > >> all
> >> > >> > > >> > those integers and revert back to using the name
MET
> knows.
> >> If
> >> > >> not,
> >> > >> > > >> then we
> >> > >> > > >> > may need to define a custom GRIB table for this
ECMWF
> GRIB2
> >> > data.
> >> > >> > > >> >
> >> > >> > > >> > Thanks,
> >> > >> > > >> > John
> >> > >> > > >> >
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >> --
> >> > >> > > >>
> >> > >> > > >> Binbin Zhou
> >> > >> > > >>
> >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> >> > >> > > >>
> >> > >> > > >> 5830 University Research Ct.
> >> > >> > > >>
> >> > >> > > >> College Park, MD 20740
> >> > >> > > >>
> >> > >> > > >> Binbin.Zhou at noaa.gov
> >> > >> > > >>
> >> > >> > > >> 301-683-3683
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >>
> >> > >> > > >> --
> >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> > >> > > >> Tara Jensen
> >> > >> > > >> Project Manager II
> >> > >> > > >> NCAR RAL and DTC
> >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> >> > >> > > >>
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >> > --
> >> > >> >
> >> > >> > Binbin Zhou
> >> > >> >
> >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> >> > >> >
> >> > >> > 5830 University Research Ct.
> >> > >> >
> >> > >> > College Park, MD 20740
> >> > >> >
> >> > >> > Binbin.Zhou at noaa.gov
> >> > >> >
> >> > >> > 301-683-3683
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> > >
> >> > > --
> >> > >
> >> > > Binbin Zhou
> >> > >
> >> > > IMSG at NOAA/NWS/NCEP/EMC
> >> > >
> >> > > 5830 University Research Ct.
> >> > >
> >> > > College Park, MD 20740
> >> > >
> >> > > Binbin.Zhou at noaa.gov
> >> > >
> >> > > 301-683-3683
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > Binbin Zhou
> >> >
> >> > IMSG at NOAA/NWS/NCEP/EMC
> >> >
> >> > 5830 University Research Ct.
> >> >
> >> > College Park, MD 20740
> >> >
> >> > Binbin.Zhou at noaa.gov
> >> >
> >> > 301-683-3683
> >> >
> >> >
> >>
> >>
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Wed Dec 09 14:28:52 2020

John,

  Great! Let me test it at first before make change in MET9.1
Another question:
   Since climatology data is still in NCEP's grib2 format, In the
ensemble_stat
config file,  climo  entry settings  should be different from those of
fcst, obs entries?
How to set climo (name, level, etc.)?

Binbin



On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
>
> Success! When I recompile with that one line commented out,
plot_data_plane
> is able to filter down all those records to the single one that we
want.
> Note in the example below that I used the "lead_time" and "GRIB_ens"
> options to filter the input records down.
>
> plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
level="P500";
> lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> DEBUG 1: Opening data file: ensposte.t00z.z500hr
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact match
for
> VarInfo "HGT/P500" in GRIB record 26 of GRIB file
"ensposte.t00z.z500hr".*
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
> DEBUG 1: Creating postscript file: plot.ps
>
> Also note that this selects out record number 26.
>
> Shall I make this change as a "bugfix" for met-9.1...which could be
> included in a met-9.1.2 bugfix release? Or shall I just include it
in
> development for the next release (planned for March 2021)?
>
> Thanks,
> John
>
> On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > John,
> >
> >   If this removing impacts other applications, or takes some time
> > to be effective, then won't try.  I will use another approach,
such as
> > borrow GEFS's grib2 data structure as tank to save EC ensemble
> > data.
> >
> > Binbin
> >
> >
> > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
> > binbin.zhou at noaa.gov> wrote:
> >
> > > John,
> > >
> > >   Yes, please try to use this approach.
> > >
> > > Thanks!
> > >
> > > Binbin
> > >
> > > Binbin
> > >
> > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Binbin,
> > >>
> > >> I used the "-PDS10" option for wgrib to dump those records:
> > >>
> > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > >> | egrep  "^11:|^26:|^43:"
> > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0
1 0 0
> 0
> > 21
> > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0
1 0 0
> 0
> > 21
> > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0 0
1 0 0
> 0
> > 21
> > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > >>
> > >> Looking VERY closely, there is a difference in the 42-nd column
which
> is
> > >> the 4-th from the last... records 11, 26, and 43 have values of
3, 1,
> > and
> > >> 2
> > >> respectively.
> > >>
> > >> On this page (
> > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > >> ),
> > >> byte 42 = 1 means "unperturbed control forecast".
> > >>
> > >> So we *should* be able to use the "GRIB_ens" config option to
> > >> differentiate
> > >> between those records. BUT when I try, it doesn't work. And
that's
> > because
> > >> of this line:
> > >>
> > >>
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > >>
> > >> We only apply those filtering options to GRIB files created by
NCEP.
> If
> > >> you'd like, I could try removing that condition to see if we
can pick
> > out
> > >> the single record of interest.
> > >>
> > >> John
> > >>
> > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > >> >
> > >> > John,
> > >> >
> > >> >   Look at my
> > >> > directory
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > >> > The grib1 file is ensposte.t00z.z500hr, this is only for 500
HGT.
> > >> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
> > >> > I get 3 records:
> > >> >
> > >> >
> > >> >
> > >>
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >> > mb:anl:type=analysis:NAve=0
> > >> >
> > >> >
> > >>
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >> > mb:anl:type=analysis:NAve=0
> > >> >
> > >> >
> > >>
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >> > mb:anl:type=analysis:NAve=0
> > >> >
> > >> > From its grib2 file, we know only  record#26 is analysis low-
res
> > >> control.
> > >> > If you have some way to determine which record is analysis
low-res
> > >> control
> > >> > from the grib1 file,
> > >> > then we can use grib1 directly (don't consider the size
issue).
> > >> >
> > >> > Binbin
> > >> >
> > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate <
> > >> > binbin.zhou at noaa.gov> wrote:
> > >> >
> > >> > > John,
> > >> > >
> > >> > >   Grib1 data is too large, another issue is the low-res
control
> > >> analysis
> > >> > > data can not be retrieved
> > >> > > from the grib1 ecmwf data files. If using wgrib, we can see
3
> > analysis
> > >> > > fields, but don't know
> > >> > > which is low-res control. After convert to grib2, wgrib2
can show
> > >> which
> > >> > > record is low-res control.
> > >> > > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
> > >> > >
> > >> > > Binbin
> > >> > >
> > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > >> Binbin,
> > >> > >>
> > >> > >> Sounds like there's a lot of moving parts there. How would
you
> like
> > >> to
> > >> > >> proceed?
> > >> > >>
> > >> > >> Should we check to see if MET can read those ECMWF GRIB1
files
> > >> directly
> > >> > >> without needing to reformat?
> > >> > >> Or do you just want to create a custom GRIB table for the
result
> of
> > >> your
> > >> > >> pre-processing and conversion steps?
> > >> > >>
> > >> > >> John
> > >> > >>
> > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via RT
<
> > >> > >> met_help at ucar.edu> wrote:
> > >> > >>
> > >> > >> >
> > >> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> > >> > >> >
> > >> > >> > John,
> > >> > >> >
> > >> > >> >   Our ECMWF grib2 ensemble data files are converted from
its
> > grib1
> > >> > >> format
> > >> > >> > data.
> > >> > >> > Before conversion, the parameter names are there, but
after
> > >> > conversion,
> > >> > >> the
> > >> > >> > names
> > >> > >> > are gone. It is because of unknown tables. The grib1
data is
> > still
> > >> > >> > original. The original
> > >> > >> > grib1 data files are very strange data format we can not
read.
> So
> > >> we
> > >> > >> have
> > >> > >> > to use
> > >> > >> > a Frotran code to re-arrange the fields and save them as
> > NCEP-like
> > >> > grib1
> > >> > >> > data files.
> > >> > >> > So after several processes, some important info like
tables
> could
> > >> be
> > >> > >> lost.
> > >> > >> >
> > >> > >> >    I'll test your first suggestion since the field
record order
> > is
> > >> > >> fixed.
> > >> > >> > Currently we still
> > >> > >> > obtained grib1 data from ECWMF, eventually, we will get
grib2
> > data
> > >> > >> > directly. So before
> > >> > >> > we can get original grib2 files from ECMWF, we just use
this
> > >> temporal
> > >> > >> > solution.
> > >> > >> > '
> > >> > >> > Thanks!
> > >> > >> >
> > >> > >> >
> > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via RT
<
> > >> > >> > met_help at ucar.edu>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Binbin,
> > >> > >> > >
> > >> > >> > > OK, working on Venus.
> > >> > >> > >
> > >> > >> > > You have a typo in the command you sent.
> > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >> > >> > >
> > >> > >>
> > >>
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> > >> > -v
> > >> > >> > > 3
> > >> > >> > >
> > >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat".
> After
> > >> > >> > correcting
> > >> > >> > > that I did still get an error but it's a very useful
one!
> > >> > >> > >
> > >> > >> > > module use
> > >> > >>
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > >> > >> > > module load met/9.1
> > >> > >> > > module load grib_util/1.1.1
> > >> > >> > > cd
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > >> > >> > 'GRIB2_disc=255;
> > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
level="P500";'
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
parameter
> > found
> > >> > with
> > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm (5).
> > Use
> > >> the
> > >> > >> > > MET_GRIB_TABLES environment variable to define custom
GRIB
> > >> > >> tables.ERROR
> > >> > >> > :
> > >> > >> > > *
> > >> > >> > >
> > >> > >> > > So this table is not yet defined in MET. I checked
their
> > website:
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > >> > >> > >
> > >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
> > >> > >> > >
> > >> > >> > > 3 things...
> > >> > >> > >
> > >> > >> > > (1) As a crude temporary solution, you could refer to
it by
> > it's
> > >> > >> record
> > >> > >> > > number. Let's assume you know that record 1 has
temperature
> > data:
> > >> > >> > >
> > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > >> > 'name="TMP";
> > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > >> > >> > >
> > >> > >> > > Here I'm reading record 1, naming it TMP and the using
> > >> > set_attr_level
> > >> > >> > write
> > >> > >> > > the output level as P500 instead of R1.
> > >> > >> > >
> > >> > >> > > (2) You could create a custom GRIB table yourself
locally,
> and
> > >> then
> > >> > >> set
> > >> > >> > the
> > >> > >> > > MET_GRIB_TABLES environment variable to point MET to
it.
> > >> > >> > >
> > >> > >> > > (3) The best solution is contacting ECMWF and
requesting they
> > >> send
> > >> > >> over
> > >> > >> > the
> > >> > >> > > GRIB2 table we should be using to decode this data.
> > >> > >> > >
> > >> > >> > > Thanks,
> > >> > >> > > John
> > >> > >> > >
> > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
> > >> johnhg at ucar.edu>
> > >> > >> > wrote:
> > >> > >> > >
> > >> > >> > > > We might already have them. I'm not sure.
> > >> > >> > > >
> > >> > >> > > > But there's an additional required piece of the
puzzle.
> > Binbin
> > >> > runs
> > >> > >> > > > "wgrib2" to figure out what "name" he should use in
the MET
> > >> config
> > >> > >> > files.
> > >> > >> > > > If wgrib2 doesn't tell him the right "name", the
problem
> will
> > >> > >> remain.
> > >> > >> > > > wgrib2 can be configured to define additional tables
as
> well,
> > >> but
> > >> > >> for
> > >> > >> > > this
> > >> > >> > > > to be easy for Binbin, that'd mean updating the
script
> that's
> > >> run
> > >> > >> for
> > >> > >> > > > "module load grib_utils" on WCOSS. And I'm guessing
that's
> > not
> > >> > very
> > >> > >> > easy.
> > >> > >> > > >
> > >> > >> > > > Yes, I do agree that the best solution is getting
the ECMWF
> > >> GRIB
> > >> > >> tables
> > >> > >> > > > and adding them for both wgrib2 and MET.
> > >> > >> > > >
> > >> > >> > > > John
> > >> > >> > > >
> > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
> jensen at ucar.edu>
> > >> > wrote:
> > >> > >> > > >
> > >> > >> > > >> John,
> > >> > >> > > >>
> > >> > >> > > >> It seems to me that it would be worthwhile to just
ask
> ECMWF
> > >> for
> > >> > >> their
> > >> > >> > > >> Grib2 Table for their Ensemble files.  And other
> operational
> > >> > >> centers
> > >> > >> > for
> > >> > >> > > >> that matter.  Do you agree this would be a good
solution?
> > >> > >> > > >>
> > >> > >> > > >> Thanks, Tara
> > >> > >> > > >>
> > >> > >> > > >> ---------- Forwarded message ---------
> > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT
<met_help at ucar.edu>
> > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
question for
> > >> > reading
> > >> > >> > ECMWF
> > >> > >> > > >> ensmeble data
> > >> > >> > > >> To:
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >> <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >> >
> > >> > >> > > >>
> > >> > >> > > >> John,
> > >> > >> > > >>
> > >> > >> > > >>   I tested using following command:
> > >> > >> > > >>
> > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > >> > >> > > >>
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> -v3
> > >> > >> > > >>
> > >> > >> > > >> ERROR  :
> > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
> > >> GRIB2_disc,
> > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
> > >> > >> information
> > >> > >> > > >>
> > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > >> > >> > > >>
> > >> > >> >
> > >> >
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >> > >> > > >> -v
> > >> > >> > > >> 3
> > >> > >> > > >>
> > >> > >> > > >> Still got error
> > >> > >> > > >> ERROR  :
> > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name or
> > >> GRIB2_disc,
> > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified in
field
> > >> > >> information
> > >> > >> > > >>
> > >> > >> > > >> Looks like discipline=255 is not recognized
> > >> > >> > > >>
> > >> > >> > > >> Binbin
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley Gotway
via RT
> <
> > >> > >> > > >> met_help at ucar.edu>
> > >> > >> > > >> wrote:
> > >> > >> > > >>
> > >> > >> > > >> > Binbin,
> > >> > >> > > >> > You can do something similar with MET. Please go
to this
> > >> link:
> > >> > >> > > >> >
> > >> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > >> > >> > > >> >
> > >> > >> > > >> > And then search for "GRIB2_mtab", and you'll find
the
> > >> content
> > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD OF
> > >> > specifying
> > >> > >> a
> > >> > >> > > >> > requested name.
> > >> > >> > > >> >
> > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
specifying the
> > >> master
> > >> > >> > table
> > >> > >> > > >> > number.
> > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
specifying the
> > >> local
> > >> > >> table
> > >> > >> > > >> > number.
> > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
specifying the
> > >> GRIB2
> > >> > >> > > >> discipline
> > >> > >> > > >> > code.
> > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
specifying
> > the
> > >> > >> > parameter
> > >> > >> > > >> > category
> > >> > >> > > >> > //         code.
> > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
specifying the
> > >> > >> parameter
> > >> > >> > > code.
> > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
specifying the
> > >> product
> > >> > >> > > >> definition
> > >> > >> > > >> > //         template (Table 4.0).
> > >> > >> > > >> > //       - The "GRIB2_process" is an integer
specifying
> > the
> > >> > >> > generating
> > >> > >> > > >> > process
> > >> > >> > > >> > //         (Table 4.3).
> > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
specifying the
> > >> > >> originating
> > >> > >> > > >> > center.
> > >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
specifying
> > the
> > >> > >> > ensemble
> > >> > >> > > >> type
> > >> > >> > > >> > //         (Table 4.6).
> > >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
specifying
> > the
> > >> > >> derived
> > >> > >> > > >> > product
> > >> > >> > > >> > //         type (Table 4.7).
> > >> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
> specifying
> > >> the
> > >> > >> > > >> statistical
> > >> > >> > > >> > //         processing type (Table 4.10).
> > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> > "GRIB2_ipdtmpl_val"
> > >> > >> entries
> > >> > >> > > are
> > >> > >> > > >> > arrays
> > >> > >> > > >> > //         of integers which specify the product
> > description
> > >> > >> > template
> > >> > >> > > >> > values to
> > >> > >> > > >> > //         be used. The indices are 0-based. For
> example,
> > >> use
> > >> > the
> > >> > >> > > >> > following to
> > >> > >> > > >> > //         request a GRIB2 record whose 9-th and
27-th
> > >> product
> > >> > >> > > >> description
> > >> > >> > > >> > //         template values are 1 and 2,
respectively:
> > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> > >> > GRIB2_ipdtmpl_val=[1,
> > >> > >> 2];
> > >> > >> > > >> >
> > >> > >> > > >> > While I don't have any ECMWF data on hand for
testing, I
> > do
> > >> > have
> > >> > >> a
> > >> > >> > > file
> > >> > >> > > >> > from the Air Force from which wgrib2 prints a
similar
> > >> result:
> > >> > >> > > >> >
> > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
center=57
> > >> > >> local_table=1
> > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > >> > >> > > >> >
> > >> > >> > > >> > So I used that for testing. I ran the following
> > >> plot_data_plane
> > >> > >> > > command
> > >> > >> > > >> > WITHOUT specifying a requested name:
> > >> > >> > > >> >
> > >> > >> > > >> >
> > >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
plot.ps
> > >> > >> > 'GRIB2_disc=0;
> > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
> > >> GRIB2_parm=192;
> > >> > >> > > >> level="Z0";'
> > >> > >> > > >> > -v 3
> > >> > >> > > >> >
> > >> > >> > > >> > And then I looked closely at the log message to
confirm
> > >> that it
> > >> > >> used
> > >> > >> > > >> > record 549, as expected:
> > >> > >> > > >> >
> > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() -> Found
exact
> > match
> > >> > for
> > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2
file
> > >> > >> > > >> >
> > >> > >> > > >>
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > >> > >> > > >> >
> > >> > >> > > >> > So MET used these GRIB2 numbers I defined to do a
table
> > >> lookup
> > >> > >> and
> > >> > >> > > find
> > >> > >> > > >> a
> > >> > >> > > >> > match in the MET GRIB tables:
> > >> > >> > > >> >
> > >> > >>
> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > >> > >> > > >> >
> > >> > >> > > >> > And it determined that that field should be
called
> > "CTSTM".
> > >> > >> > Hopefully,
> > >> > >> > > >> the
> > >> > >> > > >> > ECMWF GRIB2 tables are already defined there in
MET. If
> > >> not, we
> > >> > >> can
> > >> > >> > > >> always
> > >> > >> > > >> > add them.
> > >> > >> > > >> >
> > >> > >> > > >> > Please try running plot_data_plane at verbosity
level 3
> on
> > >> one
> > >> > of
> > >> > >> > > these
> > >> > >> > > >> > files first. And make sure it runs without error
and the
> > log
> > >> > >> message
> > >> > >> > > >> prints
> > >> > >> > > >> > out the "name" it found for that data. If so, you
can
> just
> > >> skip
> > >> > >> > using
> > >> > >> > > >> all
> > >> > >> > > >> > those integers and revert back to using the name
MET
> > knows.
> > >> If
> > >> > >> not,
> > >> > >> > > >> then we
> > >> > >> > > >> > may need to define a custom GRIB table for this
ECMWF
> > GRIB2
> > >> > data.
> > >> > >> > > >> >
> > >> > >> > > >> > Thanks,
> > >> > >> > > >> > John
> > >> > >> > > >> >
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >> --
> > >> > >> > > >>
> > >> > >> > > >> Binbin Zhou
> > >> > >> > > >>
> > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > >> > >> > > >>
> > >> > >> > > >> 5830 University Research Ct.
> > >> > >> > > >>
> > >> > >> > > >> College Park, MD 20740
> > >> > >> > > >>
> > >> > >> > > >> Binbin.Zhou at noaa.gov
> > >> > >> > > >>
> > >> > >> > > >> 301-683-3683
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >>
> > >> > >> > > >> --
> > >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >> > >> > > >> Tara Jensen
> > >> > >> > > >> Project Manager II
> > >> > >> > > >> NCAR RAL and DTC
> > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > >> > >> > > >>
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >> > --
> > >> > >> >
> > >> > >> > Binbin Zhou
> > >> > >> >
> > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >> > >> >
> > >> > >> > 5830 University Research Ct.
> > >> > >> >
> > >> > >> > College Park, MD 20740
> > >> > >> >
> > >> > >> > Binbin.Zhou at noaa.gov
> > >> > >> >
> > >> > >> > 301-683-3683
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >
> > >> > > --
> > >> > >
> > >> > > Binbin Zhou
> > >> > >
> > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > >> > >
> > >> > > 5830 University Research Ct.
> > >> > >
> > >> > > College Park, MD 20740
> > >> > >
> > >> > > Binbin.Zhou at noaa.gov
> > >> > >
> > >> > > 301-683-3683
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Binbin Zhou
> > >> >
> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >> >
> > >> > 5830 University Research Ct.
> > >> >
> > >> > College Park, MD 20740
> > >> >
> > >> > Binbin.Zhou at noaa.gov
> > >> >
> > >> > 301-683-3683
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > >
> > > Binbin Zhou
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct.
> > >
> > > College Park, MD 20740
> > >
> > > Binbin.Zhou at noaa.gov
> > >
> > > 301-683-3683
> > >
> >
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Wed Dec 09 16:53:13 2020

Binbin,

In order for you to test this change out, we'd need to recompile a
version
of the code on WCOSS. I wrote this up as a GitHub issue:
https://github.com/dtcenter/MET/issues/1604

And I'll create a branch from main_v9.1 named "bugfix_1604_ens_config"
and
commit the change there. Then we can checkout that code on WCOSS,
compile
it, and you can test it.

Yes, you're right, if the fcst fields differ from the climo fields,
you'd
need to specify them separately. That's pretty easy in the Ensemble-
Stat
config file... but are you asking how to control that via METplus?

John

On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   Great! Let me test it at first before make change in MET9.1
> Another question:
>    Since climatology data is still in NCEP's grib2 format, In the
> ensemble_stat
> config file,  climo  entry settings  should be different from those
of
> fcst, obs entries?
> How to set climo (name, level, etc.)?
>
> Binbin
>
>
>
> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binbin,
> >
> > Success! When I recompile with that one line commented out,
> plot_data_plane
> > is able to filter down all those records to the single one that we
want.
> > Note in the example below that I used the "lead_time" and
"GRIB_ens"
> > options to filter the input records down.
> >
> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
level="P500";
> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact
match for
> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> "ensposte.t00z.z500hr".*
> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> > matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
> > DEBUG 1: Creating postscript file: plot.ps
> >
> > Also note that this selects out record number 26.
> >
> > Shall I make this change as a "bugfix" for met-9.1...which could
be
> > included in a met-9.1.2 bugfix release? Or shall I just include it
in
> > development for the next release (planned for March 2021)?
> >
> > Thanks,
> > John
> >
> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >
> > > John,
> > >
> > >   If this removing impacts other applications, or takes some
time
> > > to be effective, then won't try.  I will use another approach,
such as
> > > borrow GEFS's grib2 data structure as tank to save EC ensemble
> > > data.
> > >
> > > Binbin
> > >
> > >
> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
> > > binbin.zhou at noaa.gov> wrote:
> > >
> > > > John,
> > > >
> > > >   Yes, please try to use this approach.
> > > >
> > > > Thanks!
> > > >
> > > > Binbin
> > > >
> > > > Binbin
> > > >
> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Binbin,
> > > >>
> > > >> I used the "-PDS10" option for wgrib to dump those records:
> > > >>
> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > > >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > > >> | egrep  "^11:|^26:|^43:"
> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0
0 1 0
> 0
> > 0
> > > 21
> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0
0 1 0
> 0
> > 0
> > > 21
> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1 0
0 1 0
> 0
> > 0
> > > 21
> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > > >>
> > > >> Looking VERY closely, there is a difference in the 42-nd
column
> which
> > is
> > > >> the 4-th from the last... records 11, 26, and 43 have values
of 3,
> 1,
> > > and
> > > >> 2
> > > >> respectively.
> > > >>
> > > >> On this page (
> > > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > > >> ),
> > > >> byte 42 = 1 means "unperturbed control forecast".
> > > >>
> > > >> So we *should* be able to use the "GRIB_ens" config option to
> > > >> differentiate
> > > >> between those records. BUT when I try, it doesn't work. And
that's
> > > because
> > > >> of this line:
> > > >>
> > > >>
> > >
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > > >>
> > > >> We only apply those filtering options to GRIB files created
by NCEP.
> > If
> > > >> you'd like, I could try removing that condition to see if we
can
> pick
> > > out
> > > >> the single record of interest.
> > > >>
> > > >> John
> > > >>
> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > > >> >
> > > >> > John,
> > > >> >
> > > >> >   Look at my
> > > >> > directory
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > > >> > The grib1 file is ensposte.t00z.z500hr, this is only for
500 HGT.
> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
> > > >> > I get 3 records:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >> > mb:anl:type=analysis:NAve=0
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >> > mb:anl:type=analysis:NAve=0
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >> > mb:anl:type=analysis:NAve=0
> > > >> >
> > > >> > From its grib2 file, we know only  record#26 is analysis
low-res
> > > >> control.
> > > >> > If you have some way to determine which record is analysis
low-res
> > > >> control
> > > >> > from the grib1 file,
> > > >> > then we can use grib1 directly (don't consider the size
issue).
> > > >> >
> > > >> > Binbin
> > > >> >
> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA Affiliate
<
> > > >> > binbin.zhou at noaa.gov> wrote:
> > > >> >
> > > >> > > John,
> > > >> > >
> > > >> > >   Grib1 data is too large, another issue is the low-res
control
> > > >> analysis
> > > >> > > data can not be retrieved
> > > >> > > from the grib1 ecmwf data files. If using wgrib, we can
see 3
> > > analysis
> > > >> > > fields, but don't know
> > > >> > > which is low-res control. After convert to grib2, wgrib2
can
> show
> > > >> which
> > > >> > > record is low-res control.
> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on WCOSS.
> > > >> > >
> > > >> > > Binbin
> > > >> > >
> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT
<
> > > >> > > met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > >> Binbin,
> > > >> > >>
> > > >> > >> Sounds like there's a lot of moving parts there. How
would you
> > like
> > > >> to
> > > >> > >> proceed?
> > > >> > >>
> > > >> > >> Should we check to see if MET can read those ECMWF GRIB1
files
> > > >> directly
> > > >> > >> without needing to reformat?
> > > >> > >> Or do you just want to create a custom GRIB table for
the
> result
> > of
> > > >> your
> > > >> > >> pre-processing and conversion steps?
> > > >> > >>
> > > >> > >> John
> > > >> > >>
> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via
RT <
> > > >> > >> met_help at ucar.edu> wrote:
> > > >> > >>
> > > >> > >> >
> > > >> > >> > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >
> > > >> > >> >
> > > >> > >> > John,
> > > >> > >> >
> > > >> > >> >   Our ECMWF grib2 ensemble data files are converted
from its
> > > grib1
> > > >> > >> format
> > > >> > >> > data.
> > > >> > >> > Before conversion, the parameter names are there, but
after
> > > >> > conversion,
> > > >> > >> the
> > > >> > >> > names
> > > >> > >> > are gone. It is because of unknown tables. The grib1
data is
> > > still
> > > >> > >> > original. The original
> > > >> > >> > grib1 data files are very strange data format we can
not
> read.
> > So
> > > >> we
> > > >> > >> have
> > > >> > >> > to use
> > > >> > >> > a Frotran code to re-arrange the fields and save them
as
> > > NCEP-like
> > > >> > grib1
> > > >> > >> > data files.
> > > >> > >> > So after several processes, some important info like
tables
> > could
> > > >> be
> > > >> > >> lost.
> > > >> > >> >
> > > >> > >> >    I'll test your first suggestion since the field
record
> order
> > > is
> > > >> > >> fixed.
> > > >> > >> > Currently we still
> > > >> > >> > obtained grib1 data from ECWMF, eventually, we will
get grib2
> > > data
> > > >> > >> > directly. So before
> > > >> > >> > we can get original grib2 files from ECMWF, we just
use this
> > > >> temporal
> > > >> > >> > solution.
> > > >> > >> > '
> > > >> > >> > Thanks!
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via
RT <
> > > >> > >> > met_help at ucar.edu>
> > > >> > >> > wrote:
> > > >> > >> >
> > > >> > >> > > Binbin,
> > > >> > >> > >
> > > >> > >> > > OK, working on Venus.
> > > >> > >> > >
> > > >> > >> > > You have a typo in the command you sent.
> > > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > > >> > >> > >
> > > >> > >>
> > > >>
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >> > >> > -v
> > > >> > >> > > 3
> > > >> > >> > >
> > > >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat".
> > After
> > > >> > >> > correcting
> > > >> > >> > > that I did still get an error but it's a very useful
one!
> > > >> > >> > >
> > > >> > >> > > module use
> > > >> > >>
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > >> > >> > > module load met/9.1
> > > >> > >> > > module load grib_util/1.1.1
> > > >> > >> > > cd
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > > >> > >> > 'GRIB2_disc=255;
> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
> level="P500";'
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
parameter
> > > found
> > > >> > with
> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm
> (5).
> > > Use
> > > >> the
> > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom GRIB
> > > >> > >> tables.ERROR
> > > >> > >> > :
> > > >> > >> > > *
> > > >> > >> > >
> > > >> > >> > > So this table is not yet defined in MET. I checked
their
> > > website:
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > > >> > >> > >
> > > >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
> > > >> > >> > >
> > > >> > >> > > 3 things...
> > > >> > >> > >
> > > >> > >> > > (1) As a crude temporary solution, you could refer
to it by
> > > it's
> > > >> > >> record
> > > >> > >> > > number. Let's assume you know that record 1 has
temperature
> > > data:
> > > >> > >> > >
> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > > >> > 'name="TMP";
> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > > >> > >> > >
> > > >> > >> > > Here I'm reading record 1, naming it TMP and the
using
> > > >> > set_attr_level
> > > >> > >> > write
> > > >> > >> > > the output level as P500 instead of R1.
> > > >> > >> > >
> > > >> > >> > > (2) You could create a custom GRIB table yourself
locally,
> > and
> > > >> then
> > > >> > >> set
> > > >> > >> > the
> > > >> > >> > > MET_GRIB_TABLES environment variable to point MET to
it.
> > > >> > >> > >
> > > >> > >> > > (3) The best solution is contacting ECMWF and
requesting
> they
> > > >> send
> > > >> > >> over
> > > >> > >> > the
> > > >> > >> > > GRIB2 table we should be using to decode this data.
> > > >> > >> > >
> > > >> > >> > > Thanks,
> > > >> > >> > > John
> > > >> > >> > >
> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
> > > >> johnhg at ucar.edu>
> > > >> > >> > wrote:
> > > >> > >> > >
> > > >> > >> > > > We might already have them. I'm not sure.
> > > >> > >> > > >
> > > >> > >> > > > But there's an additional required piece of the
puzzle.
> > > Binbin
> > > >> > runs
> > > >> > >> > > > "wgrib2" to figure out what "name" he should use
in the
> MET
> > > >> config
> > > >> > >> > files.
> > > >> > >> > > > If wgrib2 doesn't tell him the right "name", the
problem
> > will
> > > >> > >> remain.
> > > >> > >> > > > wgrib2 can be configured to define additional
tables as
> > well,
> > > >> but
> > > >> > >> for
> > > >> > >> > > this
> > > >> > >> > > > to be easy for Binbin, that'd mean updating the
script
> > that's
> > > >> run
> > > >> > >> for
> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
> that's
> > > not
> > > >> > very
> > > >> > >> > easy.
> > > >> > >> > > >
> > > >> > >> > > > Yes, I do agree that the best solution is getting
the
> ECMWF
> > > >> GRIB
> > > >> > >> tables
> > > >> > >> > > > and adding them for both wgrib2 and MET.
> > > >> > >> > > >
> > > >> > >> > > > John
> > > >> > >> > > >
> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
> > jensen at ucar.edu>
> > > >> > wrote:
> > > >> > >> > > >
> > > >> > >> > > >> John,
> > > >> > >> > > >>
> > > >> > >> > > >> It seems to me that it would be worthwhile to
just ask
> > ECMWF
> > > >> for
> > > >> > >> their
> > > >> > >> > > >> Grib2 Table for their Ensemble files.  And other
> > operational
> > > >> > >> centers
> > > >> > >> > for
> > > >> > >> > > >> that matter.  Do you agree this would be a good
> solution?
> > > >> > >> > > >>
> > > >> > >> > > >> Thanks, Tara
> > > >> > >> > > >>
> > > >> > >> > > >> ---------- Forwarded message ---------
> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT
<met_help at ucar.edu>
> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
question
> for
> > > >> > reading
> > > >> > >> > ECMWF
> > > >> > >> > > >> ensmeble data
> > > >> > >> > > >> To:
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >> <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >> >
> > > >> > >> > > >>
> > > >> > >> > > >> John,
> > > >> > >> > > >>
> > > >> > >> > > >>   I tested using following command:
> > > >> > >> > > >>
> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > > >> > >> > > >>
> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >> -v3
> > > >> > >> > > >>
> > > >> > >> > > >> ERROR  :
> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name
or
> > > >> GRIB2_disc,
> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in field
> > > >> > >> information
> > > >> > >> > > >>
> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > > >> > >> > > >>
> > > >> > >> >
> > > >> >
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >> > >> > > >> -v
> > > >> > >> > > >> 3
> > > >> > >> > > >>
> > > >> > >> > > >> Still got error
> > > >> > >> > > >> ERROR  :
> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name
or
> > > >> GRIB2_disc,
> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in field
> > > >> > >> information
> > > >> > >> > > >>
> > > >> > >> > > >> Looks like discipline=255 is not recognized
> > > >> > >> > > >>
> > > >> > >> > > >> Binbin
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway via
> RT
> > <
> > > >> > >> > > >> met_help at ucar.edu>
> > > >> > >> > > >> wrote:
> > > >> > >> > > >>
> > > >> > >> > > >> > Binbin,
> > > >> > >> > > >> > You can do something similar with MET. Please
go to
> this
> > > >> link:
> > > >> > >> > > >> >
> > > >>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > >> > >> > > >> >
> > > >> > >> > > >> > And then search for "GRIB2_mtab", and you'll
find the
> > > >> content
> > > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD OF
> > > >> > specifying
> > > >> > >> a
> > > >> > >> > > >> > requested name.
> > > >> > >> > > >> >
> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
specifying
> the
> > > >> master
> > > >> > >> > table
> > > >> > >> > > >> > number.
> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
specifying
> the
> > > >> local
> > > >> > >> table
> > > >> > >> > > >> > number.
> > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
specifying
> the
> > > >> GRIB2
> > > >> > >> > > >> discipline
> > > >> > >> > > >> > code.
> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
> specifying
> > > the
> > > >> > >> > parameter
> > > >> > >> > > >> > category
> > > >> > >> > > >> > //         code.
> > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
specifying
> the
> > > >> > >> parameter
> > > >> > >> > > code.
> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
specifying
> the
> > > >> product
> > > >> > >> > > >> definition
> > > >> > >> > > >> > //         template (Table 4.0).
> > > >> > >> > > >> > //       - The "GRIB2_process" is an integer
> specifying
> > > the
> > > >> > >> > generating
> > > >> > >> > > >> > process
> > > >> > >> > > >> > //         (Table 4.3).
> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
specifying
> the
> > > >> > >> originating
> > > >> > >> > > >> > center.
> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
> specifying
> > > the
> > > >> > >> > ensemble
> > > >> > >> > > >> type
> > > >> > >> > > >> > //         (Table 4.6).
> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
> specifying
> > > the
> > > >> > >> derived
> > > >> > >> > > >> > product
> > > >> > >> > > >> > //         type (Table 4.7).
> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
> > specifying
> > > >> the
> > > >> > >> > > >> statistical
> > > >> > >> > > >> > //         processing type (Table 4.10).
> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> > > "GRIB2_ipdtmpl_val"
> > > >> > >> entries
> > > >> > >> > > are
> > > >> > >> > > >> > arrays
> > > >> > >> > > >> > //         of integers which specify the
product
> > > description
> > > >> > >> > template
> > > >> > >> > > >> > values to
> > > >> > >> > > >> > //         be used. The indices are 0-based.
For
> > example,
> > > >> use
> > > >> > the
> > > >> > >> > > >> > following to
> > > >> > >> > > >> > //         request a GRIB2 record whose 9-th
and 27-th
> > > >> product
> > > >> > >> > > >> description
> > > >> > >> > > >> > //         template values are 1 and 2,
respectively:
> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> > > >> > GRIB2_ipdtmpl_val=[1,
> > > >> > >> 2];
> > > >> > >> > > >> >
> > > >> > >> > > >> > While I don't have any ECMWF data on hand for
> testing, I
> > > do
> > > >> > have
> > > >> > >> a
> > > >> > >> > > file
> > > >> > >> > > >> > from the Air Force from which wgrib2 prints a
similar
> > > >> result:
> > > >> > >> > > >> >
> > > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
center=57
> > > >> > >> local_table=1
> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > > >> > >> > > >> >
> > > >> > >> > > >> > So I used that for testing. I ran the following
> > > >> plot_data_plane
> > > >> > >> > > command
> > > >> > >> > > >> > WITHOUT specifying a requested name:
> > > >> > >> > > >> >
> > > >> > >> > > >> >
> > > >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
plot.ps
> > > >> > >> > 'GRIB2_disc=0;
> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
> > > >> GRIB2_parm=192;
> > > >> > >> > > >> level="Z0";'
> > > >> > >> > > >> > -v 3
> > > >> > >> > > >> >
> > > >> > >> > > >> > And then I looked closely at the log message to
> confirm
> > > >> that it
> > > >> > >> used
> > > >> > >> > > >> > record 549, as expected:
> > > >> > >> > > >> >
> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() ->
Found exact
> > > match
> > > >> > for
> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of GRIB2
file
> > > >> > >> > > >> >
> > > >> > >> > > >>
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > >> > >> > > >> >
> > > >> > >> > > >> > So MET used these GRIB2 numbers I defined to do
a
> table
> > > >> lookup
> > > >> > >> and
> > > >> > >> > > find
> > > >> > >> > > >> a
> > > >> > >> > > >> > match in the MET GRIB tables:
> > > >> > >> > > >> >
> > > >> > >>
> > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > >> > >> > > >> >
> > > >> > >> > > >> > And it determined that that field should be
called
> > > "CTSTM".
> > > >> > >> > Hopefully,
> > > >> > >> > > >> the
> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined there in
MET.
> If
> > > >> not, we
> > > >> > >> can
> > > >> > >> > > >> always
> > > >> > >> > > >> > add them.
> > > >> > >> > > >> >
> > > >> > >> > > >> > Please try running plot_data_plane at verbosity
level
> 3
> > on
> > > >> one
> > > >> > of
> > > >> > >> > > these
> > > >> > >> > > >> > files first. And make sure it runs without
error and
> the
> > > log
> > > >> > >> message
> > > >> > >> > > >> prints
> > > >> > >> > > >> > out the "name" it found for that data. If so,
you can
> > just
> > > >> skip
> > > >> > >> > using
> > > >> > >> > > >> all
> > > >> > >> > > >> > those integers and revert back to using the
name MET
> > > knows.
> > > >> If
> > > >> > >> not,
> > > >> > >> > > >> then we
> > > >> > >> > > >> > may need to define a custom GRIB table for this
ECMWF
> > > GRIB2
> > > >> > data.
> > > >> > >> > > >> >
> > > >> > >> > > >> > Thanks,
> > > >> > >> > > >> > John
> > > >> > >> > > >> >
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >> --
> > > >> > >> > > >>
> > > >> > >> > > >> Binbin Zhou
> > > >> > >> > > >>
> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > >> > >> > > >>
> > > >> > >> > > >> 5830 University Research Ct.
> > > >> > >> > > >>
> > > >> > >> > > >> College Park, MD 20740
> > > >> > >> > > >>
> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > > >> > >> > > >>
> > > >> > >> > > >> 301-683-3683
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >>
> > > >> > >> > > >> --
> > > >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >> > >> > > >> Tara Jensen
> > > >> > >> > > >> Project Manager II
> > > >> > >> > > >> NCAR RAL and DTC
> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > > >> > >> > > >>
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >> > --
> > > >> > >> >
> > > >> > >> > Binbin Zhou
> > > >> > >> >
> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >> > >> >
> > > >> > >> > 5830 University Research Ct.
> > > >> > >> >
> > > >> > >> > College Park, MD 20740
> > > >> > >> >
> > > >> > >> > Binbin.Zhou at noaa.gov
> > > >> > >> >
> > > >> > >> > 301-683-3683
> > > >> > >> >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> > > --
> > > >> > >
> > > >> > > Binbin Zhou
> > > >> > >
> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >> > >
> > > >> > > 5830 University Research Ct.
> > > >> > >
> > > >> > > College Park, MD 20740
> > > >> > >
> > > >> > > Binbin.Zhou at noaa.gov
> > > >> > >
> > > >> > > 301-683-3683
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Binbin Zhou
> > > >> >
> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >> >
> > > >> > 5830 University Research Ct.
> > > >> >
> > > >> > College Park, MD 20740
> > > >> >
> > > >> > Binbin.Zhou at noaa.gov
> > > >> >
> > > >> > 301-683-3683
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > >
> > > > Binbin Zhou
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct.
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Binbin.Zhou at noaa.gov
> > > >
> > > > 301-683-3683
> > > >
> > >
> > >
> > > --
> > >
> > > Binbin Zhou
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct.
> > >
> > > College Park, MD 20740
> > >
> > > Binbin.Zhou at noaa.gov
> > >
> > > 301-683-3683
> > >
> > >
> >
> >
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Wed Dec 09 17:18:11 2020

Julie,

I have created a bugfix branch for main_v9.1 named
"feature_1604_ens_config" and it's up on GitHub.

Only a single file has changed in it. But I need to get this version
compiled on WCOSS with which Binbin can test.

Are you to help me with this? Would it be easier to use git to
checkout the
repo on WCOSS? Or would it be easier if I built a tarfile for this
version
of the code?

Thanks,
John

On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Binbin,
>
> In order for you to test this change out, we'd need to recompile a
version
> of the code on WCOSS. I wrote this up as a GitHub issue:
> https://github.com/dtcenter/MET/issues/1604
>
> And I'll create a branch from main_v9.1 named
"bugfix_1604_ens_config" and
> commit the change there. Then we can checkout that code on WCOSS,
compile
> it, and you can test it.
>
> Yes, you're right, if the fcst fields differ from the climo fields,
you'd
> need to specify them separately. That's pretty easy in the Ensemble-
Stat
> config file... but are you asking how to control that via METplus?
>
> John
>
> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>>
>> John,
>>
>>   Great! Let me test it at first before make change in MET9.1
>> Another question:
>>    Since climatology data is still in NCEP's grib2 format, In the
>> ensemble_stat
>> config file,  climo  entry settings  should be different from those
of
>> fcst, obs entries?
>> How to set climo (name, level, etc.)?
>>
>> Binbin
>>
>>
>>
>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Binbin,
>> >
>> > Success! When I recompile with that one line commented out,
>> plot_data_plane
>> > is able to filter down all those records to the single one that
we want.
>> > Note in the example below that I used the "lead_time" and
"GRIB_ens"
>> > options to filter the input records down.
>> >
>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
>> level="P500";
>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact
match for
>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
>> "ensposte.t00z.z500hr".*
>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
>> > matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
>> > DEBUG 1: Creating postscript file: plot.ps
>> >
>> > Also note that this selects out record number 26.
>> >
>> > Shall I make this change as a "bugfix" for met-9.1...which could
be
>> > included in a met-9.1.2 bugfix release? Or shall I just include
it in
>> > development for the next release (planned for March 2021)?
>> >
>> > Thanks,
>> > John
>> >
>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > >
>> > > John,
>> > >
>> > >   If this removing impacts other applications, or takes some
time
>> > > to be effective, then won't try.  I will use another approach,
such as
>> > > borrow GEFS's grib2 data structure as tank to save EC ensemble
>> > > data.
>> > >
>> > > Binbin
>> > >
>> > >
>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
>> > > binbin.zhou at noaa.gov> wrote:
>> > >
>> > > > John,
>> > > >
>> > > >   Yes, please try to use this approach.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > Binbin
>> > > >
>> > > > Binbin
>> > > >
>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > >> Binbin,
>> > > >>
>> > > >> I used the "-PDS10" option for wgrib to dump those records:
>> > > >>
>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
>> > > >>
>> > > >>
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
>> > > >> | egrep  "^11:|^26:|^43:"
>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>> 0 0
>> > 0
>> > > 21
>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>> 0 0
>> > 0
>> > > 21
>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>> 0 0
>> > 0
>> > > 21
>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
>> > > >>
>> > > >> Looking VERY closely, there is a difference in the 42-nd
column
>> which
>> > is
>> > > >> the 4-th from the last... records 11, 26, and 43 have values
of 3,
>> 1,
>> > > and
>> > > >> 2
>> > > >> respectively.
>> > > >>
>> > > >> On this page (
>> > > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
>> > > >> ),
>> > > >> byte 42 = 1 means "unperturbed control forecast".
>> > > >>
>> > > >> So we *should* be able to use the "GRIB_ens" config option
to
>> > > >> differentiate
>> > > >> between those records. BUT when I try, it doesn't work. And
that's
>> > > because
>> > > >> of this line:
>> > > >>
>> > > >>
>> > >
>> >
>>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
>> > > >>
>> > > >> We only apply those filtering options to GRIB files created
by
>> NCEP.
>> > If
>> > > >> you'd like, I could try removing that condition to see if we
can
>> pick
>> > > out
>> > > >> the single record of interest.
>> > > >>
>> > > >> John
>> > > >>
>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT <
>> > > >> met_help at ucar.edu> wrote:
>> > > >>
>> > > >> >
>> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > > >> >
>> > > >> > John,
>> > > >> >
>> > > >> >   Look at my
>> > > >> > directory
>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is only for
500 HGT.
>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep anl:type=analysis
>> > > >> > I get 3 records:
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >>
>> > >
>> >
>>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > >> > mb:anl:type=analysis:NAve=0
>> > > >> >
>> > > >> >
>> > > >>
>> > >
>> >
>>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > >> > mb:anl:type=analysis:NAve=0
>> > > >> >
>> > > >> >
>> > > >>
>> > >
>> >
>>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > >> > mb:anl:type=analysis:NAve=0
>> > > >> >
>> > > >> > From its grib2 file, we know only  record#26 is analysis
low-res
>> > > >> control.
>> > > >> > If you have some way to determine which record is analysis
>> low-res
>> > > >> control
>> > > >> > from the grib1 file,
>> > > >> > then we can use grib1 directly (don't consider the size
issue).
>> > > >> >
>> > > >> > Binbin
>> > > >> >
>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
Affiliate <
>> > > >> > binbin.zhou at noaa.gov> wrote:
>> > > >> >
>> > > >> > > John,
>> > > >> > >
>> > > >> > >   Grib1 data is too large, another issue is the low-res
control
>> > > >> analysis
>> > > >> > > data can not be retrieved
>> > > >> > > from the grib1 ecmwf data files. If using wgrib, we can
see 3
>> > > analysis
>> > > >> > > fields, but don't know
>> > > >> > > which is low-res control. After convert to grib2, wgrib2
can
>> show
>> > > >> which
>> > > >> > > record is low-res control.
>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on
WCOSS.
>> > > >> > >
>> > > >> > > Binbin
>> > > >> > >
>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via RT
<
>> > > >> > > met_help at ucar.edu> wrote:
>> > > >> > >
>> > > >> > >> Binbin,
>> > > >> > >>
>> > > >> > >> Sounds like there's a lot of moving parts there. How
would you
>> > like
>> > > >> to
>> > > >> > >> proceed?
>> > > >> > >>
>> > > >> > >> Should we check to see if MET can read those ECMWF
GRIB1 files
>> > > >> directly
>> > > >> > >> without needing to reformat?
>> > > >> > >> Or do you just want to create a custom GRIB table for
the
>> result
>> > of
>> > > >> your
>> > > >> > >> pre-processing and conversion steps?
>> > > >> > >>
>> > > >> > >> John
>> > > >> > >>
>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov via
RT <
>> > > >> > >> met_help at ucar.edu> wrote:
>> > > >> > >>
>> > > >> > >> >
>> > > >> > >> > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > >
>> > > >> > >> >
>> > > >> > >> > John,
>> > > >> > >> >
>> > > >> > >> >   Our ECMWF grib2 ensemble data files are converted
from its
>> > > grib1
>> > > >> > >> format
>> > > >> > >> > data.
>> > > >> > >> > Before conversion, the parameter names are there, but
after
>> > > >> > conversion,
>> > > >> > >> the
>> > > >> > >> > names
>> > > >> > >> > are gone. It is because of unknown tables. The grib1
data is
>> > > still
>> > > >> > >> > original. The original
>> > > >> > >> > grib1 data files are very strange data format we can
not
>> read.
>> > So
>> > > >> we
>> > > >> > >> have
>> > > >> > >> > to use
>> > > >> > >> > a Frotran code to re-arrange the fields and save them
as
>> > > NCEP-like
>> > > >> > grib1
>> > > >> > >> > data files.
>> > > >> > >> > So after several processes, some important info like
tables
>> > could
>> > > >> be
>> > > >> > >> lost.
>> > > >> > >> >
>> > > >> > >> >    I'll test your first suggestion since the field
record
>> order
>> > > is
>> > > >> > >> fixed.
>> > > >> > >> > Currently we still
>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we will
get
>> grib2
>> > > data
>> > > >> > >> > directly. So before
>> > > >> > >> > we can get original grib2 files from ECMWF, we just
use this
>> > > >> temporal
>> > > >> > >> > solution.
>> > > >> > >> > '
>> > > >> > >> > Thanks!
>> > > >> > >> >
>> > > >> > >> >
>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway via
RT <
>> > > >> > >> > met_help at ucar.edu>
>> > > >> > >> > wrote:
>> > > >> > >> >
>> > > >> > >> > > Binbin,
>> > > >> > >> > >
>> > > >> > >> > > OK, working on Venus.
>> > > >> > >> > >
>> > > >> > >> > > You have a typo in the command you sent.
>> > > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>> > > >> > >> > >
>> > > >> > >>
>> > > >>
>> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > >> > >> > -v
>> > > >> > >> > > 3
>> > > >> > >> > >
>> > > >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat".
>> > After
>> > > >> > >> > correcting
>> > > >> > >> > > that I did still get an error but it's a very
useful one!
>> > > >> > >> > >
>> > > >> > >> > > module use
>> > > >> > >>
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>> > > >> > >> > > module load met/9.1
>> > > >> > >> > > module load grib_util/1.1.1
>> > > >> > >> > > cd
>> > > >> > >> > >
>> > > >> > >> > >
>> > > >> > >> >
>> > > >> > >>
>> > > >> >
>> > > >>
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
>> > > >> > >> > 'GRIB2_disc=255;
>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
>> level="P500";'
>> > > >> > >> > >
>> > > >> > >> > >
>> > > >> > >> > >
>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
>> parameter
>> > > found
>> > > >> > with
>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm
>> (5).
>> > > Use
>> > > >> the
>> > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom GRIB
>> > > >> > >> tables.ERROR
>> > > >> > >> > :
>> > > >> > >> > > *
>> > > >> > >> > >
>> > > >> > >> > > So this table is not yet defined in MET. I checked
their
>> > > website:
>> > > >> > >> > >
>> > > >> > >> > >
>> > > >> > >> >
>> > > >> > >>
>> > > >> >
>> > > >>
>> > >
>> >
>> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>> > > >> > >> > >
>> > > >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
>> > > >> > >> > >
>> > > >> > >> > > 3 things...
>> > > >> > >> > >
>> > > >> > >> > > (1) As a crude temporary solution, you could refer
to it
>> by
>> > > it's
>> > > >> > >> record
>> > > >> > >> > > number. Let's assume you know that record 1 has
>> temperature
>> > > data:
>> > > >> > >> > >
>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
>> > > >> > 'name="TMP";
>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
>> > > >> > >> > >
>> > > >> > >> > > Here I'm reading record 1, naming it TMP and the
using
>> > > >> > set_attr_level
>> > > >> > >> > write
>> > > >> > >> > > the output level as P500 instead of R1.
>> > > >> > >> > >
>> > > >> > >> > > (2) You could create a custom GRIB table yourself
locally,
>> > and
>> > > >> then
>> > > >> > >> set
>> > > >> > >> > the
>> > > >> > >> > > MET_GRIB_TABLES environment variable to point MET
to it.
>> > > >> > >> > >
>> > > >> > >> > > (3) The best solution is contacting ECMWF and
requesting
>> they
>> > > >> send
>> > > >> > >> over
>> > > >> > >> > the
>> > > >> > >> > > GRIB2 table we should be using to decode this data.
>> > > >> > >> > >
>> > > >> > >> > > Thanks,
>> > > >> > >> > > John
>> > > >> > >> > >
>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway <
>> > > >> johnhg at ucar.edu>
>> > > >> > >> > wrote:
>> > > >> > >> > >
>> > > >> > >> > > > We might already have them. I'm not sure.
>> > > >> > >> > > >
>> > > >> > >> > > > But there's an additional required piece of the
puzzle.
>> > > Binbin
>> > > >> > runs
>> > > >> > >> > > > "wgrib2" to figure out what "name" he should use
in the
>> MET
>> > > >> config
>> > > >> > >> > files.
>> > > >> > >> > > > If wgrib2 doesn't tell him the right "name", the
problem
>> > will
>> > > >> > >> remain.
>> > > >> > >> > > > wgrib2 can be configured to define additional
tables as
>> > well,
>> > > >> but
>> > > >> > >> for
>> > > >> > >> > > this
>> > > >> > >> > > > to be easy for Binbin, that'd mean updating the
script
>> > that's
>> > > >> run
>> > > >> > >> for
>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
>> that's
>> > > not
>> > > >> > very
>> > > >> > >> > easy.
>> > > >> > >> > > >
>> > > >> > >> > > > Yes, I do agree that the best solution is getting
the
>> ECMWF
>> > > >> GRIB
>> > > >> > >> tables
>> > > >> > >> > > > and adding them for both wgrib2 and MET.
>> > > >> > >> > > >
>> > > >> > >> > > > John
>> > > >> > >> > > >
>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
>> > jensen at ucar.edu>
>> > > >> > wrote:
>> > > >> > >> > > >
>> > > >> > >> > > >> John,
>> > > >> > >> > > >>
>> > > >> > >> > > >> It seems to me that it would be worthwhile to
just ask
>> > ECMWF
>> > > >> for
>> > > >> > >> their
>> > > >> > >> > > >> Grib2 Table for their Ensemble files.  And other
>> > operational
>> > > >> > >> centers
>> > > >> > >> > for
>> > > >> > >> > > >> that matter.  Do you agree this would be a good
>> solution?
>> > > >> > >> > > >>
>> > > >> > >> > > >> Thanks, Tara
>> > > >> > >> > > >>
>> > > >> > >> > > >> ---------- Forwarded message ---------
>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT
<met_help at ucar.edu>
>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
question
>> for
>> > > >> > reading
>> > > >> > >> > ECMWF
>> > > >> > >> > > >> ensmeble data
>> > > >> > >> > > >> To:
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >> <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > > >> >
>> > > >> > >> > > >>
>> > > >> > >> > > >> John,
>> > > >> > >> > > >>
>> > > >> > >> > > >>   I tested using following command:
>> > > >> > >> > > >>
>> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>> > > >> > >> > > >>
>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > >> -v3
>> > > >> > >> > > >>
>> > > >> > >> > > >> ERROR  :
>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name
or
>> > > >> GRIB2_disc,
>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
>> field
>> > > >> > >> information
>> > > >> > >> > > >>
>> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>> > > >> > >> > > >>
>> > > >> > >> >
>> > > >> >
>> > >
>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > >> > >> > > >> -v
>> > > >> > >> > > >> 3
>> > > >> > >> > > >>
>> > > >> > >> > > >> Still got error
>> > > >> > >> > > >> ERROR  :
>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either name
or
>> > > >> GRIB2_disc,
>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
>> field
>> > > >> > >> information
>> > > >> > >> > > >>
>> > > >> > >> > > >> Looks like discipline=255 is not recognized
>> > > >> > >> > > >>
>> > > >> > >> > > >> Binbin
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway via
>> RT
>> > <
>> > > >> > >> > > >> met_help at ucar.edu>
>> > > >> > >> > > >> wrote:
>> > > >> > >> > > >>
>> > > >> > >> > > >> > Binbin,
>> > > >> > >> > > >> > You can do something similar with MET. Please
go to
>> this
>> > > >> link:
>> > > >> > >> > > >> >
>> > > >>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and you'll
find the
>> > > >> content
>> > > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD OF
>> > > >> > specifying
>> > > >> > >> a
>> > > >> > >> > > >> > requested name.
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
specifying
>> the
>> > > >> master
>> > > >> > >> > table
>> > > >> > >> > > >> > number.
>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
specifying
>> the
>> > > >> local
>> > > >> > >> table
>> > > >> > >> > > >> > number.
>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
specifying
>> the
>> > > >> GRIB2
>> > > >> > >> > > >> discipline
>> > > >> > >> > > >> > code.
>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
>> specifying
>> > > the
>> > > >> > >> > parameter
>> > > >> > >> > > >> > category
>> > > >> > >> > > >> > //         code.
>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
specifying
>> the
>> > > >> > >> parameter
>> > > >> > >> > > code.
>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
specifying
>> the
>> > > >> product
>> > > >> > >> > > >> definition
>> > > >> > >> > > >> > //         template (Table 4.0).
>> > > >> > >> > > >> > //       - The "GRIB2_process" is an integer
>> specifying
>> > > the
>> > > >> > >> > generating
>> > > >> > >> > > >> > process
>> > > >> > >> > > >> > //         (Table 4.3).
>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
specifying
>> the
>> > > >> > >> originating
>> > > >> > >> > > >> > center.
>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
>> specifying
>> > > the
>> > > >> > >> > ensemble
>> > > >> > >> > > >> type
>> > > >> > >> > > >> > //         (Table 4.6).
>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
>> specifying
>> > > the
>> > > >> > >> derived
>> > > >> > >> > > >> > product
>> > > >> > >> > > >> > //         type (Table 4.7).
>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an integer
>> > specifying
>> > > >> the
>> > > >> > >> > > >> statistical
>> > > >> > >> > > >> > //         processing type (Table 4.10).
>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
>> > > "GRIB2_ipdtmpl_val"
>> > > >> > >> entries
>> > > >> > >> > > are
>> > > >> > >> > > >> > arrays
>> > > >> > >> > > >> > //         of integers which specify the
product
>> > > description
>> > > >> > >> > template
>> > > >> > >> > > >> > values to
>> > > >> > >> > > >> > //         be used. The indices are 0-based.
For
>> > example,
>> > > >> use
>> > > >> > the
>> > > >> > >> > > >> > following to
>> > > >> > >> > > >> > //         request a GRIB2 record whose 9-th
and
>> 27-th
>> > > >> product
>> > > >> > >> > > >> description
>> > > >> > >> > > >> > //         template values are 1 and 2,
respectively:
>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
>> > > >> > GRIB2_ipdtmpl_val=[1,
>> > > >> > >> 2];
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > While I don't have any ECMWF data on hand for
>> testing, I
>> > > do
>> > > >> > have
>> > > >> > >> a
>> > > >> > >> > > file
>> > > >> > >> > > >> > from the Air Force from which wgrib2 prints a
similar
>> > > >> result:
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
center=57
>> > > >> > >> local_table=1
>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > So I used that for testing. I ran the
following
>> > > >> plot_data_plane
>> > > >> > >> > > command
>> > > >> > >> > > >> > WITHOUT specifying a requested name:
>> > > >> > >> > > >> >
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
>> plot.ps
>> > > >> > >> > 'GRIB2_disc=0;
>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1; GRIB2_parm_cat=1;
>> > > >> GRIB2_parm=192;
>> > > >> > >> > > >> level="Z0";'
>> > > >> > >> > > >> > -v 3
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > And then I looked closely at the log message
to
>> confirm
>> > > >> that it
>> > > >> > >> used
>> > > >> > >> > > >> > record 549, as expected:
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() ->
Found
>> exact
>> > > match
>> > > >> > for
>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of
GRIB2 file
>> > > >> > >> > > >> >
>> > > >> > >> > > >>
>> > > >> > >> > >
>> > > >> > >> >
>> > > >> > >>
>> > > >> >
>> > > >>
>> > >
>> >
>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > So MET used these GRIB2 numbers I defined to
do a
>> table
>> > > >> lookup
>> > > >> > >> and
>> > > >> > >> > > find
>> > > >> > >> > > >> a
>> > > >> > >> > > >> > match in the MET GRIB tables:
>> > > >> > >> > > >> >
>> > > >> > >>
>> > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > And it determined that that field should be
called
>> > > "CTSTM".
>> > > >> > >> > Hopefully,
>> > > >> > >> > > >> the
>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined there
in MET.
>> If
>> > > >> not, we
>> > > >> > >> can
>> > > >> > >> > > >> always
>> > > >> > >> > > >> > add them.
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > Please try running plot_data_plane at
verbosity
>> level 3
>> > on
>> > > >> one
>> > > >> > of
>> > > >> > >> > > these
>> > > >> > >> > > >> > files first. And make sure it runs without
error and
>> the
>> > > log
>> > > >> > >> message
>> > > >> > >> > > >> prints
>> > > >> > >> > > >> > out the "name" it found for that data. If so,
you can
>> > just
>> > > >> skip
>> > > >> > >> > using
>> > > >> > >> > > >> all
>> > > >> > >> > > >> > those integers and revert back to using the
name MET
>> > > knows.
>> > > >> If
>> > > >> > >> not,
>> > > >> > >> > > >> then we
>> > > >> > >> > > >> > may need to define a custom GRIB table for
this ECMWF
>> > > GRIB2
>> > > >> > data.
>> > > >> > >> > > >> >
>> > > >> > >> > > >> > Thanks,
>> > > >> > >> > > >> > John
>> > > >> > >> > > >> >
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >> --
>> > > >> > >> > > >>
>> > > >> > >> > > >> Binbin Zhou
>> > > >> > >> > > >>
>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
>> > > >> > >> > > >>
>> > > >> > >> > > >> 5830 University Research Ct.
>> > > >> > >> > > >>
>> > > >> > >> > > >> College Park, MD 20740
>> > > >> > >> > > >>
>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
>> > > >> > >> > > >>
>> > > >> > >> > > >> 301-683-3683
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >>
>> > > >> > >> > > >> --
>> > > >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > > >> > >> > > >> Tara Jensen
>> > > >> > >> > > >> Project Manager II
>> > > >> > >> > > >> NCAR RAL and DTC
>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
>> > > >> > >> > > >>
>> > > >> > >> > > >
>> > > >> > >> > >
>> > > >> > >> > >
>> > > >> > >> >
>> > > >> > >> > --
>> > > >> > >> >
>> > > >> > >> > Binbin Zhou
>> > > >> > >> >
>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > > >> > >> >
>> > > >> > >> > 5830 University Research Ct.
>> > > >> > >> >
>> > > >> > >> > College Park, MD 20740
>> > > >> > >> >
>> > > >> > >> > Binbin.Zhou at noaa.gov
>> > > >> > >> >
>> > > >> > >> > 301-683-3683
>> > > >> > >> >
>> > > >> > >> >
>> > > >> > >>
>> > > >> > >>
>> > > >> > >
>> > > >> > > --
>> > > >> > >
>> > > >> > > Binbin Zhou
>> > > >> > >
>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
>> > > >> > >
>> > > >> > > 5830 University Research Ct.
>> > > >> > >
>> > > >> > > College Park, MD 20740
>> > > >> > >
>> > > >> > > Binbin.Zhou at noaa.gov
>> > > >> > >
>> > > >> > > 301-683-3683
>> > > >> > >
>> > > >> >
>> > > >> >
>> > > >> > --
>> > > >> >
>> > > >> > Binbin Zhou
>> > > >> >
>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > > >> >
>> > > >> > 5830 University Research Ct.
>> > > >> >
>> > > >> > College Park, MD 20740
>> > > >> >
>> > > >> > Binbin.Zhou at noaa.gov
>> > > >> >
>> > > >> > 301-683-3683
>> > > >> >
>> > > >> >
>> > > >>
>> > > >>
>> > > >
>> > > > --
>> > > >
>> > > > Binbin Zhou
>> > > >
>> > > > IMSG at NOAA/NWS/NCEP/EMC
>> > > >
>> > > > 5830 University Research Ct.
>> > > >
>> > > > College Park, MD 20740
>> > > >
>> > > > Binbin.Zhou at noaa.gov
>> > > >
>> > > > 301-683-3683
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Binbin Zhou
>> > >
>> > > IMSG at NOAA/NWS/NCEP/EMC
>> > >
>> > > 5830 University Research Ct.
>> > >
>> > > College Park, MD 20740
>> > >
>> > > Binbin.Zhou at noaa.gov
>> > >
>> > > 301-683-3683
>> > >
>> > >
>> >
>> >
>>
>> --
>>
>> Binbin Zhou
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct.
>>
>> College Park, MD 20740
>>
>> Binbin.Zhou at noaa.gov
>>
>> 301-683-3683
>>
>>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Thu Dec 10 06:25:18 2020

John,

   Sure, I'll test the testing version of MET_9.1 after it is
available.
   First step is just MET, not METplus. Could you please provide
an example of how to specify the climo entry in ensemble_stat
config file?

Thanks!




On Wed, Dec 9, 2020 at 6:53 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Binbin,
>
> In order for you to test this change out, we'd need to recompile a
version
> of the code on WCOSS. I wrote this up as a GitHub issue:
> https://github.com/dtcenter/MET/issues/1604
>
> And I'll create a branch from main_v9.1 named
"bugfix_1604_ens_config" and
> commit the change there. Then we can checkout that code on WCOSS,
compile
> it, and you can test it.
>
> Yes, you're right, if the fcst fields differ from the climo fields,
you'd
> need to specify them separately. That's pretty easy in the Ensemble-
Stat
> config file... but are you asking how to control that via METplus?
>
> John
>
> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > John,
> >
> >   Great! Let me test it at first before make change in MET9.1
> > Another question:
> >    Since climatology data is still in NCEP's grib2 format, In the
> > ensemble_stat
> > config file,  climo  entry settings  should be different from
those of
> > fcst, obs entries?
> > How to set climo (name, level, etc.)?
> >
> > Binbin
> >
> >
> >
> > On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binbin,
> > >
> > > Success! When I recompile with that one line commented out,
> > plot_data_plane
> > > is able to filter down all those records to the single one that
we
> want.
> > > Note in the example below that I used the "lead_time" and
"GRIB_ens"
> > > options to filter the input records down.
> > >
> > > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
> level="P500";
> > > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact
match for
> > > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> > "ensposte.t00z.z500hr".*
> > > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> > > matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
> > > DEBUG 1: Creating postscript file: plot.ps
> > >
> > > Also note that this selects out record number 26.
> > >
> > > Shall I make this change as a "bugfix" for met-9.1...which could
be
> > > included in a met-9.1.2 bugfix release? Or shall I just include
it in
> > > development for the next release (planned for March 2021)?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > > >
> > > > John,
> > > >
> > > >   If this removing impacts other applications, or takes some
time
> > > > to be effective, then won't try.  I will use another approach,
such
> as
> > > > borrow GEFS's grib2 data structure as tank to save EC ensemble
> > > > data.
> > > >
> > > > Binbin
> > > >
> > > >
> > > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
> > > > binbin.zhou at noaa.gov> wrote:
> > > >
> > > > > John,
> > > > >
> > > > >   Yes, please try to use this approach.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Binbin
> > > > >
> > > > > Binbin
> > > > >
> > > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Binbin,
> > > > >>
> > > > >> I used the "-PDS10" option for wgrib to dump those records:
> > > > >>
> > > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > > > >>
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > > > >> | egrep  "^11:|^26:|^43:"
> > > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
> 0
> > 0
> > > 0
> > > > 21
> > > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
> 0
> > 0
> > > 0
> > > > 21
> > > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
> 0
> > 0
> > > 0
> > > > 21
> > > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > > > >>
> > > > >> Looking VERY closely, there is a difference in the 42-nd
column
> > which
> > > is
> > > > >> the 4-th from the last... records 11, 26, and 43 have
values of 3,
> > 1,
> > > > and
> > > > >> 2
> > > > >> respectively.
> > > > >>
> > > > >> On this page (
> > > > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > > > >> ),
> > > > >> byte 42 = 1 means "unperturbed control forecast".
> > > > >>
> > > > >> So we *should* be able to use the "GRIB_ens" config option
to
> > > > >> differentiate
> > > > >> between those records. BUT when I try, it doesn't work. And
that's
> > > > because
> > > > >> of this line:
> > > > >>
> > > > >>
> > > >
> > >
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > > > >>
> > > > >> We only apply those filtering options to GRIB files created
by
> NCEP.
> > > If
> > > > >> you'd like, I could try removing that condition to see if
we can
> > pick
> > > > out
> > > > >> the single record of interest.
> > > > >>
> > > > >> John
> > > > >>
> > > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT
<
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > > > >> >
> > > > >> > John,
> > > > >> >
> > > > >> >   Look at my
> > > > >> > directory
> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > > > >> > The grib1 file is ensposte.t00z.z500hr, this is only for
500
> HGT.
> > > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
> > > > >> > I get 3 records:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >> > mb:anl:type=analysis:NAve=0
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >> > mb:anl:type=analysis:NAve=0
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >> > mb:anl:type=analysis:NAve=0
> > > > >> >
> > > > >> > From its grib2 file, we know only  record#26 is analysis
low-res
> > > > >> control.
> > > > >> > If you have some way to determine which record is
analysis
> low-res
> > > > >> control
> > > > >> > from the grib1 file,
> > > > >> > then we can use grib1 directly (don't consider the size
issue).
> > > > >> >
> > > > >> > Binbin
> > > > >> >
> > > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
Affiliate <
> > > > >> > binbin.zhou at noaa.gov> wrote:
> > > > >> >
> > > > >> > > John,
> > > > >> > >
> > > > >> > >   Grib1 data is too large, another issue is the low-res
> control
> > > > >> analysis
> > > > >> > > data can not be retrieved
> > > > >> > > from the grib1 ecmwf data files. If using wgrib, we can
see 3
> > > > analysis
> > > > >> > > fields, but don't know
> > > > >> > > which is low-res control. After convert to grib2,
wgrib2 can
> > show
> > > > >> which
> > > > >> > > record is low-res control.
> > > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on
WCOSS.
> > > > >> > >
> > > > >> > > Binbin
> > > > >> > >
> > > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via
RT <
> > > > >> > > met_help at ucar.edu> wrote:
> > > > >> > >
> > > > >> > >> Binbin,
> > > > >> > >>
> > > > >> > >> Sounds like there's a lot of moving parts there. How
would
> you
> > > like
> > > > >> to
> > > > >> > >> proceed?
> > > > >> > >>
> > > > >> > >> Should we check to see if MET can read those ECMWF
GRIB1
> files
> > > > >> directly
> > > > >> > >> without needing to reformat?
> > > > >> > >> Or do you just want to create a custom GRIB table for
the
> > result
> > > of
> > > > >> your
> > > > >> > >> pre-processing and conversion steps?
> > > > >> > >>
> > > > >> > >> John
> > > > >> > >>
> > > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov
via RT <
> > > > >> > >> met_help at ucar.edu> wrote:
> > > > >> > >>
> > > > >> > >> >
> > > > >> > >> > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >
> > > > >> > >> >
> > > > >> > >> > John,
> > > > >> > >> >
> > > > >> > >> >   Our ECMWF grib2 ensemble data files are converted
from
> its
> > > > grib1
> > > > >> > >> format
> > > > >> > >> > data.
> > > > >> > >> > Before conversion, the parameter names are there,
but after
> > > > >> > conversion,
> > > > >> > >> the
> > > > >> > >> > names
> > > > >> > >> > are gone. It is because of unknown tables. The grib1
data
> is
> > > > still
> > > > >> > >> > original. The original
> > > > >> > >> > grib1 data files are very strange data format we can
not
> > read.
> > > So
> > > > >> we
> > > > >> > >> have
> > > > >> > >> > to use
> > > > >> > >> > a Frotran code to re-arrange the fields and save
them as
> > > > NCEP-like
> > > > >> > grib1
> > > > >> > >> > data files.
> > > > >> > >> > So after several processes, some important info like
tables
> > > could
> > > > >> be
> > > > >> > >> lost.
> > > > >> > >> >
> > > > >> > >> >    I'll test your first suggestion since the field
record
> > order
> > > > is
> > > > >> > >> fixed.
> > > > >> > >> > Currently we still
> > > > >> > >> > obtained grib1 data from ECWMF, eventually, we will
get
> grib2
> > > > data
> > > > >> > >> > directly. So before
> > > > >> > >> > we can get original grib2 files from ECMWF, we just
use
> this
> > > > >> temporal
> > > > >> > >> > solution.
> > > > >> > >> > '
> > > > >> > >> > Thanks!
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway
via RT <
> > > > >> > >> > met_help at ucar.edu>
> > > > >> > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > > Binbin,
> > > > >> > >> > >
> > > > >> > >> > > OK, working on Venus.
> > > > >> > >> > >
> > > > >> > >> > > You have a typo in the command you sent.
> > > > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > > > >> > >> > >
> > > > >> > >>
> > > > >>
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >> > >> > -v
> > > > >> > >> > > 3
> > > > >> > >> > >
> > > > >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat".
> > > After
> > > > >> > >> > correcting
> > > > >> > >> > > that I did still get an error but it's a very
useful one!
> > > > >> > >> > >
> > > > >> > >> > > module use
> > > > >> > >>
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > > >> > >> > > module load met/9.1
> > > > >> > >> > > module load grib_util/1.1.1
> > > > >> > >> > > cd
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > > > >> > >> > 'GRIB2_disc=255;
> > > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
> > level="P500";'
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
> parameter
> > > > found
> > > > >> > with
> > > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm
> > (5).
> > > > Use
> > > > >> the
> > > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom
> GRIB
> > > > >> > >> tables.ERROR
> > > > >> > >> > :
> > > > >> > >> > > *
> > > > >> > >> > >
> > > > >> > >> > > So this table is not yet defined in MET. I checked
their
> > > > website:
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > > > >> > >> > >
> > > > >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
> > > > >> > >> > >
> > > > >> > >> > > 3 things...
> > > > >> > >> > >
> > > > >> > >> > > (1) As a crude temporary solution, you could refer
to it
> by
> > > > it's
> > > > >> > >> record
> > > > >> > >> > > number. Let's assume you know that record 1 has
> temperature
> > > > data:
> > > > >> > >> > >
> > > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
> > > > >> > 'name="TMP";
> > > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > > > >> > >> > >
> > > > >> > >> > > Here I'm reading record 1, naming it TMP and the
using
> > > > >> > set_attr_level
> > > > >> > >> > write
> > > > >> > >> > > the output level as P500 instead of R1.
> > > > >> > >> > >
> > > > >> > >> > > (2) You could create a custom GRIB table yourself
> locally,
> > > and
> > > > >> then
> > > > >> > >> set
> > > > >> > >> > the
> > > > >> > >> > > MET_GRIB_TABLES environment variable to point MET
to it.
> > > > >> > >> > >
> > > > >> > >> > > (3) The best solution is contacting ECMWF and
requesting
> > they
> > > > >> send
> > > > >> > >> over
> > > > >> > >> > the
> > > > >> > >> > > GRIB2 table we should be using to decode this
data.
> > > > >> > >> > >
> > > > >> > >> > > Thanks,
> > > > >> > >> > > John
> > > > >> > >> > >
> > > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<
> > > > >> johnhg at ucar.edu>
> > > > >> > >> > wrote:
> > > > >> > >> > >
> > > > >> > >> > > > We might already have them. I'm not sure.
> > > > >> > >> > > >
> > > > >> > >> > > > But there's an additional required piece of the
puzzle.
> > > > Binbin
> > > > >> > runs
> > > > >> > >> > > > "wgrib2" to figure out what "name" he should use
in the
> > MET
> > > > >> config
> > > > >> > >> > files.
> > > > >> > >> > > > If wgrib2 doesn't tell him the right "name", the
> problem
> > > will
> > > > >> > >> remain.
> > > > >> > >> > > > wgrib2 can be configured to define additional
tables as
> > > well,
> > > > >> but
> > > > >> > >> for
> > > > >> > >> > > this
> > > > >> > >> > > > to be easy for Binbin, that'd mean updating the
script
> > > that's
> > > > >> run
> > > > >> > >> for
> > > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
> > that's
> > > > not
> > > > >> > very
> > > > >> > >> > easy.
> > > > >> > >> > > >
> > > > >> > >> > > > Yes, I do agree that the best solution is
getting the
> > ECMWF
> > > > >> GRIB
> > > > >> > >> tables
> > > > >> > >> > > > and adding them for both wgrib2 and MET.
> > > > >> > >> > > >
> > > > >> > >> > > > John
> > > > >> > >> > > >
> > > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
> > > jensen at ucar.edu>
> > > > >> > wrote:
> > > > >> > >> > > >
> > > > >> > >> > > >> John,
> > > > >> > >> > > >>
> > > > >> > >> > > >> It seems to me that it would be worthwhile to
just ask
> > > ECMWF
> > > > >> for
> > > > >> > >> their
> > > > >> > >> > > >> Grib2 Table for their Ensemble files.  And
other
> > > operational
> > > > >> > >> centers
> > > > >> > >> > for
> > > > >> > >> > > >> that matter.  Do you agree this would be a good
> > solution?
> > > > >> > >> > > >>
> > > > >> > >> > > >> Thanks, Tara
> > > > >> > >> > > >>
> > > > >> > >> > > >> ---------- Forwarded message ---------
> > > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT
<met_help at ucar.edu>
> > > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
question
> > for
> > > > >> > reading
> > > > >> > >> > ECMWF
> > > > >> > >> > > >> ensmeble data
> > > > >> > >> > > >> To:
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >> <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > > >> >
> > > > >> > >> > > >>
> > > > >> > >> > > >> John,
> > > > >> > >> > > >>
> > > > >> > >> > > >>   I tested using following command:
> > > > >> > >> > > >>
> > > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > > > >> > >> > > >>
> > 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >> -v3
> > > > >> > >> > > >>
> > > > >> > >> > > >> ERROR  :
> > > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
> > > > >> GRIB2_disc,
> > > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
> field
> > > > >> > >> information
> > > > >> > >> > > >>
> > > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> > > > >> > >> > > >>
> > > > >> > >> >
> > > > >> >
> > > >
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >> > >> > > >> -v
> > > > >> > >> > > >> 3
> > > > >> > >> > > >>
> > > > >> > >> > > >> Still got error
> > > > >> > >> > > >> ERROR  :
> > > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
> > > > >> GRIB2_disc,
> > > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
> field
> > > > >> > >> information
> > > > >> > >> > > >>
> > > > >> > >> > > >> Looks like discipline=255 is not recognized
> > > > >> > >> > > >>
> > > > >> > >> > > >> Binbin
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway via
> > RT
> > > <
> > > > >> > >> > > >> met_help at ucar.edu>
> > > > >> > >> > > >> wrote:
> > > > >> > >> > > >>
> > > > >> > >> > > >> > Binbin,
> > > > >> > >> > > >> > You can do something similar with MET. Please
go to
> > this
> > > > >> link:
> > > > >> > >> > > >> >
> > > > >>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > And then search for "GRIB2_mtab", and you'll
find
> the
> > > > >> content
> > > > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD
> OF
> > > > >> > specifying
> > > > >> > >> a
> > > > >> > >> > > >> > requested name.
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
specifying
> > the
> > > > >> master
> > > > >> > >> > table
> > > > >> > >> > > >> > number.
> > > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
specifying
> > the
> > > > >> local
> > > > >> > >> table
> > > > >> > >> > > >> > number.
> > > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
specifying
> > the
> > > > >> GRIB2
> > > > >> > >> > > >> discipline
> > > > >> > >> > > >> > code.
> > > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
> > specifying
> > > > the
> > > > >> > >> > parameter
> > > > >> > >> > > >> > category
> > > > >> > >> > > >> > //         code.
> > > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
specifying
> > the
> > > > >> > >> parameter
> > > > >> > >> > > code.
> > > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
specifying
> > the
> > > > >> product
> > > > >> > >> > > >> definition
> > > > >> > >> > > >> > //         template (Table 4.0).
> > > > >> > >> > > >> > //       - The "GRIB2_process" is an integer
> > specifying
> > > > the
> > > > >> > >> > generating
> > > > >> > >> > > >> > process
> > > > >> > >> > > >> > //         (Table 4.3).
> > > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
specifying
> > the
> > > > >> > >> originating
> > > > >> > >> > > >> > center.
> > > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
> > specifying
> > > > the
> > > > >> > >> > ensemble
> > > > >> > >> > > >> type
> > > > >> > >> > > >> > //         (Table 4.6).
> > > > >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
> > specifying
> > > > the
> > > > >> > >> derived
> > > > >> > >> > > >> > product
> > > > >> > >> > > >> > //         type (Table 4.7).
> > > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
> > > specifying
> > > > >> the
> > > > >> > >> > > >> statistical
> > > > >> > >> > > >> > //         processing type (Table 4.10).
> > > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> > > > "GRIB2_ipdtmpl_val"
> > > > >> > >> entries
> > > > >> > >> > > are
> > > > >> > >> > > >> > arrays
> > > > >> > >> > > >> > //         of integers which specify the
product
> > > > description
> > > > >> > >> > template
> > > > >> > >> > > >> > values to
> > > > >> > >> > > >> > //         be used. The indices are 0-based.
For
> > > example,
> > > > >> use
> > > > >> > the
> > > > >> > >> > > >> > following to
> > > > >> > >> > > >> > //         request a GRIB2 record whose 9-th
and
> 27-th
> > > > >> product
> > > > >> > >> > > >> description
> > > > >> > >> > > >> > //         template values are 1 and 2,
> respectively:
> > > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> > > > >> > GRIB2_ipdtmpl_val=[1,
> > > > >> > >> 2];
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > While I don't have any ECMWF data on hand for
> > testing, I
> > > > do
> > > > >> > have
> > > > >> > >> a
> > > > >> > >> > > file
> > > > >> > >> > > >> > from the Air Force from which wgrib2 prints a
> similar
> > > > >> result:
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
> center=57
> > > > >> > >> local_table=1
> > > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > So I used that for testing. I ran the
following
> > > > >> plot_data_plane
> > > > >> > >> > > command
> > > > >> > >> > > >> > WITHOUT specifying a requested name:
> > > > >> > >> > > >> >
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
> plot.ps
> > > > >> > >> > 'GRIB2_disc=0;
> > > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
> > > > >> GRIB2_parm=192;
> > > > >> > >> > > >> level="Z0";'
> > > > >> > >> > > >> > -v 3
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > And then I looked closely at the log message
to
> > confirm
> > > > >> that it
> > > > >> > >> used
> > > > >> > >> > > >> > record 549, as expected:
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() ->
Found
> exact
> > > > match
> > > > >> > for
> > > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of
GRIB2 file
> > > > >> > >> > > >> >
> > > > >> > >> > > >>
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > So MET used these GRIB2 numbers I defined to
do a
> > table
> > > > >> lookup
> > > > >> > >> and
> > > > >> > >> > > find
> > > > >> > >> > > >> a
> > > > >> > >> > > >> > match in the MET GRIB tables:
> > > > >> > >> > > >> >
> > > > >> > >>
> > > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > And it determined that that field should be
called
> > > > "CTSTM".
> > > > >> > >> > Hopefully,
> > > > >> > >> > > >> the
> > > > >> > >> > > >> > ECMWF GRIB2 tables are already defined there
in MET.
> > If
> > > > >> not, we
> > > > >> > >> can
> > > > >> > >> > > >> always
> > > > >> > >> > > >> > add them.
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > Please try running plot_data_plane at
verbosity
> level
> > 3
> > > on
> > > > >> one
> > > > >> > of
> > > > >> > >> > > these
> > > > >> > >> > > >> > files first. And make sure it runs without
error and
> > the
> > > > log
> > > > >> > >> message
> > > > >> > >> > > >> prints
> > > > >> > >> > > >> > out the "name" it found for that data. If so,
you
> can
> > > just
> > > > >> skip
> > > > >> > >> > using
> > > > >> > >> > > >> all
> > > > >> > >> > > >> > those integers and revert back to using the
name MET
> > > > knows.
> > > > >> If
> > > > >> > >> not,
> > > > >> > >> > > >> then we
> > > > >> > >> > > >> > may need to define a custom GRIB table for
this
> ECMWF
> > > > GRIB2
> > > > >> > data.
> > > > >> > >> > > >> >
> > > > >> > >> > > >> > Thanks,
> > > > >> > >> > > >> > John
> > > > >> > >> > > >> >
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >> --
> > > > >> > >> > > >>
> > > > >> > >> > > >> Binbin Zhou
> > > > >> > >> > > >>
> > > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > >> > >> > > >>
> > > > >> > >> > > >> 5830 University Research Ct.
> > > > >> > >> > > >>
> > > > >> > >> > > >> College Park, MD 20740
> > > > >> > >> > > >>
> > > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > > > >> > >> > > >>
> > > > >> > >> > > >> 301-683-3683
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >>
> > > > >> > >> > > >> --
> > > > >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > >> > >> > > >> Tara Jensen
> > > > >> > >> > > >> Project Manager II
> > > > >> > >> > > >> NCAR RAL and DTC
> > > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > > > >> > >> > > >>
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >> > --
> > > > >> > >> >
> > > > >> > >> > Binbin Zhou
> > > > >> > >> >
> > > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >> > >> >
> > > > >> > >> > 5830 University Research Ct.
> > > > >> > >> >
> > > > >> > >> > College Park, MD 20740
> > > > >> > >> >
> > > > >> > >> > Binbin.Zhou at noaa.gov
> > > > >> > >> >
> > > > >> > >> > 301-683-3683
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> > > --
> > > > >> > >
> > > > >> > > Binbin Zhou
> > > > >> > >
> > > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >> > >
> > > > >> > > 5830 University Research Ct.
> > > > >> > >
> > > > >> > > College Park, MD 20740
> > > > >> > >
> > > > >> > > Binbin.Zhou at noaa.gov
> > > > >> > >
> > > > >> > > 301-683-3683
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > Binbin Zhou
> > > > >> >
> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >> >
> > > > >> > 5830 University Research Ct.
> > > > >> >
> > > > >> > College Park, MD 20740
> > > > >> >
> > > > >> > Binbin.Zhou at noaa.gov
> > > > >> >
> > > > >> > 301-683-3683
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > >
> > > > > Binbin Zhou
> > > > >
> > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >
> > > > > 5830 University Research Ct.
> > > > >
> > > > > College Park, MD 20740
> > > > >
> > > > > Binbin.Zhou at noaa.gov
> > > > >
> > > > > 301-683-3683
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Binbin Zhou
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct.
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Binbin.Zhou at noaa.gov
> > > >
> > > > 301-683-3683
> > > >
> > > >
> > >
> > >
> >
> > --
> >
> > Binbin Zhou
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct.
> >
> > College Park, MD 20740
> >
> > Binbin.Zhou at noaa.gov
> >
> > 301-683-3683
> >
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Julie Prestopnik
Time: Thu Dec 10 09:24:29 2020

John,

I am able to help with this install on WCOSS.

Would it be easier to use git to checkout the repo on WCOSS? Or would
it be
> easier if I built a tarfile for this version of the code?
>
The last time I tried to build from git on WCOSS (a while ago), I
couldn't.  I don't remember all of the details, but I think I couldn't
run
bootstrap and maybe it had to do with the version of autoconf being
too
old?  I'm happy to try it again if you'd like though.  Otherwise, I
know a
tarfile would work, but I know that is extra work for you.

Julie

On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Julie,
>
> I have created a bugfix branch for main_v9.1 named
> "feature_1604_ens_config" and it's up on GitHub.
>
> Only a single file has changed in it. But I need to get this version
> compiled on WCOSS with which Binbin can test.
>
> Are you to help me with this? Would it be easier to use git to
checkout
> the repo on WCOSS? Or would it be easier if I built a tarfile for
this
> version of the code?
>
> Thanks,
> John
>
> On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Binbin,
>>
>> In order for you to test this change out, we'd need to recompile a
>> version of the code on WCOSS. I wrote this up as a GitHub issue:
>> https://github.com/dtcenter/MET/issues/1604
>>
>> And I'll create a branch from main_v9.1 named
"bugfix_1604_ens_config"
>> and commit the change there. Then we can checkout that code on
WCOSS,
>> compile it, and you can test it.
>>
>> Yes, you're right, if the fcst fields differ from the climo fields,
you'd
>> need to specify them separately. That's pretty easy in the
Ensemble-Stat
>> config file... but are you asking how to control that via METplus?
>>
>> John
>>
>> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>>>
>>> John,
>>>
>>>   Great! Let me test it at first before make change in MET9.1
>>> Another question:
>>>    Since climatology data is still in NCEP's grib2 format, In the
>>> ensemble_stat
>>> config file,  climo  entry settings  should be different from
those of
>>> fcst, obs entries?
>>> How to set climo (name, level, etc.)?
>>>
>>> Binbin
>>>
>>>
>>>
>>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> > Binbin,
>>> >
>>> > Success! When I recompile with that one line commented out,
>>> plot_data_plane
>>> > is able to filter down all those records to the single one that
we
>>> want.
>>> > Note in the example below that I used the "lead_time" and
"GRIB_ens"
>>> > options to filter the input records down.
>>> >
>>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
>>> level="P500";
>>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
>>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
>>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact
match for
>>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
>>> "ensposte.t00z.z500hr".*
>>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
>>> > matching VarInfo "HGT/P500" in GRIB file "ensposte.t00z.z500hr".
>>> > DEBUG 1: Creating postscript file: plot.ps
>>> >
>>> > Also note that this selects out record number 26.
>>> >
>>> > Shall I make this change as a "bugfix" for met-9.1...which could
be
>>> > included in a met-9.1.2 bugfix release? Or shall I just include
it in
>>> > development for the next release (planned for March 2021)?
>>> >
>>> > Thanks,
>>> > John
>>> >
>>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
>>> > met_help at ucar.edu> wrote:
>>> >
>>> > >
>>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
>>> > >
>>> > > John,
>>> > >
>>> > >   If this removing impacts other applications, or takes some
time
>>> > > to be effective, then won't try.  I will use another approach,
such
>>> as
>>> > > borrow GEFS's grib2 data structure as tank to save EC ensemble
>>> > > data.
>>> > >
>>> > > Binbin
>>> > >
>>> > >
>>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate <
>>> > > binbin.zhou at noaa.gov> wrote:
>>> > >
>>> > > > John,
>>> > > >
>>> > > >   Yes, please try to use this approach.
>>> > > >
>>> > > > Thanks!
>>> > > >
>>> > > > Binbin
>>> > > >
>>> > > > Binbin
>>> > > >
>>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
>>> > > > met_help at ucar.edu> wrote:
>>> > > >
>>> > > >> Binbin,
>>> > > >>
>>> > > >> I used the "-PDS10" option for wgrib to dump those records:
>>> > > >>
>>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
>>> > > >>
>>> > > >>
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
>>> > > >> | egrep  "^11:|^26:|^43:"
>>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>>> 0 0
>>> > 0
>>> > > 21
>>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
>>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>>> 0 0
>>> > 0
>>> > > 21
>>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
>>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0 1
0 0 1
>>> 0 0
>>> > 0
>>> > > 21
>>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
>>> > > >>
>>> > > >> Looking VERY closely, there is a difference in the 42-nd
column
>>> which
>>> > is
>>> > > >> the 4-th from the last... records 11, 26, and 43 have
values of
>>> 3, 1,
>>> > > and
>>> > > >> 2
>>> > > >> respectively.
>>> > > >>
>>> > > >> On this page (
>>> > > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
>>> > > >> ),
>>> > > >> byte 42 = 1 means "unperturbed control forecast".
>>> > > >>
>>> > > >> So we *should* be able to use the "GRIB_ens" config option
to
>>> > > >> differentiate
>>> > > >> between those records. BUT when I try, it doesn't work. And
that's
>>> > > because
>>> > > >> of this line:
>>> > > >>
>>> > > >>
>>> > >
>>> >
>>>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
>>> > > >>
>>> > > >> We only apply those filtering options to GRIB files created
by
>>> NCEP.
>>> > If
>>> > > >> you'd like, I could try removing that condition to see if
we can
>>> pick
>>> > > out
>>> > > >> the single record of interest.
>>> > > >>
>>> > > >> John
>>> > > >>
>>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via RT
<
>>> > > >> met_help at ucar.edu> wrote:
>>> > > >>
>>> > > >> >
>>> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>>> > > >> >
>>> > > >> > John,
>>> > > >> >
>>> > > >> >   Look at my
>>> > > >> > directory
>>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
>>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is only for
500
>>> HGT.
>>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
>>> > > >> > I get 3 records:
>>> > > >> >
>>> > > >> >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>>> > > >> > mb:anl:type=analysis:NAve=0
>>> > > >> >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>>> > > >> > mb:anl:type=analysis:NAve=0
>>> > > >> >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>>> > > >> > mb:anl:type=analysis:NAve=0
>>> > > >> >
>>> > > >> > From its grib2 file, we know only  record#26 is analysis
low-res
>>> > > >> control.
>>> > > >> > If you have some way to determine which record is
analysis
>>> low-res
>>> > > >> control
>>> > > >> > from the grib1 file,
>>> > > >> > then we can use grib1 directly (don't consider the size
issue).
>>> > > >> >
>>> > > >> > Binbin
>>> > > >> >
>>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
Affiliate <
>>> > > >> > binbin.zhou at noaa.gov> wrote:
>>> > > >> >
>>> > > >> > > John,
>>> > > >> > >
>>> > > >> > >   Grib1 data is too large, another issue is the low-res
>>> control
>>> > > >> analysis
>>> > > >> > > data can not be retrieved
>>> > > >> > > from the grib1 ecmwf data files. If using wgrib, we can
see 3
>>> > > analysis
>>> > > >> > > fields, but don't know
>>> > > >> > > which is low-res control. After convert to grib2,
wgrib2 can
>>> show
>>> > > >> which
>>> > > >> > > record is low-res control.
>>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on
WCOSS.
>>> > > >> > >
>>> > > >> > > Binbin
>>> > > >> > >
>>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via
RT <
>>> > > >> > > met_help at ucar.edu> wrote:
>>> > > >> > >
>>> > > >> > >> Binbin,
>>> > > >> > >>
>>> > > >> > >> Sounds like there's a lot of moving parts there. How
would
>>> you
>>> > like
>>> > > >> to
>>> > > >> > >> proceed?
>>> > > >> > >>
>>> > > >> > >> Should we check to see if MET can read those ECMWF
GRIB1
>>> files
>>> > > >> directly
>>> > > >> > >> without needing to reformat?
>>> > > >> > >> Or do you just want to create a custom GRIB table for
the
>>> result
>>> > of
>>> > > >> your
>>> > > >> > >> pre-processing and conversion steps?
>>> > > >> > >>
>>> > > >> > >> John
>>> > > >> > >>
>>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov
via RT <
>>> > > >> > >> met_help at ucar.edu> wrote:
>>> > > >> > >>
>>> > > >> > >> >
>>> > > >> > >> > <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>>> > >
>>> > > >> > >> >
>>> > > >> > >> > John,
>>> > > >> > >> >
>>> > > >> > >> >   Our ECMWF grib2 ensemble data files are converted
from
>>> its
>>> > > grib1
>>> > > >> > >> format
>>> > > >> > >> > data.
>>> > > >> > >> > Before conversion, the parameter names are there,
but after
>>> > > >> > conversion,
>>> > > >> > >> the
>>> > > >> > >> > names
>>> > > >> > >> > are gone. It is because of unknown tables. The grib1
data
>>> is
>>> > > still
>>> > > >> > >> > original. The original
>>> > > >> > >> > grib1 data files are very strange data format we can
not
>>> read.
>>> > So
>>> > > >> we
>>> > > >> > >> have
>>> > > >> > >> > to use
>>> > > >> > >> > a Frotran code to re-arrange the fields and save
them as
>>> > > NCEP-like
>>> > > >> > grib1
>>> > > >> > >> > data files.
>>> > > >> > >> > So after several processes, some important info like
tables
>>> > could
>>> > > >> be
>>> > > >> > >> lost.
>>> > > >> > >> >
>>> > > >> > >> >    I'll test your first suggestion since the field
record
>>> order
>>> > > is
>>> > > >> > >> fixed.
>>> > > >> > >> > Currently we still
>>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we will
get
>>> grib2
>>> > > data
>>> > > >> > >> > directly. So before
>>> > > >> > >> > we can get original grib2 files from ECMWF, we just
use
>>> this
>>> > > >> temporal
>>> > > >> > >> > solution.
>>> > > >> > >> > '
>>> > > >> > >> > Thanks!
>>> > > >> > >> >
>>> > > >> > >> >
>>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway
via RT <
>>> > > >> > >> > met_help at ucar.edu>
>>> > > >> > >> > wrote:
>>> > > >> > >> >
>>> > > >> > >> > > Binbin,
>>> > > >> > >> > >
>>> > > >> > >> > > OK, working on Venus.
>>> > > >> > >> > >
>>> > > >> > >> > > You have a typo in the command you sent.
>>> > > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>>> > > >> > >> > >
>>> > > >> > >>
>>> > > >>
>>> >
>>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>>> > > >> > >> > -v
>>> > > >> > >> > > 3
>>> > > >> > >> > >
>>> > > >> > >> > > You have "RIB2_parm_cat" that should be
"GRIB2_parm_cat".
>>> > After
>>> > > >> > >> > correcting
>>> > > >> > >> > > that I did still get an error but it's a very
useful one!
>>> > > >> > >> > >
>>> > > >> > >> > > module use
>>> > > >> > >>
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>>> > > >> > >> > > module load met/9.1
>>> > > >> > >> > > module load grib_util/1.1.1
>>> > > >> > >> > > cd
>>> > > >> > >> > >
>>> > > >> > >> > >
>>> > > >> > >> >
>>> > > >> > >>
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
>>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
>>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
>>> > > >> > >> > 'GRIB2_disc=255;
>>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
>>> level="P500";'
>>> > > >> > >> > >
>>> > > >> > >> > >
>>> > > >> > >> > >
>>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() -> no
>>> parameter
>>> > > found
>>> > > >> > with
>>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm
>>> (5).
>>> > > Use
>>> > > >> the
>>> > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom
>>> GRIB
>>> > > >> > >> tables.ERROR
>>> > > >> > >> > :
>>> > > >> > >> > > *
>>> > > >> > >> > >
>>> > > >> > >> > > So this table is not yet defined in MET. I checked
their
>>> > > website:
>>> > > >> > >> > >
>>> > > >> > >> > >
>>> > > >> > >> >
>>> > > >> > >>
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>>> > > >> > >> > >
>>> > > >> > >> > > But I don't see an entry defined for GRIB2_parm=5;
>>> > > >> > >> > >
>>> > > >> > >> > > 3 things...
>>> > > >> > >> > >
>>> > > >> > >> > > (1) As a crude temporary solution, you could refer
to it
>>> by
>>> > > it's
>>> > > >> > >> record
>>> > > >> > >> > > number. Let's assume you know that record 1 has
>>> temperature
>>> > > data:
>>> > > >> > >> > >
>>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/plot.ps
>>> > > >> > 'name="TMP";
>>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
>>> > > >> > >> > >
>>> > > >> > >> > > Here I'm reading record 1, naming it TMP and the
using
>>> > > >> > set_attr_level
>>> > > >> > >> > write
>>> > > >> > >> > > the output level as P500 instead of R1.
>>> > > >> > >> > >
>>> > > >> > >> > > (2) You could create a custom GRIB table yourself
>>> locally,
>>> > and
>>> > > >> then
>>> > > >> > >> set
>>> > > >> > >> > the
>>> > > >> > >> > > MET_GRIB_TABLES environment variable to point MET
to it.
>>> > > >> > >> > >
>>> > > >> > >> > > (3) The best solution is contacting ECMWF and
requesting
>>> they
>>> > > >> send
>>> > > >> > >> over
>>> > > >> > >> > the
>>> > > >> > >> > > GRIB2 table we should be using to decode this
data.
>>> > > >> > >> > >
>>> > > >> > >> > > Thanks,
>>> > > >> > >> > > John
>>> > > >> > >> > >
>>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley Gotway
<
>>> > > >> johnhg at ucar.edu>
>>> > > >> > >> > wrote:
>>> > > >> > >> > >
>>> > > >> > >> > > > We might already have them. I'm not sure.
>>> > > >> > >> > > >
>>> > > >> > >> > > > But there's an additional required piece of the
puzzle.
>>> > > Binbin
>>> > > >> > runs
>>> > > >> > >> > > > "wgrib2" to figure out what "name" he should use
in
>>> the MET
>>> > > >> config
>>> > > >> > >> > files.
>>> > > >> > >> > > > If wgrib2 doesn't tell him the right "name", the
>>> problem
>>> > will
>>> > > >> > >> remain.
>>> > > >> > >> > > > wgrib2 can be configured to define additional
tables as
>>> > well,
>>> > > >> but
>>> > > >> > >> for
>>> > > >> > >> > > this
>>> > > >> > >> > > > to be easy for Binbin, that'd mean updating the
script
>>> > that's
>>> > > >> run
>>> > > >> > >> for
>>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
>>> that's
>>> > > not
>>> > > >> > very
>>> > > >> > >> > easy.
>>> > > >> > >> > > >
>>> > > >> > >> > > > Yes, I do agree that the best solution is
getting the
>>> ECMWF
>>> > > >> GRIB
>>> > > >> > >> tables
>>> > > >> > >> > > > and adding them for both wgrib2 and MET.
>>> > > >> > >> > > >
>>> > > >> > >> > > > John
>>> > > >> > >> > > >
>>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
>>> > jensen at ucar.edu>
>>> > > >> > wrote:
>>> > > >> > >> > > >
>>> > > >> > >> > > >> John,
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> It seems to me that it would be worthwhile to
just ask
>>> > ECMWF
>>> > > >> for
>>> > > >> > >> their
>>> > > >> > >> > > >> Grib2 Table for their Ensemble files.  And
other
>>> > operational
>>> > > >> > >> centers
>>> > > >> > >> > for
>>> > > >> > >> > > >> that matter.  Do you agree this would be a good
>>> solution?
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Thanks, Tara
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> ---------- Forwarded message ---------
>>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT
<met_help at ucar.edu>
>>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
>>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
>>> question for
>>> > > >> > reading
>>> > > >> > >> > ECMWF
>>> > > >> > >> > > >> ensmeble data
>>> > > >> > >> > > >> To:
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> <URL:
>>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>>> > > >> >
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> John,
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>   I tested using following command:
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>>> > > >> > >> > > >>
>>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>>> > > >> -v3
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> ERROR  :
>>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
>>> > > >> GRIB2_disc,
>>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
>>> field
>>> > > >> > >> information
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
>>> > > >> > >> > > >>
>>> > > >> > >> >
>>> > > >> >
>>> > >
>>>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>>> > > >> > >> > > >> -v
>>> > > >> > >> > > >> 3
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Still got error
>>> > > >> > >> > > >> ERROR  :
>>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
>>> > > >> GRIB2_disc,
>>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be specified
in
>>> field
>>> > > >> > >> information
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Looks like discipline=255 is not recognized
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Binbin
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway
>>> via RT
>>> > <
>>> > > >> > >> > > >> met_help at ucar.edu>
>>> > > >> > >> > > >> wrote:
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> > Binbin,
>>> > > >> > >> > > >> > You can do something similar with MET. Please
go to
>>> this
>>> > > >> link:
>>> > > >> > >> > > >> >
>>> > > >>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and you'll
find
>>> the
>>> > > >> content
>>> > > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD
>>> OF
>>> > > >> > specifying
>>> > > >> > >> a
>>> > > >> > >> > > >> > requested name.
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
>>> specifying the
>>> > > >> master
>>> > > >> > >> > table
>>> > > >> > >> > > >> > number.
>>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
>>> specifying the
>>> > > >> local
>>> > > >> > >> table
>>> > > >> > >> > > >> > number.
>>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
>>> specifying the
>>> > > >> GRIB2
>>> > > >> > >> > > >> discipline
>>> > > >> > >> > > >> > code.
>>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an integer
>>> specifying
>>> > > the
>>> > > >> > >> > parameter
>>> > > >> > >> > > >> > category
>>> > > >> > >> > > >> > //         code.
>>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
>>> specifying the
>>> > > >> > >> parameter
>>> > > >> > >> > > code.
>>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
specifying
>>> the
>>> > > >> product
>>> > > >> > >> > > >> definition
>>> > > >> > >> > > >> > //         template (Table 4.0).
>>> > > >> > >> > > >> > //       - The "GRIB2_process" is an integer
>>> specifying
>>> > > the
>>> > > >> > >> > generating
>>> > > >> > >> > > >> > process
>>> > > >> > >> > > >> > //         (Table 4.3).
>>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
>>> specifying the
>>> > > >> > >> originating
>>> > > >> > >> > > >> > center.
>>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an integer
>>> specifying
>>> > > the
>>> > > >> > >> > ensemble
>>> > > >> > >> > > >> type
>>> > > >> > >> > > >> > //         (Table 4.6).
>>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an integer
>>> specifying
>>> > > the
>>> > > >> > >> derived
>>> > > >> > >> > > >> > product
>>> > > >> > >> > > >> > //         type (Table 4.7).
>>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
>>> > specifying
>>> > > >> the
>>> > > >> > >> > > >> statistical
>>> > > >> > >> > > >> > //         processing type (Table 4.10).
>>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
>>> > > "GRIB2_ipdtmpl_val"
>>> > > >> > >> entries
>>> > > >> > >> > > are
>>> > > >> > >> > > >> > arrays
>>> > > >> > >> > > >> > //         of integers which specify the
product
>>> > > description
>>> > > >> > >> > template
>>> > > >> > >> > > >> > values to
>>> > > >> > >> > > >> > //         be used. The indices are 0-based.
For
>>> > example,
>>> > > >> use
>>> > > >> > the
>>> > > >> > >> > > >> > following to
>>> > > >> > >> > > >> > //         request a GRIB2 record whose 9-th
and
>>> 27-th
>>> > > >> product
>>> > > >> > >> > > >> description
>>> > > >> > >> > > >> > //         template values are 1 and 2,
>>> respectively:
>>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
>>> > > >> > GRIB2_ipdtmpl_val=[1,
>>> > > >> > >> 2];
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > While I don't have any ECMWF data on hand for
>>> testing, I
>>> > > do
>>> > > >> > have
>>> > > >> > >> a
>>> > > >> > >> > > file
>>> > > >> > >> > > >> > from the Air Force from which wgrib2 prints a
>>> similar
>>> > > >> result:
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
>>> center=57
>>> > > >> > >> local_table=1
>>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > So I used that for testing. I ran the
following
>>> > > >> plot_data_plane
>>> > > >> > >> > > command
>>> > > >> > >> > > >> > WITHOUT specifying a requested name:
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > plot_data_plane GALWEM_20170610_CY00_FH18.GR2
>>> plot.ps
>>> > > >> > >> > 'GRIB2_disc=0;
>>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
>>> > > >> GRIB2_parm=192;
>>> > > >> > >> > > >> level="Z0";'
>>> > > >> > >> > > >> > -v 3
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > And then I looked closely at the log message
to
>>> confirm
>>> > > >> that it
>>> > > >> > >> used
>>> > > >> > >> > > >> > record 549, as expected:
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() ->
Found
>>> exact
>>> > > match
>>> > > >> > for
>>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of
GRIB2 file
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >>
>>> > > >> > >> > >
>>> > > >> > >> >
>>> > > >> > >>
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > So MET used these GRIB2 numbers I defined to
do a
>>> table
>>> > > >> lookup
>>> > > >> > >> and
>>> > > >> > >> > > find
>>> > > >> > >> > > >> a
>>> > > >> > >> > > >> > match in the MET GRIB tables:
>>> > > >> > >> > > >> >
>>> > > >> > >>
>>> > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > And it determined that that field should be
called
>>> > > "CTSTM".
>>> > > >> > >> > Hopefully,
>>> > > >> > >> > > >> the
>>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined there
in
>>> MET. If
>>> > > >> not, we
>>> > > >> > >> can
>>> > > >> > >> > > >> always
>>> > > >> > >> > > >> > add them.
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > Please try running plot_data_plane at
verbosity
>>> level 3
>>> > on
>>> > > >> one
>>> > > >> > of
>>> > > >> > >> > > these
>>> > > >> > >> > > >> > files first. And make sure it runs without
error
>>> and the
>>> > > log
>>> > > >> > >> message
>>> > > >> > >> > > >> prints
>>> > > >> > >> > > >> > out the "name" it found for that data. If so,
you
>>> can
>>> > just
>>> > > >> skip
>>> > > >> > >> > using
>>> > > >> > >> > > >> all
>>> > > >> > >> > > >> > those integers and revert back to using the
name MET
>>> > > knows.
>>> > > >> If
>>> > > >> > >> not,
>>> > > >> > >> > > >> then we
>>> > > >> > >> > > >> > may need to define a custom GRIB table for
this
>>> ECMWF
>>> > > GRIB2
>>> > > >> > data.
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >> > Thanks,
>>> > > >> > >> > > >> > John
>>> > > >> > >> > > >> >
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> --
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Binbin Zhou
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> 5830 University Research Ct.
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> College Park, MD 20740
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> 301-683-3683
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >>
>>> > > >> > >> > > >> --
>>> > > >> > >> > > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> > > >> > >> > > >> Tara Jensen
>>> > > >> > >> > > >> Project Manager II
>>> > > >> > >> > > >> NCAR RAL and DTC
>>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
>>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
>>> > > >> > >> > > >>
>>> > > >> > >> > > >
>>> > > >> > >> > >
>>> > > >> > >> > >
>>> > > >> > >> >
>>> > > >> > >> > --
>>> > > >> > >> >
>>> > > >> > >> > Binbin Zhou
>>> > > >> > >> >
>>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
>>> > > >> > >> >
>>> > > >> > >> > 5830 University Research Ct.
>>> > > >> > >> >
>>> > > >> > >> > College Park, MD 20740
>>> > > >> > >> >
>>> > > >> > >> > Binbin.Zhou at noaa.gov
>>> > > >> > >> >
>>> > > >> > >> > 301-683-3683
>>> > > >> > >> >
>>> > > >> > >> >
>>> > > >> > >>
>>> > > >> > >>
>>> > > >> > >
>>> > > >> > > --
>>> > > >> > >
>>> > > >> > > Binbin Zhou
>>> > > >> > >
>>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
>>> > > >> > >
>>> > > >> > > 5830 University Research Ct.
>>> > > >> > >
>>> > > >> > > College Park, MD 20740
>>> > > >> > >
>>> > > >> > > Binbin.Zhou at noaa.gov
>>> > > >> > >
>>> > > >> > > 301-683-3683
>>> > > >> > >
>>> > > >> >
>>> > > >> >
>>> > > >> > --
>>> > > >> >
>>> > > >> > Binbin Zhou
>>> > > >> >
>>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
>>> > > >> >
>>> > > >> > 5830 University Research Ct.
>>> > > >> >
>>> > > >> > College Park, MD 20740
>>> > > >> >
>>> > > >> > Binbin.Zhou at noaa.gov
>>> > > >> >
>>> > > >> > 301-683-3683
>>> > > >> >
>>> > > >> >
>>> > > >>
>>> > > >>
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Binbin Zhou
>>> > > >
>>> > > > IMSG at NOAA/NWS/NCEP/EMC
>>> > > >
>>> > > > 5830 University Research Ct.
>>> > > >
>>> > > > College Park, MD 20740
>>> > > >
>>> > > > Binbin.Zhou at noaa.gov
>>> > > >
>>> > > > 301-683-3683
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > >
>>> > > Binbin Zhou
>>> > >
>>> > > IMSG at NOAA/NWS/NCEP/EMC
>>> > >
>>> > > 5830 University Research Ct.
>>> > >
>>> > > College Park, MD 20740
>>> > >
>>> > > Binbin.Zhou at noaa.gov
>>> > >
>>> > > 301-683-3683
>>> > >
>>> > >
>>> >
>>> >
>>>
>>> --
>>>
>>> Binbin Zhou
>>>
>>> IMSG at NOAA/NWS/NCEP/EMC
>>>
>>> 5830 University Research Ct.
>>>
>>> College Park, MD 20740
>>>
>>> Binbin.Zhou at noaa.gov
>>>
>>> 301-683-3683
>>>
>>>

--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Thu Dec 10 10:05:37 2020

Julie,

Building the tarfile is very easy. I just ran this script from the MET
repo
and specified the branch to be built:
   MET/scripts/met_checkout_and_build.sh branch
feature_1604_ens_config

And that created a tarfile which I posted to RAL's anonymous ftp
server:

ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz

I assume that'll work for you.

Thanks,
John

On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
> I am able to help with this install on WCOSS.
>
> Would it be easier to use git to checkout the repo on WCOSS? Or
would it be
> > easier if I built a tarfile for this version of the code?
> >
> The last time I tried to build from git on WCOSS (a while ago), I
> couldn't.  I don't remember all of the details, but I think I
couldn't run
> bootstrap and maybe it had to do with the version of autoconf being
too
> old?  I'm happy to try it again if you'd like though.  Otherwise, I
know a
> tarfile would work, but I know that is extra work for you.
>
> Julie
>
> On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Julie,
> >
> > I have created a bugfix branch for main_v9.1 named
> > "feature_1604_ens_config" and it's up on GitHub.
> >
> > Only a single file has changed in it. But I need to get this
version
> > compiled on WCOSS with which Binbin can test.
> >
> > Are you to help me with this? Would it be easier to use git to
checkout
> > the repo on WCOSS? Or would it be easier if I built a tarfile for
this
> > version of the code?
> >
> > Thanks,
> > John
> >
> > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> >> Binbin,
> >>
> >> In order for you to test this change out, we'd need to recompile
a
> >> version of the code on WCOSS. I wrote this up as a GitHub issue:
> >> https://github.com/dtcenter/MET/issues/1604
> >>
> >> And I'll create a branch from main_v9.1 named
"bugfix_1604_ens_config"
> >> and commit the change there. Then we can checkout that code on
WCOSS,
> >> compile it, and you can test it.
> >>
> >> Yes, you're right, if the fcst fields differ from the climo
fields,
> you'd
> >> need to specify them separately. That's pretty easy in the
Ensemble-Stat
> >> config file... but are you asking how to control that via
METplus?
> >>
> >> John
> >>
> >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >>>
> >>> John,
> >>>
> >>>   Great! Let me test it at first before make change in MET9.1
> >>> Another question:
> >>>    Since climatology data is still in NCEP's grib2 format, In
the
> >>> ensemble_stat
> >>> config file,  climo  entry settings  should be different from
those of
> >>> fcst, obs entries?
> >>> How to set climo (name, level, etc.)?
> >>>
> >>> Binbin
> >>>
> >>>
> >>>
> >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> >>> met_help at ucar.edu>
> >>> wrote:
> >>>
> >>> > Binbin,
> >>> >
> >>> > Success! When I recompile with that one line commented out,
> >>> plot_data_plane
> >>> > is able to filter down all those records to the single one
that we
> >>> want.
> >>> > Note in the example below that I used the "lead_time" and
"GRIB_ens"
> >>> > options to filter the input records down.
> >>> >
> >>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
> >>> level="P500";
> >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found exact
match
> for
> >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> >>> "ensposte.t00z.z500hr".*
> >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB
records
> >>> > matching VarInfo "HGT/P500" in GRIB file
"ensposte.t00z.z500hr".
> >>> > DEBUG 1: Creating postscript file: plot.ps
> >>> >
> >>> > Also note that this selects out record number 26.
> >>> >
> >>> > Shall I make this change as a "bugfix" for met-9.1...which
could be
> >>> > included in a met-9.1.2 bugfix release? Or shall I just
include it in
> >>> > development for the next release (planned for March 2021)?
> >>> >
> >>> > Thanks,
> >>> > John
> >>> >
> >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
> >>> > met_help at ucar.edu> wrote:
> >>> >
> >>> > >
> >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >>> > >
> >>> > > John,
> >>> > >
> >>> > >   If this removing impacts other applications, or takes some
time
> >>> > > to be effective, then won't try.  I will use another
approach, such
> >>> as
> >>> > > borrow GEFS's grib2 data structure as tank to save EC
ensemble
> >>> > > data.
> >>> > >
> >>> > > Binbin
> >>> > >
> >>> > >
> >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA Affiliate
<
> >>> > > binbin.zhou at noaa.gov> wrote:
> >>> > >
> >>> > > > John,
> >>> > > >
> >>> > > >   Yes, please try to use this approach.
> >>> > > >
> >>> > > > Thanks!
> >>> > > >
> >>> > > > Binbin
> >>> > > >
> >>> > > > Binbin
> >>> > > >
> >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT <
> >>> > > > met_help at ucar.edu> wrote:
> >>> > > >
> >>> > > >> Binbin,
> >>> > > >>
> >>> > > >> I used the "-PDS10" option for wgrib to dump those
records:
> >>> > > >>
> >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> >>> > > >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> >>> > > >> | egrep  "^11:|^26:|^43:"
> >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0
1 0 0
> 1
> >>> 0 0
> >>> > 0
> >>> > > 21
> >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0
1 0 0
> 1
> >>> 0 0
> >>> > 0
> >>> > > 21
> >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0 0
1 0 0
> 1
> >>> 0 0
> >>> > 0
> >>> > > 21
> >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> >>> > > >>
> >>> > > >> Looking VERY closely, there is a difference in the 42-nd
column
> >>> which
> >>> > is
> >>> > > >> the 4-th from the last... records 11, 26, and 43 have
values of
> >>> 3, 1,
> >>> > > and
> >>> > > >> 2
> >>> > > >> respectively.
> >>> > > >>
> >>> > > >> On this page (
> >>> > > https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> >>> > > >> ),
> >>> > > >> byte 42 = 1 means "unperturbed control forecast".
> >>> > > >>
> >>> > > >> So we *should* be able to use the "GRIB_ens" config
option to
> >>> > > >> differentiate
> >>> > > >> between those records. BUT when I try, it doesn't work.
And
> that's
> >>> > > because
> >>> > > >> of this line:
> >>> > > >>
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> >>> > > >>
> >>> > > >> We only apply those filtering options to GRIB files
created by
> >>> NCEP.
> >>> > If
> >>> > > >> you'd like, I could try removing that condition to see if
we can
> >>> pick
> >>> > > out
> >>> > > >> the single record of interest.
> >>> > > >>
> >>> > > >> John
> >>> > > >>
> >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via
RT <
> >>> > > >> met_help at ucar.edu> wrote:
> >>> > > >>
> >>> > > >> >
> >>> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> >>> > > >> >
> >>> > > >> > John,
> >>> > > >> >
> >>> > > >> >   Look at my
> >>> > > >> > directory
> >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is only
for 500
> >>> HGT.
> >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
> >>> > > >> > I get 3 records:
> >>> > > >> >
> >>> > > >> >
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >>> > > >> > mb:anl:type=analysis:NAve=0
> >>> > > >> >
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >>> > > >> > mb:anl:type=analysis:NAve=0
> >>> > > >> >
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >>> > > >> > mb:anl:type=analysis:NAve=0
> >>> > > >> >
> >>> > > >> > From its grib2 file, we know only  record#26 is
analysis
> low-res
> >>> > > >> control.
> >>> > > >> > If you have some way to determine which record is
analysis
> >>> low-res
> >>> > > >> control
> >>> > > >> > from the grib1 file,
> >>> > > >> > then we can use grib1 directly (don't consider the size
> issue).
> >>> > > >> >
> >>> > > >> > Binbin
> >>> > > >> >
> >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
Affiliate <
> >>> > > >> > binbin.zhou at noaa.gov> wrote:
> >>> > > >> >
> >>> > > >> > > John,
> >>> > > >> > >
> >>> > > >> > >   Grib1 data is too large, another issue is the low-
res
> >>> control
> >>> > > >> analysis
> >>> > > >> > > data can not be retrieved
> >>> > > >> > > from the grib1 ecmwf data files. If using wgrib, we
can see
> 3
> >>> > > analysis
> >>> > > >> > > fields, but don't know
> >>> > > >> > > which is low-res control. After convert to grib2,
wgrib2 can
> >>> show
> >>> > > >> which
> >>> > > >> > > record is low-res control.
> >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on
WCOSS.
> >>> > > >> > >
> >>> > > >> > > Binbin
> >>> > > >> > >
> >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway via
RT <
> >>> > > >> > > met_help at ucar.edu> wrote:
> >>> > > >> > >
> >>> > > >> > >> Binbin,
> >>> > > >> > >>
> >>> > > >> > >> Sounds like there's a lot of moving parts there. How
would
> >>> you
> >>> > like
> >>> > > >> to
> >>> > > >> > >> proceed?
> >>> > > >> > >>
> >>> > > >> > >> Should we check to see if MET can read those ECMWF
GRIB1
> >>> files
> >>> > > >> directly
> >>> > > >> > >> without needing to reformat?
> >>> > > >> > >> Or do you just want to create a custom GRIB table
for the
> >>> result
> >>> > of
> >>> > > >> your
> >>> > > >> > >> pre-processing and conversion steps?
> >>> > > >> > >>
> >>> > > >> > >> John
> >>> > > >> > >>
> >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM Binbin.Zhou at noaa.gov
via
> RT <
> >>> > > >> > >> met_help at ucar.edu> wrote:
> >>> > > >> > >>
> >>> > > >> > >> >
> >>> > > >> > >> > <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >>> > >
> >>> > > >> > >> >
> >>> > > >> > >> > John,
> >>> > > >> > >> >
> >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
converted from
> >>> its
> >>> > > grib1
> >>> > > >> > >> format
> >>> > > >> > >> > data.
> >>> > > >> > >> > Before conversion, the parameter names are there,
but
> after
> >>> > > >> > conversion,
> >>> > > >> > >> the
> >>> > > >> > >> > names
> >>> > > >> > >> > are gone. It is because of unknown tables. The
grib1 data
> >>> is
> >>> > > still
> >>> > > >> > >> > original. The original
> >>> > > >> > >> > grib1 data files are very strange data format we
can not
> >>> read.
> >>> > So
> >>> > > >> we
> >>> > > >> > >> have
> >>> > > >> > >> > to use
> >>> > > >> > >> > a Frotran code to re-arrange the fields and save
them as
> >>> > > NCEP-like
> >>> > > >> > grib1
> >>> > > >> > >> > data files.
> >>> > > >> > >> > So after several processes, some important info
like
> tables
> >>> > could
> >>> > > >> be
> >>> > > >> > >> lost.
> >>> > > >> > >> >
> >>> > > >> > >> >    I'll test your first suggestion since the field
record
> >>> order
> >>> > > is
> >>> > > >> > >> fixed.
> >>> > > >> > >> > Currently we still
> >>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we
will get
> >>> grib2
> >>> > > data
> >>> > > >> > >> > directly. So before
> >>> > > >> > >> > we can get original grib2 files from ECMWF, we
just use
> >>> this
> >>> > > >> temporal
> >>> > > >> > >> > solution.
> >>> > > >> > >> > '
> >>> > > >> > >> > Thanks!
> >>> > > >> > >> >
> >>> > > >> > >> >
> >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley Gotway
via RT
> <
> >>> > > >> > >> > met_help at ucar.edu>
> >>> > > >> > >> > wrote:
> >>> > > >> > >> >
> >>> > > >> > >> > > Binbin,
> >>> > > >> > >> > >
> >>> > > >> > >> > > OK, working on Venus.
> >>> > > >> > >> > >
> >>> > > >> > >> > > You have a typo in the command you sent.
> >>> > > >> > >> > >  plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
a.ps
> >>> > > >> > >> > >
> >>> > > >> > >>
> >>> > > >>
> >>> >
> >>>
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >>> > > >> > >> > -v
> >>> > > >> > >> > > 3
> >>> > > >> > >> > >
> >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
> "GRIB2_parm_cat".
> >>> > After
> >>> > > >> > >> > correcting
> >>> > > >> > >> > > that I did still get an error but it's a very
useful
> one!
> >>> > > >> > >> > >
> >>> > > >> > >> > > module use
> >>> > > >> > >>
> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> >>> > > >> > >> > > module load met/9.1
> >>> > > >> > >> > > module load grib_util/1.1.1
> >>> > > >> > >> > > cd
> >>> > > >> > >> > >
> >>> > > >> > >> > >
> >>> > > >> > >> >
> >>> > > >> > >>
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> >>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/
> plot.ps
> >>> > > >> > >> > 'GRIB2_disc=255;
> >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
> >>> level="P500";'
> >>> > > >> > >> > >
> >>> > > >> > >> > >
> >>> > > >> > >> > >
> >>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() ->
no
> >>> parameter
> >>> > > found
> >>> > > >> > with
> >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
GRIB2_parm
> >>> (5).
> >>> > > Use
> >>> > > >> the
> >>> > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom
> >>> GRIB
> >>> > > >> > >> tables.ERROR
> >>> > > >> > >> > :
> >>> > > >> > >> > > *
> >>> > > >> > >> > >
> >>> > > >> > >> > > So this table is not yet defined in MET. I
checked
> their
> >>> > > website:
> >>> > > >> > >> > >
> >>> > > >> > >> > >
> >>> > > >> > >> >
> >>> > > >> > >>
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> >>> > > >> > >> > >
> >>> > > >> > >> > > But I don't see an entry defined for
GRIB2_parm=5;
> >>> > > >> > >> > >
> >>> > > >> > >> > > 3 things...
> >>> > > >> > >> > >
> >>> > > >> > >> > > (1) As a crude temporary solution, you could
refer to
> it
> >>> by
> >>> > > it's
> >>> > > >> > >> record
> >>> > > >> > >> > > number. Let's assume you know that record 1 has
> >>> temperature
> >>> > > data:
> >>> > > >> > >> > >
> >>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/
> plot.ps
> >>> > > >> > 'name="TMP";
> >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> >>> > > >> > >> > >
> >>> > > >> > >> > > Here I'm reading record 1, naming it TMP and the
using
> >>> > > >> > set_attr_level
> >>> > > >> > >> > write
> >>> > > >> > >> > > the output level as P500 instead of R1.
> >>> > > >> > >> > >
> >>> > > >> > >> > > (2) You could create a custom GRIB table
yourself
> >>> locally,
> >>> > and
> >>> > > >> then
> >>> > > >> > >> set
> >>> > > >> > >> > the
> >>> > > >> > >> > > MET_GRIB_TABLES environment variable to point
MET to
> it.
> >>> > > >> > >> > >
> >>> > > >> > >> > > (3) The best solution is contacting ECMWF and
> requesting
> >>> they
> >>> > > >> send
> >>> > > >> > >> over
> >>> > > >> > >> > the
> >>> > > >> > >> > > GRIB2 table we should be using to decode this
data.
> >>> > > >> > >> > >
> >>> > > >> > >> > > Thanks,
> >>> > > >> > >> > > John
> >>> > > >> > >> > >
> >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley
Gotway <
> >>> > > >> johnhg at ucar.edu>
> >>> > > >> > >> > wrote:
> >>> > > >> > >> > >
> >>> > > >> > >> > > > We might already have them. I'm not sure.
> >>> > > >> > >> > > >
> >>> > > >> > >> > > > But there's an additional required piece of
the
> puzzle.
> >>> > > Binbin
> >>> > > >> > runs
> >>> > > >> > >> > > > "wgrib2" to figure out what "name" he should
use in
> >>> the MET
> >>> > > >> config
> >>> > > >> > >> > files.
> >>> > > >> > >> > > > If wgrib2 doesn't tell him the right "name",
the
> >>> problem
> >>> > will
> >>> > > >> > >> remain.
> >>> > > >> > >> > > > wgrib2 can be configured to define additional
tables
> as
> >>> > well,
> >>> > > >> but
> >>> > > >> > >> for
> >>> > > >> > >> > > this
> >>> > > >> > >> > > > to be easy for Binbin, that'd mean updating
the
> script
> >>> > that's
> >>> > > >> run
> >>> > > >> > >> for
> >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
> >>> that's
> >>> > > not
> >>> > > >> > very
> >>> > > >> > >> > easy.
> >>> > > >> > >> > > >
> >>> > > >> > >> > > > Yes, I do agree that the best solution is
getting the
> >>> ECMWF
> >>> > > >> GRIB
> >>> > > >> > >> tables
> >>> > > >> > >> > > > and adding them for both wgrib2 and MET.
> >>> > > >> > >> > > >
> >>> > > >> > >> > > > John
> >>> > > >> > >> > > >
> >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
> >>> > jensen at ucar.edu>
> >>> > > >> > wrote:
> >>> > > >> > >> > > >
> >>> > > >> > >> > > >> John,
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> It seems to me that it would be worthwhile to
just
> ask
> >>> > ECMWF
> >>> > > >> for
> >>> > > >> > >> their
> >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.  And
other
> >>> > operational
> >>> > > >> > >> centers
> >>> > > >> > >> > for
> >>> > > >> > >> > > >> that matter.  Do you agree this would be a
good
> >>> solution?
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Thanks, Tara
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> ---------- Forwarded message ---------
> >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu>
> >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769] urgent
> >>> question for
> >>> > > >> > reading
> >>> > > >> > >> > ECMWF
> >>> > > >> > >> > > >> ensmeble data
> >>> > > >> > >> > > >> To:
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> <URL:
> >>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >>> > > >> >
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> John,
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>   I tested using following command:
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> a.ps
> >>> > > >> > >> > > >>
> >>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >>> > > >> -v3
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> ERROR  :
> >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
> >>> > > >> GRIB2_disc,
> >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified in
> >>> field
> >>> > > >> > >> information
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> a.ps
> >>> > > >> > >> > > >>
> >>> > > >> > >> >
> >>> > > >> >
> >>> > >
> >>>
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> >>> > > >> > >> > > >> -v
> >>> > > >> > >> > > >> 3
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Still got error
> >>> > > >> > >> > > >> ERROR  :
> >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name or
> >>> > > >> GRIB2_disc,
> >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified in
> >>> field
> >>> > > >> > >> information
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Looks like discipline=255 is not recognized
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Binbin
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway
> >>> via RT
> >>> > <
> >>> > > >> > >> > > >> met_help at ucar.edu>
> >>> > > >> > >> > > >> wrote:
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> > Binbin,
> >>> > > >> > >> > > >> > You can do something similar with MET.
Please go
> to
> >>> this
> >>> > > >> link:
> >>> > > >> > >> > > >> >
> >>> > > >>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and
you'll find
> >>> the
> >>> > > >> content
> >>> > > >> > >> > > >> > copied/pasted below. You can specify these
INSTEAD
> >>> OF
> >>> > > >> > specifying
> >>> > > >> > >> a
> >>> > > >> > >> > > >> > requested name.
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
> >>> specifying the
> >>> > > >> master
> >>> > > >> > >> > table
> >>> > > >> > >> > > >> > number.
> >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
> >>> specifying the
> >>> > > >> local
> >>> > > >> > >> table
> >>> > > >> > >> > > >> > number.
> >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
> >>> specifying the
> >>> > > >> GRIB2
> >>> > > >> > >> > > >> discipline
> >>> > > >> > >> > > >> > code.
> >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an
integer
> >>> specifying
> >>> > > the
> >>> > > >> > >> > parameter
> >>> > > >> > >> > > >> > category
> >>> > > >> > >> > > >> > //         code.
> >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
> >>> specifying the
> >>> > > >> > >> parameter
> >>> > > >> > >> > > code.
> >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
> specifying
> >>> the
> >>> > > >> product
> >>> > > >> > >> > > >> definition
> >>> > > >> > >> > > >> > //         template (Table 4.0).
> >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
integer
> >>> specifying
> >>> > > the
> >>> > > >> > >> > generating
> >>> > > >> > >> > > >> > process
> >>> > > >> > >> > > >> > //         (Table 4.3).
> >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
> >>> specifying the
> >>> > > >> > >> originating
> >>> > > >> > >> > > >> > center.
> >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an
integer
> >>> specifying
> >>> > > the
> >>> > > >> > >> > ensemble
> >>> > > >> > >> > > >> type
> >>> > > >> > >> > > >> > //         (Table 4.6).
> >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an
integer
> >>> specifying
> >>> > > the
> >>> > > >> > >> derived
> >>> > > >> > >> > > >> > product
> >>> > > >> > >> > > >> > //         type (Table 4.7).
> >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
> >>> > specifying
> >>> > > >> the
> >>> > > >> > >> > > >> statistical
> >>> > > >> > >> > > >> > //         processing type (Table 4.10).
> >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> >>> > > "GRIB2_ipdtmpl_val"
> >>> > > >> > >> entries
> >>> > > >> > >> > > are
> >>> > > >> > >> > > >> > arrays
> >>> > > >> > >> > > >> > //         of integers which specify the
product
> >>> > > description
> >>> > > >> > >> > template
> >>> > > >> > >> > > >> > values to
> >>> > > >> > >> > > >> > //         be used. The indices are 0-
based. For
> >>> > example,
> >>> > > >> use
> >>> > > >> > the
> >>> > > >> > >> > > >> > following to
> >>> > > >> > >> > > >> > //         request a GRIB2 record whose 9-
th and
> >>> 27-th
> >>> > > >> product
> >>> > > >> > >> > > >> description
> >>> > > >> > >> > > >> > //         template values are 1 and 2,
> >>> respectively:
> >>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8, 26];
> >>> > > >> > GRIB2_ipdtmpl_val=[1,
> >>> > > >> > >> 2];
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > While I don't have any ECMWF data on hand
for
> >>> testing, I
> >>> > > do
> >>> > > >> > have
> >>> > > >> > >> a
> >>> > > >> > >> > > file
> >>> > > >> > >> > > >> > from the Air Force from which wgrib2 prints
a
> >>> similar
> >>> > > >> result:
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var discipline=0
> >>> center=57
> >>> > > >> > >> local_table=1
> >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > So I used that for testing. I ran the
following
> >>> > > >> plot_data_plane
> >>> > > >> > >> > > command
> >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > plot_data_plane
GALWEM_20170610_CY00_FH18.GR2
> >>> plot.ps
> >>> > > >> > >> > 'GRIB2_disc=0;
> >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
> >>> > > >> GRIB2_parm=192;
> >>> > > >> > >> > > >> level="Z0";'
> >>> > > >> > >> > > >> > -v 3
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > And then I looked closely at the log
message to
> >>> confirm
> >>> > > >> that it
> >>> > > >> > >> used
> >>> > > >> > >> > > >> > record 549, as expected:
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane() ->
Found
> >>> exact
> >>> > > match
> >>> > > >> > for
> >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of
GRIB2
> file
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >>
> >>> > > >> > >> > >
> >>> > > >> > >> >
> >>> > > >> > >>
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> >
> >>>
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I defined
to do a
> >>> table
> >>> > > >> lookup
> >>> > > >> > >> and
> >>> > > >> > >> > > find
> >>> > > >> > >> > > >> a
> >>> > > >> > >> > > >> > match in the MET GRIB tables:
> >>> > > >> > >> > > >> >
> >>> > > >> > >>
> >>> > >
> https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > And it determined that that field should be
called
> >>> > > "CTSTM".
> >>> > > >> > >> > Hopefully,
> >>> > > >> > >> > > >> the
> >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined
there in
> >>> MET. If
> >>> > > >> not, we
> >>> > > >> > >> can
> >>> > > >> > >> > > >> always
> >>> > > >> > >> > > >> > add them.
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > Please try running plot_data_plane at
verbosity
> >>> level 3
> >>> > on
> >>> > > >> one
> >>> > > >> > of
> >>> > > >> > >> > > these
> >>> > > >> > >> > > >> > files first. And make sure it runs without
error
> >>> and the
> >>> > > log
> >>> > > >> > >> message
> >>> > > >> > >> > > >> prints
> >>> > > >> > >> > > >> > out the "name" it found for that data. If
so, you
> >>> can
> >>> > just
> >>> > > >> skip
> >>> > > >> > >> > using
> >>> > > >> > >> > > >> all
> >>> > > >> > >> > > >> > those integers and revert back to using the
name
> MET
> >>> > > knows.
> >>> > > >> If
> >>> > > >> > >> not,
> >>> > > >> > >> > > >> then we
> >>> > > >> > >> > > >> > may need to define a custom GRIB table for
this
> >>> ECMWF
> >>> > > GRIB2
> >>> > > >> > data.
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >> > Thanks,
> >>> > > >> > >> > > >> > John
> >>> > > >> > >> > > >> >
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> --
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Binbin Zhou
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> 5830 University Research Ct.
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> College Park, MD 20740
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> 301-683-3683
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >> --
> >>> > > >> > >> > > >>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> > > >> > >> > > >> Tara Jensen
> >>> > > >> > >> > > >> Project Manager II
> >>> > > >> > >> > > >> NCAR RAL and DTC
> >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> >>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> >>> > > >> > >> > > >>
> >>> > > >> > >> > > >
> >>> > > >> > >> > >
> >>> > > >> > >> > >
> >>> > > >> > >> >
> >>> > > >> > >> > --
> >>> > > >> > >> >
> >>> > > >> > >> > Binbin Zhou
> >>> > > >> > >> >
> >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> >>> > > >> > >> >
> >>> > > >> > >> > 5830 University Research Ct.
> >>> > > >> > >> >
> >>> > > >> > >> > College Park, MD 20740
> >>> > > >> > >> >
> >>> > > >> > >> > Binbin.Zhou at noaa.gov
> >>> > > >> > >> >
> >>> > > >> > >> > 301-683-3683
> >>> > > >> > >> >
> >>> > > >> > >> >
> >>> > > >> > >>
> >>> > > >> > >>
> >>> > > >> > >
> >>> > > >> > > --
> >>> > > >> > >
> >>> > > >> > > Binbin Zhou
> >>> > > >> > >
> >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> >>> > > >> > >
> >>> > > >> > > 5830 University Research Ct.
> >>> > > >> > >
> >>> > > >> > > College Park, MD 20740
> >>> > > >> > >
> >>> > > >> > > Binbin.Zhou at noaa.gov
> >>> > > >> > >
> >>> > > >> > > 301-683-3683
> >>> > > >> > >
> >>> > > >> >
> >>> > > >> >
> >>> > > >> > --
> >>> > > >> >
> >>> > > >> > Binbin Zhou
> >>> > > >> >
> >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> >>> > > >> >
> >>> > > >> > 5830 University Research Ct.
> >>> > > >> >
> >>> > > >> > College Park, MD 20740
> >>> > > >> >
> >>> > > >> > Binbin.Zhou at noaa.gov
> >>> > > >> >
> >>> > > >> > 301-683-3683
> >>> > > >> >
> >>> > > >> >
> >>> > > >>
> >>> > > >>
> >>> > > >
> >>> > > > --
> >>> > > >
> >>> > > > Binbin Zhou
> >>> > > >
> >>> > > > IMSG at NOAA/NWS/NCEP/EMC
> >>> > > >
> >>> > > > 5830 University Research Ct.
> >>> > > >
> >>> > > > College Park, MD 20740
> >>> > > >
> >>> > > > Binbin.Zhou at noaa.gov
> >>> > > >
> >>> > > > 301-683-3683
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > >
> >>> > > Binbin Zhou
> >>> > >
> >>> > > IMSG at NOAA/NWS/NCEP/EMC
> >>> > >
> >>> > > 5830 University Research Ct.
> >>> > >
> >>> > > College Park, MD 20740
> >>> > >
> >>> > > Binbin.Zhou at noaa.gov
> >>> > >
> >>> > > 301-683-3683
> >>> > >
> >>> > >
> >>> >
> >>> >
> >>>
> >>> --
> >>>
> >>> Binbin Zhou
> >>>
> >>> IMSG at NOAA/NWS/NCEP/EMC
> >>>
> >>> 5830 University Research Ct.
> >>>
> >>> College Park, MD 20740
> >>>
> >>> Binbin.Zhou at noaa.gov
> >>>
> >>> 301-683-3683
> >>>
> >>>
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Julie Prestopnik
Time: Thu Dec 10 11:59:29 2020

Thanks, John.  That worked great.

Binbin, this version of MET is now accessible to you by running the
following on venus:

module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
module load met/ens_config_test

Please let us know if you encounter any problems.

Julie

On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> Julie,
>
> Building the tarfile is very easy. I just ran this script from the
MET repo
> and specified the branch to be built:
>    MET/scripts/met_checkout_and_build.sh branch
feature_1604_ens_config
>
> And that created a tarfile which I posted to RAL's anonymous ftp
server:
>
>
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
>
> I assume that'll work for you.
>
> Thanks,
> John
>
> On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > John,
> >
> > I am able to help with this install on WCOSS.
> >
> > Would it be easier to use git to checkout the repo on WCOSS? Or
would it
> be
> > > easier if I built a tarfile for this version of the code?
> > >
> > The last time I tried to build from git on WCOSS (a while ago), I
> > couldn't.  I don't remember all of the details, but I think I
couldn't
> run
> > bootstrap and maybe it had to do with the version of autoconf
being too
> > old?  I'm happy to try it again if you'd like though.  Otherwise,
I know
> a
> > tarfile would work, but I know that is extra work for you.
> >
> > Julie
> >
> > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Julie,
> > >
> > > I have created a bugfix branch for main_v9.1 named
> > > "feature_1604_ens_config" and it's up on GitHub.
> > >
> > > Only a single file has changed in it. But I need to get this
version
> > > compiled on WCOSS with which Binbin can test.
> > >
> > > Are you to help me with this? Would it be easier to use git to
checkout
> > > the repo on WCOSS? Or would it be easier if I built a tarfile
for this
> > > version of the code?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > >> Binbin,
> > >>
> > >> In order for you to test this change out, we'd need to
recompile a
> > >> version of the code on WCOSS. I wrote this up as a GitHub
issue:
> > >> https://github.com/dtcenter/MET/issues/1604
> > >>
> > >> And I'll create a branch from main_v9.1 named
"bugfix_1604_ens_config"
> > >> and commit the change there. Then we can checkout that code on
WCOSS,
> > >> compile it, and you can test it.
> > >>
> > >> Yes, you're right, if the fcst fields differ from the climo
fields,
> > you'd
> > >> need to specify them separately. That's pretty easy in the
> Ensemble-Stat
> > >> config file... but are you asking how to control that via
METplus?
> > >>
> > >> John
> > >>
> > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >>>
> > >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > >>>
> > >>> John,
> > >>>
> > >>>   Great! Let me test it at first before make change in MET9.1
> > >>> Another question:
> > >>>    Since climatology data is still in NCEP's grib2 format, In
the
> > >>> ensemble_stat
> > >>> config file,  climo  entry settings  should be different from
those
> of
> > >>> fcst, obs entries?
> > >>> How to set climo (name, level, etc.)?
> > >>>
> > >>> Binbin
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> > >>> met_help at ucar.edu>
> > >>> wrote:
> > >>>
> > >>> > Binbin,
> > >>> >
> > >>> > Success! When I recompile with that one line commented out,
> > >>> plot_data_plane
> > >>> > is able to filter down all those records to the single one
that we
> > >>> want.
> > >>> > Note in the example below that I used the "lead_time" and
> "GRIB_ens"
> > >>> > options to filter the input records down.
> > >>> >
> > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
> > >>> level="P500";
> > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found
exact match
> > for
> > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> > >>> "ensposte.t00z.z500hr".*
> > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
> records
> > >>> > matching VarInfo "HGT/P500" in GRIB file
"ensposte.t00z.z500hr".
> > >>> > DEBUG 1: Creating postscript file: plot.ps
> > >>> >
> > >>> > Also note that this selects out record number 26.
> > >>> >
> > >>> > Shall I make this change as a "bugfix" for met-9.1...which
could be
> > >>> > included in a met-9.1.2 bugfix release? Or shall I just
include it
> in
> > >>> > development for the next release (planned for March 2021)?
> > >>> >
> > >>> > Thanks,
> > >>> > John
> > >>> >
> > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT <
> > >>> > met_help at ucar.edu> wrote:
> > >>> >
> > >>> > >
> > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >>> > >
> > >>> > > John,
> > >>> > >
> > >>> > >   If this removing impacts other applications, or takes
some time
> > >>> > > to be effective, then won't try.  I will use another
approach,
> such
> > >>> as
> > >>> > > borrow GEFS's grib2 data structure as tank to save EC
ensemble
> > >>> > > data.
> > >>> > >
> > >>> > > Binbin
> > >>> > >
> > >>> > >
> > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
Affiliate <
> > >>> > > binbin.zhou at noaa.gov> wrote:
> > >>> > >
> > >>> > > > John,
> > >>> > > >
> > >>> > > >   Yes, please try to use this approach.
> > >>> > > >
> > >>> > > > Thanks!
> > >>> > > >
> > >>> > > > Binbin
> > >>> > > >
> > >>> > > > Binbin
> > >>> > > >
> > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via RT
<
> > >>> > > > met_help at ucar.edu> wrote:
> > >>> > > >
> > >>> > > >> Binbin,
> > >>> > > >>
> > >>> > > >> I used the "-PDS10" option for wgrib to dump those
records:
> > >>> > > >>
> > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > >>> > > >>
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > >>> > > >> | egrep  "^11:|^26:|^43:"
> > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0
0 1 0
> 0
> > 1
> > >>> 0 0
> > >>> > 0
> > >>> > > 21
> > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0
0 1 0
> 0
> > 1
> > >>> 0 0
> > >>> > 0
> > >>> > > 21
> > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6 0
0 1 0
> 0
> > 1
> > >>> 0 0
> > >>> > 0
> > >>> > > 21
> > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > >>> > > >>
> > >>> > > >> Looking VERY closely, there is a difference in the 42-
nd
> column
> > >>> which
> > >>> > is
> > >>> > > >> the 4-th from the last... records 11, 26, and 43 have
values
> of
> > >>> 3, 1,
> > >>> > > and
> > >>> > > >> 2
> > >>> > > >> respectively.
> > >>> > > >>
> > >>> > > >> On this page (
> > >>> > >
https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > >>> > > >> ),
> > >>> > > >> byte 42 = 1 means "unperturbed control forecast".
> > >>> > > >>
> > >>> > > >> So we *should* be able to use the "GRIB_ens" config
option to
> > >>> > > >> differentiate
> > >>> > > >> between those records. BUT when I try, it doesn't work.
And
> > that's
> > >>> > > because
> > >>> > > >> of this line:
> > >>> > > >>
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > >>> > > >>
> > >>> > > >> We only apply those filtering options to GRIB files
created by
> > >>> NCEP.
> > >>> > If
> > >>> > > >> you'd like, I could try removing that condition to see
if we
> can
> > >>> pick
> > >>> > > out
> > >>> > > >> the single record of interest.
> > >>> > > >>
> > >>> > > >> John
> > >>> > > >>
> > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov via
RT <
> > >>> > > >> met_help at ucar.edu> wrote:
> > >>> > > >>
> > >>> > > >> >
> > >>> > > >> > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >
> > >>> > > >> >
> > >>> > > >> > John,
> > >>> > > >> >
> > >>> > > >> >   Look at my
> > >>> > > >> > directory
> > >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is only
for 500
> > >>> HGT.
> > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
> > >>> > > >> > I get 3 records:
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >>> > > >> > mb:anl:type=analysis:NAve=0
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >>> > > >> > mb:anl:type=analysis:NAve=0
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > >>> > > >> > mb:anl:type=analysis:NAve=0
> > >>> > > >> >
> > >>> > > >> > From its grib2 file, we know only  record#26 is
analysis
> > low-res
> > >>> > > >> control.
> > >>> > > >> > If you have some way to determine which record is
analysis
> > >>> low-res
> > >>> > > >> control
> > >>> > > >> > from the grib1 file,
> > >>> > > >> > then we can use grib1 directly (don't consider the
size
> > issue).
> > >>> > > >> >
> > >>> > > >> > Binbin
> > >>> > > >> >
> > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
Affiliate
> <
> > >>> > > >> > binbin.zhou at noaa.gov> wrote:
> > >>> > > >> >
> > >>> > > >> > > John,
> > >>> > > >> > >
> > >>> > > >> > >   Grib1 data is too large, another issue is the
low-res
> > >>> control
> > >>> > > >> analysis
> > >>> > > >> > > data can not be retrieved
> > >>> > > >> > > from the grib1 ecmwf data files. If using wgrib, we
can
> see
> > 3
> > >>> > > analysis
> > >>> > > >> > > fields, but don't know
> > >>> > > >> > > which is low-res control. After convert to grib2,
wgrib2
> can
> > >>> show
> > >>> > > >> which
> > >>> > > >> > > record is low-res control.
> > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set on
WCOSS.
> > >>> > > >> > >
> > >>> > > >> > > Binbin
> > >>> > > >> > >
> > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway
via RT <
> > >>> > > >> > > met_help at ucar.edu> wrote:
> > >>> > > >> > >
> > >>> > > >> > >> Binbin,
> > >>> > > >> > >>
> > >>> > > >> > >> Sounds like there's a lot of moving parts there.
How
> would
> > >>> you
> > >>> > like
> > >>> > > >> to
> > >>> > > >> > >> proceed?
> > >>> > > >> > >>
> > >>> > > >> > >> Should we check to see if MET can read those ECMWF
GRIB1
> > >>> files
> > >>> > > >> directly
> > >>> > > >> > >> without needing to reformat?
> > >>> > > >> > >> Or do you just want to create a custom GRIB table
for the
> > >>> result
> > >>> > of
> > >>> > > >> your
> > >>> > > >> > >> pre-processing and conversion steps?
> > >>> > > >> > >>
> > >>> > > >> > >> John
> > >>> > > >> > >>
> > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
Binbin.Zhou at noaa.gov via
> > RT <
> > >>> > > >> > >> met_help at ucar.edu> wrote:
> > >>> > > >> > >>
> > >>> > > >> > >> >
> > >>> > > >> > >> > <URL:
> > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >>> > >
> > >>> > > >> > >> >
> > >>> > > >> > >> > John,
> > >>> > > >> > >> >
> > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
converted
> from
> > >>> its
> > >>> > > grib1
> > >>> > > >> > >> format
> > >>> > > >> > >> > data.
> > >>> > > >> > >> > Before conversion, the parameter names are
there, but
> > after
> > >>> > > >> > conversion,
> > >>> > > >> > >> the
> > >>> > > >> > >> > names
> > >>> > > >> > >> > are gone. It is because of unknown tables. The
grib1
> data
> > >>> is
> > >>> > > still
> > >>> > > >> > >> > original. The original
> > >>> > > >> > >> > grib1 data files are very strange data format we
can
> not
> > >>> read.
> > >>> > So
> > >>> > > >> we
> > >>> > > >> > >> have
> > >>> > > >> > >> > to use
> > >>> > > >> > >> > a Frotran code to re-arrange the fields and save
them
> as
> > >>> > > NCEP-like
> > >>> > > >> > grib1
> > >>> > > >> > >> > data files.
> > >>> > > >> > >> > So after several processes, some important info
like
> > tables
> > >>> > could
> > >>> > > >> be
> > >>> > > >> > >> lost.
> > >>> > > >> > >> >
> > >>> > > >> > >> >    I'll test your first suggestion since the
field
> record
> > >>> order
> > >>> > > is
> > >>> > > >> > >> fixed.
> > >>> > > >> > >> > Currently we still
> > >>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we
will get
> > >>> grib2
> > >>> > > data
> > >>> > > >> > >> > directly. So before
> > >>> > > >> > >> > we can get original grib2 files from ECMWF, we
just use
> > >>> this
> > >>> > > >> temporal
> > >>> > > >> > >> > solution.
> > >>> > > >> > >> > '
> > >>> > > >> > >> > Thanks!
> > >>> > > >> > >> >
> > >>> > > >> > >> >
> > >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley
Gotway via
> RT
> > <
> > >>> > > >> > >> > met_help at ucar.edu>
> > >>> > > >> > >> > wrote:
> > >>> > > >> > >> >
> > >>> > > >> > >> > > Binbin,
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > OK, working on Venus.
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > You have a typo in the command you sent.
> > >>> > > >> > >> > >  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 a.ps
> > >>> > > >> > >> > >
> > >>> > > >> > >>
> > >>> > > >>
> > >>> >
> > >>>
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >>> > > >> > >> > -v
> > >>> > > >> > >> > > 3
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
> > "GRIB2_parm_cat".
> > >>> > After
> > >>> > > >> > >> > correcting
> > >>> > > >> > >> > > that I did still get an error but it's a very
useful
> > one!
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > module use
> > >>> > > >> > >>
> > /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > >>> > > >> > >> > > module load met/9.1
> > >>> > > >> > >> > > module load grib_util/1.1.1
> > >>> > > >> > >> > > cd
> > >>> > > >> > >> > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> >
> > >>> > > >> > >>
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > >>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/
> > plot.ps
> > >>> > > >> > >> > 'GRIB2_disc=255;
> > >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3; GRIB2_parm=5;
> > >>> level="P500";'
> > >>> > > >> > >> > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict() ->
no
> > >>> parameter
> > >>> > > found
> > >>> > > >> > with
> > >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
> GRIB2_parm
> > >>> (5).
> > >>> > > Use
> > >>> > > >> the
> > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to define
custom
> > >>> GRIB
> > >>> > > >> > >> tables.ERROR
> > >>> > > >> > >> > :
> > >>> > > >> > >> > > *
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > So this table is not yet defined in MET. I
checked
> > their
> > >>> > > website:
> > >>> > > >> > >> > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> >
> > >>> > > >> > >>
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > But I don't see an entry defined for
GRIB2_parm=5;
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > 3 things...
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > (1) As a crude temporary solution, you could
refer to
> > it
> > >>> by
> > >>> > > it's
> > >>> > > >> > >> record
> > >>> > > >> > >> > > number. Let's assume you know that record 1
has
> > >>> temperature
> > >>> > > data:
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > plot_data_plane ecme.ens01.t00z.grd3.f12.grib2
~/
> > plot.ps
> > >>> > > >> > 'name="TMP";
> > >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > Here I'm reading record 1, naming it TMP and
the
> using
> > >>> > > >> > set_attr_level
> > >>> > > >> > >> > write
> > >>> > > >> > >> > > the output level as P500 instead of R1.
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > (2) You could create a custom GRIB table
yourself
> > >>> locally,
> > >>> > and
> > >>> > > >> then
> > >>> > > >> > >> set
> > >>> > > >> > >> > the
> > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to point
MET to
> > it.
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > (3) The best solution is contacting ECMWF and
> > requesting
> > >>> they
> > >>> > > >> send
> > >>> > > >> > >> over
> > >>> > > >> > >> > the
> > >>> > > >> > >> > > GRIB2 table we should be using to decode this
data.
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > Thanks,
> > >>> > > >> > >> > > John
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley
Gotway <
> > >>> > > >> johnhg at ucar.edu>
> > >>> > > >> > >> > wrote:
> > >>> > > >> > >> > >
> > >>> > > >> > >> > > > We might already have them. I'm not sure.
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > > > But there's an additional required piece of
the
> > puzzle.
> > >>> > > Binbin
> > >>> > > >> > runs
> > >>> > > >> > >> > > > "wgrib2" to figure out what "name" he should
use in
> > >>> the MET
> > >>> > > >> config
> > >>> > > >> > >> > files.
> > >>> > > >> > >> > > > If wgrib2 doesn't tell him the right "name",
the
> > >>> problem
> > >>> > will
> > >>> > > >> > >> remain.
> > >>> > > >> > >> > > > wgrib2 can be configured to define
additional
> tables
> > as
> > >>> > well,
> > >>> > > >> but
> > >>> > > >> > >> for
> > >>> > > >> > >> > > this
> > >>> > > >> > >> > > > to be easy for Binbin, that'd mean updating
the
> > script
> > >>> > that's
> > >>> > > >> run
> > >>> > > >> > >> for
> > >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
guessing
> > >>> that's
> > >>> > > not
> > >>> > > >> > very
> > >>> > > >> > >> > easy.
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > > > Yes, I do agree that the best solution is
getting
> the
> > >>> ECMWF
> > >>> > > >> GRIB
> > >>> > > >> > >> tables
> > >>> > > >> > >> > > > and adding them for both wgrib2 and MET.
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > > > John
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen <
> > >>> > jensen at ucar.edu>
> > >>> > > >> > wrote:
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > > >> John,
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> It seems to me that it would be worthwhile
to just
> > ask
> > >>> > ECMWF
> > >>> > > >> for
> > >>> > > >> > >> their
> > >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.  And
other
> > >>> > operational
> > >>> > > >> > >> centers
> > >>> > > >> > >> > for
> > >>> > > >> > >> > > >> that matter.  Do you agree this would be a
good
> > >>> solution?
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Thanks, Tara
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> ---------- Forwarded message ---------
> > >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
> > met_help at ucar.edu>
> > >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769]
urgent
> > >>> question for
> > >>> > > >> > reading
> > >>> > > >> > >> > ECMWF
> > >>> > > >> > >> > > >> ensmeble data
> > >>> > > >> > >> > > >> To:
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> <URL:
> > >>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >>> > > >> >
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> John,
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>   I tested using following command:
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > a.ps
> > >>> > > >> > >> > > >>
> > >>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >>> > > >> -v3
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> ERROR  :
> > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name
> or
> > >>> > > >> GRIB2_disc,
> > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified in
> > >>> field
> > >>> > > >> > >> information
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > a.ps
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> >
> > >>> > > >> >
> > >>> > >
> > >>>
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > >>> > > >> > >> > > >> -v
> > >>> > > >> > >> > > >> 3
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Still got error
> > >>> > > >> > >> > > >> ERROR  :
> > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() -> either
name
> or
> > >>> > > >> GRIB2_disc,
> > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified in
> > >>> field
> > >>> > > >> > >> information
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Looks like discipline=255 is not recognized
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Binbin
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John Halley
Gotway
> > >>> via RT
> > >>> > <
> > >>> > > >> > >> > > >> met_help at ucar.edu>
> > >>> > > >> > >> > > >> wrote:
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> > Binbin,
> > >>> > > >> > >> > > >> > You can do something similar with MET.
Please go
> > to
> > >>> this
> > >>> > > >> link:
> > >>> > > >> > >> > > >> >
> > >>> > > >>
> https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and
you'll
> find
> > >>> the
> > >>> > > >> content
> > >>> > > >> > >> > > >> > copied/pasted below. You can specify
these
> INSTEAD
> > >>> OF
> > >>> > > >> > specifying
> > >>> > > >> > >> a
> > >>> > > >> > >> > > >> > requested name.
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an integer
> > >>> specifying the
> > >>> > > >> master
> > >>> > > >> > >> > table
> > >>> > > >> > >> > > >> > number.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an integer
> > >>> specifying the
> > >>> > > >> local
> > >>> > > >> > >> table
> > >>> > > >> > >> > > >> > number.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an integer
> > >>> specifying the
> > >>> > > >> GRIB2
> > >>> > > >> > >> > > >> discipline
> > >>> > > >> > >> > > >> > code.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an
integer
> > >>> specifying
> > >>> > > the
> > >>> > > >> > >> > parameter
> > >>> > > >> > >> > > >> > category
> > >>> > > >> > >> > > >> > //         code.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an integer
> > >>> specifying the
> > >>> > > >> > >> parameter
> > >>> > > >> > >> > > code.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an integer
> > specifying
> > >>> the
> > >>> > > >> product
> > >>> > > >> > >> > > >> definition
> > >>> > > >> > >> > > >> > //         template (Table 4.0).
> > >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
integer
> > >>> specifying
> > >>> > > the
> > >>> > > >> > >> > generating
> > >>> > > >> > >> > > >> > process
> > >>> > > >> > >> > > >> > //         (Table 4.3).
> > >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an integer
> > >>> specifying the
> > >>> > > >> > >> originating
> > >>> > > >> > >> > > >> > center.
> > >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an
integer
> > >>> specifying
> > >>> > > the
> > >>> > > >> > >> > ensemble
> > >>> > > >> > >> > > >> type
> > >>> > > >> > >> > > >> > //         (Table 4.6).
> > >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an
integer
> > >>> specifying
> > >>> > > the
> > >>> > > >> > >> derived
> > >>> > > >> > >> > > >> > product
> > >>> > > >> > >> > > >> > //         type (Table 4.7).
> > >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
> > >>> > specifying
> > >>> > > >> the
> > >>> > > >> > >> > > >> statistical
> > >>> > > >> > >> > > >> > //         processing type (Table 4.10).
> > >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index" and
> > >>> > > "GRIB2_ipdtmpl_val"
> > >>> > > >> > >> entries
> > >>> > > >> > >> > > are
> > >>> > > >> > >> > > >> > arrays
> > >>> > > >> > >> > > >> > //         of integers which specify the
product
> > >>> > > description
> > >>> > > >> > >> > template
> > >>> > > >> > >> > > >> > values to
> > >>> > > >> > >> > > >> > //         be used. The indices are 0-
based. For
> > >>> > example,
> > >>> > > >> use
> > >>> > > >> > the
> > >>> > > >> > >> > > >> > following to
> > >>> > > >> > >> > > >> > //         request a GRIB2 record whose
9-th and
> > >>> 27-th
> > >>> > > >> product
> > >>> > > >> > >> > > >> description
> > >>> > > >> > >> > > >> > //         template values are 1 and 2,
> > >>> respectively:
> > >>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8,
26];
> > >>> > > >> > GRIB2_ipdtmpl_val=[1,
> > >>> > > >> > >> 2];
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > While I don't have any ECMWF data on hand
for
> > >>> testing, I
> > >>> > > do
> > >>> > > >> > have
> > >>> > > >> > >> a
> > >>> > > >> > >> > > file
> > >>> > > >> > >> > > >> > from the Air Force from which wgrib2
prints a
> > >>> similar
> > >>> > > >> result:
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var
discipline=0
> > >>> center=57
> > >>> > > >> > >> local_table=1
> > >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour fcst:
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > So I used that for testing. I ran the
following
> > >>> > > >> plot_data_plane
> > >>> > > >> > >> > > command
> > >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > plot_data_plane
GALWEM_20170610_CY00_FH18.GR2
> > >>> plot.ps
> > >>> > > >> > >> > 'GRIB2_disc=0;
> > >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
> > >>> > > >> GRIB2_parm=192;
> > >>> > > >> > >> > > >> level="Z0";'
> > >>> > > >> > >> > > >> > -v 3
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > And then I looked closely at the log
message to
> > >>> confirm
> > >>> > > >> that it
> > >>> > > >> > >> used
> > >>> > > >> > >> > > >> > record 549, as expected:
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane()
-> Found
> > >>> exact
> > >>> > > match
> > >>> > > >> > for
> > >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1 of
GRIB2
> > file
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > >
> > >>> > > >> > >> >
> > >>> > > >> > >>
> > >>> > > >> >
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I defined
to do
> a
> > >>> table
> > >>> > > >> lookup
> > >>> > > >> > >> and
> > >>> > > >> > >> > > find
> > >>> > > >> > >> > > >> a
> > >>> > > >> > >> > > >> > match in the MET GRIB tables:
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >>
> > >>> > >
> >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > And it determined that that field should
be
> called
> > >>> > > "CTSTM".
> > >>> > > >> > >> > Hopefully,
> > >>> > > >> > >> > > >> the
> > >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined
there in
> > >>> MET. If
> > >>> > > >> not, we
> > >>> > > >> > >> can
> > >>> > > >> > >> > > >> always
> > >>> > > >> > >> > > >> > add them.
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > Please try running plot_data_plane at
verbosity
> > >>> level 3
> > >>> > on
> > >>> > > >> one
> > >>> > > >> > of
> > >>> > > >> > >> > > these
> > >>> > > >> > >> > > >> > files first. And make sure it runs
without error
> > >>> and the
> > >>> > > log
> > >>> > > >> > >> message
> > >>> > > >> > >> > > >> prints
> > >>> > > >> > >> > > >> > out the "name" it found for that data. If
so,
> you
> > >>> can
> > >>> > just
> > >>> > > >> skip
> > >>> > > >> > >> > using
> > >>> > > >> > >> > > >> all
> > >>> > > >> > >> > > >> > those integers and revert back to using
the name
> > MET
> > >>> > > knows.
> > >>> > > >> If
> > >>> > > >> > >> not,
> > >>> > > >> > >> > > >> then we
> > >>> > > >> > >> > > >> > may need to define a custom GRIB table
for this
> > >>> ECMWF
> > >>> > > GRIB2
> > >>> > > >> > data.
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >> > Thanks,
> > >>> > > >> > >> > > >> > John
> > >>> > > >> > >> > > >> >
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> --
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Binbin Zhou
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> 5830 University Research Ct.
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> College Park, MD 20740
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> 301-683-3683
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >> --
> > >>> > > >> > >> > > >>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >>> > > >> > >> > > >> Tara Jensen
> > >>> > > >> > >> > > >> Project Manager II
> > >>> > > >> > >> > > >> NCAR RAL and DTC
> > >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > >>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > >>> > > >> > >> > > >>
> > >>> > > >> > >> > > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> > >
> > >>> > > >> > >> >
> > >>> > > >> > >> > --
> > >>> > > >> > >> >
> > >>> > > >> > >> > Binbin Zhou
> > >>> > > >> > >> >
> > >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >>> > > >> > >> >
> > >>> > > >> > >> > 5830 University Research Ct.
> > >>> > > >> > >> >
> > >>> > > >> > >> > College Park, MD 20740
> > >>> > > >> > >> >
> > >>> > > >> > >> > Binbin.Zhou at noaa.gov
> > >>> > > >> > >> >
> > >>> > > >> > >> > 301-683-3683
> > >>> > > >> > >> >
> > >>> > > >> > >> >
> > >>> > > >> > >>
> > >>> > > >> > >>
> > >>> > > >> > >
> > >>> > > >> > > --
> > >>> > > >> > >
> > >>> > > >> > > Binbin Zhou
> > >>> > > >> > >
> > >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > >>> > > >> > >
> > >>> > > >> > > 5830 University Research Ct.
> > >>> > > >> > >
> > >>> > > >> > > College Park, MD 20740
> > >>> > > >> > >
> > >>> > > >> > > Binbin.Zhou at noaa.gov
> > >>> > > >> > >
> > >>> > > >> > > 301-683-3683
> > >>> > > >> > >
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >> > --
> > >>> > > >> >
> > >>> > > >> > Binbin Zhou
> > >>> > > >> >
> > >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >>> > > >> >
> > >>> > > >> > 5830 University Research Ct.
> > >>> > > >> >
> > >>> > > >> > College Park, MD 20740
> > >>> > > >> >
> > >>> > > >> > Binbin.Zhou at noaa.gov
> > >>> > > >> >
> > >>> > > >> > 301-683-3683
> > >>> > > >> >
> > >>> > > >> >
> > >>> > > >>
> > >>> > > >>
> > >>> > > >
> > >>> > > > --
> > >>> > > >
> > >>> > > > Binbin Zhou
> > >>> > > >
> > >>> > > > IMSG at NOAA/NWS/NCEP/EMC
> > >>> > > >
> > >>> > > > 5830 University Research Ct.
> > >>> > > >
> > >>> > > > College Park, MD 20740
> > >>> > > >
> > >>> > > > Binbin.Zhou at noaa.gov
> > >>> > > >
> > >>> > > > 301-683-3683
> > >>> > > >
> > >>> > >
> > >>> > >
> > >>> > > --
> > >>> > >
> > >>> > > Binbin Zhou
> > >>> > >
> > >>> > > IMSG at NOAA/NWS/NCEP/EMC
> > >>> > >
> > >>> > > 5830 University Research Ct.
> > >>> > >
> > >>> > > College Park, MD 20740
> > >>> > >
> > >>> > > Binbin.Zhou at noaa.gov
> > >>> > >
> > >>> > > 301-683-3683
> > >>> > >
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> > >>> --
> > >>>
> > >>> Binbin Zhou
> > >>>
> > >>> IMSG at NOAA/NWS/NCEP/EMC
> > >>>
> > >>> 5830 University Research Ct.
> > >>>
> > >>> College Park, MD 20740
> > >>>
> > >>> Binbin.Zhou at noaa.gov
> > >>>
> > >>> 301-683-3683
> > >>>
> > >>>
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Thu Dec 10 12:05:57 2020

Thanks Julie.

And Binbin, here's an EnsembleStatConfig file where climos are used:

https://github.com/dtcenter/MET/blob/main_v9.1/test/config/EnsembleStatConfig_climo

Since the list of forecast fields is exactly the same as the climo
fields,
this config file just copies them over using:
   climo_mean = fcst;

In your case, when they differ, you'd just specify the "field" array
inside
the "climo_mean" dictionary. For example:

climo_mean = {

   field = [
      { name  = "TMP"; level = "Z2";   },
      { name  = "TMP"; level = "P850"; }
   ];
   ...
}

On Thu, Dec 10, 2020 at 12:00 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> Thanks, John.  That worked great.
>
> Binbin, this version of MET is now accessible to you by running the
> following on venus:
>
> module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> module load met/ens_config_test
>
> Please let us know if you encounter any problems.
>
> Julie
>
> On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > Julie,
> >
> > Building the tarfile is very easy. I just ran this script from the
MET
> repo
> > and specified the branch to be built:
> >    MET/scripts/met_checkout_and_build.sh branch
feature_1604_ens_config
> >
> > And that created a tarfile which I posted to RAL's anonymous ftp
server:
> >
> >
> >
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
> >
> > I assume that'll work for you.
> >
> > Thanks,
> > John
> >
> > On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >
> > > John,
> > >
> > > I am able to help with this install on WCOSS.
> > >
> > > Would it be easier to use git to checkout the repo on WCOSS? Or
would
> it
> > be
> > > > easier if I built a tarfile for this version of the code?
> > > >
> > > The last time I tried to build from git on WCOSS (a while ago),
I
> > > couldn't.  I don't remember all of the details, but I think I
couldn't
> > run
> > > bootstrap and maybe it had to do with the version of autoconf
being too
> > > old?  I'm happy to try it again if you'd like though.
Otherwise, I
> know
> > a
> > > tarfile would work, but I know that is extra work for you.
> > >
> > > Julie
> > >
> > > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Julie,
> > > >
> > > > I have created a bugfix branch for main_v9.1 named
> > > > "feature_1604_ens_config" and it's up on GitHub.
> > > >
> > > > Only a single file has changed in it. But I need to get this
version
> > > > compiled on WCOSS with which Binbin can test.
> > > >
> > > > Are you to help me with this? Would it be easier to use git to
> checkout
> > > > the repo on WCOSS? Or would it be easier if I built a tarfile
for
> this
> > > > version of the code?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > > >
> > > >> Binbin,
> > > >>
> > > >> In order for you to test this change out, we'd need to
recompile a
> > > >> version of the code on WCOSS. I wrote this up as a GitHub
issue:
> > > >> https://github.com/dtcenter/MET/issues/1604
> > > >>
> > > >> And I'll create a branch from main_v9.1 named
> "bugfix_1604_ens_config"
> > > >> and commit the change there. Then we can checkout that code
on
> WCOSS,
> > > >> compile it, and you can test it.
> > > >>
> > > >> Yes, you're right, if the fcst fields differ from the climo
fields,
> > > you'd
> > > >> need to specify them separately. That's pretty easy in the
> > Ensemble-Stat
> > > >> config file... but are you asking how to control that via
METplus?
> > > >>
> > > >> John
> > > >>
> > > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >>>
> > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > > >>>
> > > >>> John,
> > > >>>
> > > >>>   Great! Let me test it at first before make change in
MET9.1
> > > >>> Another question:
> > > >>>    Since climatology data is still in NCEP's grib2 format,
In the
> > > >>> ensemble_stat
> > > >>> config file,  climo  entry settings  should be different
from those
> > of
> > > >>> fcst, obs entries?
> > > >>> How to set climo (name, level, etc.)?
> > > >>>
> > > >>> Binbin
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> > > >>> met_help at ucar.edu>
> > > >>> wrote:
> > > >>>
> > > >>> > Binbin,
> > > >>> >
> > > >>> > Success! When I recompile with that one line commented
out,
> > > >>> plot_data_plane
> > > >>> > is able to filter down all those records to the single one
that
> we
> > > >>> want.
> > > >>> > Note in the example below that I used the "lead_time" and
> > "GRIB_ens"
> > > >>> > options to filter the input records down.
> > > >>> >
> > > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
> > > >>> level="P500";
> > > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found
exact
> match
> > > for
> > > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> > > >>> "ensposte.t00z.z500hr".*
> > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
> > records
> > > >>> > matching VarInfo "HGT/P500" in GRIB file
"ensposte.t00z.z500hr".
> > > >>> > DEBUG 1: Creating postscript file: plot.ps
> > > >>> >
> > > >>> > Also note that this selects out record number 26.
> > > >>> >
> > > >>> > Shall I make this change as a "bugfix" for met-9.1...which
could
> be
> > > >>> > included in a met-9.1.2 bugfix release? Or shall I just
include
> it
> > in
> > > >>> > development for the next release (planned for March 2021)?
> > > >>> >
> > > >>> > Thanks,
> > > >>> > John
> > > >>> >
> > > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT
<
> > > >>> > met_help at ucar.edu> wrote:
> > > >>> >
> > > >>> > >
> > > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> > > >>> > >
> > > >>> > > John,
> > > >>> > >
> > > >>> > >   If this removing impacts other applications, or takes
some
> time
> > > >>> > > to be effective, then won't try.  I will use another
approach,
> > such
> > > >>> as
> > > >>> > > borrow GEFS's grib2 data structure as tank to save EC
ensemble
> > > >>> > > data.
> > > >>> > >
> > > >>> > > Binbin
> > > >>> > >
> > > >>> > >
> > > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
Affiliate <
> > > >>> > > binbin.zhou at noaa.gov> wrote:
> > > >>> > >
> > > >>> > > > John,
> > > >>> > > >
> > > >>> > > >   Yes, please try to use this approach.
> > > >>> > > >
> > > >>> > > > Thanks!
> > > >>> > > >
> > > >>> > > > Binbin
> > > >>> > > >
> > > >>> > > > Binbin
> > > >>> > > >
> > > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via
RT <
> > > >>> > > > met_help at ucar.edu> wrote:
> > > >>> > > >
> > > >>> > > >> Binbin,
> > > >>> > > >>
> > > >>> > > >> I used the "-PDS10" option for wgrib to dump those
records:
> > > >>> > > >>
> > > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > > >>> > > >> | egrep  "^11:|^26:|^43:"
> > > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > > >>> > > >>
> > > >>> > > >> Looking VERY closely, there is a difference in the
42-nd
> > column
> > > >>> which
> > > >>> > is
> > > >>> > > >> the 4-th from the last... records 11, 26, and 43 have
values
> > of
> > > >>> 3, 1,
> > > >>> > > and
> > > >>> > > >> 2
> > > >>> > > >> respectively.
> > > >>> > > >>
> > > >>> > > >> On this page (
> > > >>> > >
https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > > >>> > > >> ),
> > > >>> > > >> byte 42 = 1 means "unperturbed control forecast".
> > > >>> > > >>
> > > >>> > > >> So we *should* be able to use the "GRIB_ens" config
option
> to
> > > >>> > > >> differentiate
> > > >>> > > >> between those records. BUT when I try, it doesn't
work. And
> > > that's
> > > >>> > > because
> > > >>> > > >> of this line:
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > > >>> > > >>
> > > >>> > > >> We only apply those filtering options to GRIB files
created
> by
> > > >>> NCEP.
> > > >>> > If
> > > >>> > > >> you'd like, I could try removing that condition to
see if we
> > can
> > > >>> pick
> > > >>> > > out
> > > >>> > > >> the single record of interest.
> > > >>> > > >>
> > > >>> > > >> John
> > > >>> > > >>
> > > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov
via RT
> <
> > > >>> > > >> met_help at ucar.edu> wrote:
> > > >>> > > >>
> > > >>> > > >> >
> > > >>> > > >> > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >
> > > >>> > > >> >
> > > >>> > > >> > John,
> > > >>> > > >> >
> > > >>> > > >> >   Look at my
> > > >>> > > >> > directory
> > > >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is
only for
> 500
> > > >>> HGT.
> > > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
> > > >>> > > >> > I get 3 records:
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> > From its grib2 file, we know only  record#26 is
analysis
> > > low-res
> > > >>> > > >> control.
> > > >>> > > >> > If you have some way to determine which record is
analysis
> > > >>> low-res
> > > >>> > > >> control
> > > >>> > > >> > from the grib1 file,
> > > >>> > > >> > then we can use grib1 directly (don't consider the
size
> > > issue).
> > > >>> > > >> >
> > > >>> > > >> > Binbin
> > > >>> > > >> >
> > > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
> Affiliate
> > <
> > > >>> > > >> > binbin.zhou at noaa.gov> wrote:
> > > >>> > > >> >
> > > >>> > > >> > > John,
> > > >>> > > >> > >
> > > >>> > > >> > >   Grib1 data is too large, another issue is the
low-res
> > > >>> control
> > > >>> > > >> analysis
> > > >>> > > >> > > data can not be retrieved
> > > >>> > > >> > > from the grib1 ecmwf data files. If using wgrib,
we can
> > see
> > > 3
> > > >>> > > analysis
> > > >>> > > >> > > fields, but don't know
> > > >>> > > >> > > which is low-res control. After convert to grib2,
wgrib2
> > can
> > > >>> show
> > > >>> > > >> which
> > > >>> > > >> > > record is low-res control.
> > > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set
on
> WCOSS.
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin
> > > >>> > > >> > >
> > > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway
via
> RT <
> > > >>> > > >> > > met_help at ucar.edu> wrote:
> > > >>> > > >> > >
> > > >>> > > >> > >> Binbin,
> > > >>> > > >> > >>
> > > >>> > > >> > >> Sounds like there's a lot of moving parts there.
How
> > would
> > > >>> you
> > > >>> > like
> > > >>> > > >> to
> > > >>> > > >> > >> proceed?
> > > >>> > > >> > >>
> > > >>> > > >> > >> Should we check to see if MET can read those
ECMWF
> GRIB1
> > > >>> files
> > > >>> > > >> directly
> > > >>> > > >> > >> without needing to reformat?
> > > >>> > > >> > >> Or do you just want to create a custom GRIB
table for
> the
> > > >>> result
> > > >>> > of
> > > >>> > > >> your
> > > >>> > > >> > >> pre-processing and conversion steps?
> > > >>> > > >> > >>
> > > >>> > > >> > >> John
> > > >>> > > >> > >>
> > > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
Binbin.Zhou at noaa.gov
> via
> > > RT <
> > > >>> > > >> > >> met_help at ucar.edu> wrote:
> > > >>> > > >> > >>
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > <URL:
> > > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >>> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > John,
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
converted
> > from
> > > >>> its
> > > >>> > > grib1
> > > >>> > > >> > >> format
> > > >>> > > >> > >> > data.
> > > >>> > > >> > >> > Before conversion, the parameter names are
there, but
> > > after
> > > >>> > > >> > conversion,
> > > >>> > > >> > >> the
> > > >>> > > >> > >> > names
> > > >>> > > >> > >> > are gone. It is because of unknown tables. The
grib1
> > data
> > > >>> is
> > > >>> > > still
> > > >>> > > >> > >> > original. The original
> > > >>> > > >> > >> > grib1 data files are very strange data format
we can
> > not
> > > >>> read.
> > > >>> > So
> > > >>> > > >> we
> > > >>> > > >> > >> have
> > > >>> > > >> > >> > to use
> > > >>> > > >> > >> > a Frotran code to re-arrange the fields and
save them
> > as
> > > >>> > > NCEP-like
> > > >>> > > >> > grib1
> > > >>> > > >> > >> > data files.
> > > >>> > > >> > >> > So after several processes, some important
info like
> > > tables
> > > >>> > could
> > > >>> > > >> be
> > > >>> > > >> > >> lost.
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >    I'll test your first suggestion since the
field
> > record
> > > >>> order
> > > >>> > > is
> > > >>> > > >> > >> fixed.
> > > >>> > > >> > >> > Currently we still
> > > >>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we
will
> get
> > > >>> grib2
> > > >>> > > data
> > > >>> > > >> > >> > directly. So before
> > > >>> > > >> > >> > we can get original grib2 files from ECMWF, we
just
> use
> > > >>> this
> > > >>> > > >> temporal
> > > >>> > > >> > >> > solution.
> > > >>> > > >> > >> > '
> > > >>> > > >> > >> > Thanks!
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley
Gotway via
> > RT
> > > <
> > > >>> > > >> > >> > met_help at ucar.edu>
> > > >>> > > >> > >> > wrote:
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > > Binbin,
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > OK, working on Venus.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > You have a typo in the command you sent.
> > > >>> > > >> > >> > >  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> a.ps
> > > >>> > > >> > >> > >
> > > >>> > > >> > >>
> > > >>> > > >>
> > > >>> >
> > > >>>
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> > >> > -v
> > > >>> > > >> > >> > > 3
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
> > > "GRIB2_parm_cat".
> > > >>> > After
> > > >>> > > >> > >> > correcting
> > > >>> > > >> > >> > > that I did still get an error but it's a
very
> useful
> > > one!
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > module use
> > > >>> > > >> > >>
> > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > >>> > > >> > >> > > module load met/9.1
> > > >>> > > >> > >> > > module load grib_util/1.1.1
> > > >>> > > >> > >> > > cd
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > plot.ps
> > > >>> > > >> > >> > 'GRIB2_disc=255;
> > > >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3;
GRIB2_parm=5;
> > > >>> level="P500";'
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict()
-> no
> > > >>> parameter
> > > >>> > > found
> > > >>> > > >> > with
> > > >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
> > GRIB2_parm
> > > >>> (5).
> > > >>> > > Use
> > > >>> > > >> the
> > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
define
> custom
> > > >>> GRIB
> > > >>> > > >> > >> tables.ERROR
> > > >>> > > >> > >> > :
> > > >>> > > >> > >> > > *
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > So this table is not yet defined in MET. I
checked
> > > their
> > > >>> > > website:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > But I don't see an entry defined for
GRIB2_parm=5;
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > 3 things...
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (1) As a crude temporary solution, you could
refer
> to
> > > it
> > > >>> by
> > > >>> > > it's
> > > >>> > > >> > >> record
> > > >>> > > >> > >> > > number. Let's assume you know that record 1
has
> > > >>> temperature
> > > >>> > > data:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > plot.ps
> > > >>> > > >> > 'name="TMP";
> > > >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > Here I'm reading record 1, naming it TMP and
the
> > using
> > > >>> > > >> > set_attr_level
> > > >>> > > >> > >> > write
> > > >>> > > >> > >> > > the output level as P500 instead of R1.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (2) You could create a custom GRIB table
yourself
> > > >>> locally,
> > > >>> > and
> > > >>> > > >> then
> > > >>> > > >> > >> set
> > > >>> > > >> > >> > the
> > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
point MET
> to
> > > it.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (3) The best solution is contacting ECMWF
and
> > > requesting
> > > >>> they
> > > >>> > > >> send
> > > >>> > > >> > >> over
> > > >>> > > >> > >> > the
> > > >>> > > >> > >> > > GRIB2 table we should be using to decode
this data.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > Thanks,
> > > >>> > > >> > >> > > John
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley
Gotway <
> > > >>> > > >> johnhg at ucar.edu>
> > > >>> > > >> > >> > wrote:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > > We might already have them. I'm not sure.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > But there's an additional required piece
of the
> > > puzzle.
> > > >>> > > Binbin
> > > >>> > > >> > runs
> > > >>> > > >> > >> > > > "wgrib2" to figure out what "name" he
should use
> in
> > > >>> the MET
> > > >>> > > >> config
> > > >>> > > >> > >> > files.
> > > >>> > > >> > >> > > > If wgrib2 doesn't tell him the right
"name", the
> > > >>> problem
> > > >>> > will
> > > >>> > > >> > >> remain.
> > > >>> > > >> > >> > > > wgrib2 can be configured to define
additional
> > tables
> > > as
> > > >>> > well,
> > > >>> > > >> but
> > > >>> > > >> > >> for
> > > >>> > > >> > >> > > this
> > > >>> > > >> > >> > > > to be easy for Binbin, that'd mean
updating the
> > > script
> > > >>> > that's
> > > >>> > > >> run
> > > >>> > > >> > >> for
> > > >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
> guessing
> > > >>> that's
> > > >>> > > not
> > > >>> > > >> > very
> > > >>> > > >> > >> > easy.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > Yes, I do agree that the best solution is
getting
> > the
> > > >>> ECMWF
> > > >>> > > >> GRIB
> > > >>> > > >> > >> tables
> > > >>> > > >> > >> > > > and adding them for both wgrib2 and MET.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > John
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<
> > > >>> > jensen at ucar.edu>
> > > >>> > > >> > wrote:
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > >> John,
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> It seems to me that it would be
worthwhile to
> just
> > > ask
> > > >>> > ECMWF
> > > >>> > > >> for
> > > >>> > > >> > >> their
> > > >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.
And other
> > > >>> > operational
> > > >>> > > >> > >> centers
> > > >>> > > >> > >> > for
> > > >>> > > >> > >> > > >> that matter.  Do you agree this would be
a good
> > > >>> solution?
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Thanks, Tara
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> ---------- Forwarded message ---------
> > > >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769]
urgent
> > > >>> question for
> > > >>> > > >> > reading
> > > >>> > > >> > >> > ECMWF
> > > >>> > > >> > >> > > >> ensmeble data
> > > >>> > > >> > >> > > >> To:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> <URL:
> > > >>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >>> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> John,
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>   I tested using following command:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > > a.ps
> > > >>> > > >> > >> > > >>
> > > >>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> -v3
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> ERROR  :
> > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either name
> > or
> > > >>> > > >> GRIB2_disc,
> > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified
> in
> > > >>> field
> > > >>> > > >> > >> information
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > > a.ps
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> >
> > > >>> > > >> >
> > > >>> > >
> > > >>>
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> > >> > > >> -v
> > > >>> > > >> > >> > > >> 3
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Still got error
> > > >>> > > >> > >> > > >> ERROR  :
> > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either name
> > or
> > > >>> > > >> GRIB2_disc,
> > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified
> in
> > > >>> field
> > > >>> > > >> > >> information
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Looks like discipline=255 is not
recognized
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John
Halley
> Gotway
> > > >>> via RT
> > > >>> > <
> > > >>> > > >> > >> > > >> met_help at ucar.edu>
> > > >>> > > >> > >> > > >> wrote:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> > Binbin,
> > > >>> > > >> > >> > > >> > You can do something similar with MET.
Please
> go
> > > to
> > > >>> this
> > > >>> > > >> link:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >>
> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and
you'll
> > find
> > > >>> the
> > > >>> > > >> content
> > > >>> > > >> > >> > > >> > copied/pasted below. You can specify
these
> > INSTEAD
> > > >>> OF
> > > >>> > > >> > specifying
> > > >>> > > >> > >> a
> > > >>> > > >> > >> > > >> > requested name.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an
integer
> > > >>> specifying the
> > > >>> > > >> master
> > > >>> > > >> > >> > table
> > > >>> > > >> > >> > > >> > number.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an
integer
> > > >>> specifying the
> > > >>> > > >> local
> > > >>> > > >> > >> table
> > > >>> > > >> > >> > > >> > number.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an
integer
> > > >>> specifying the
> > > >>> > > >> GRIB2
> > > >>> > > >> > >> > > >> discipline
> > > >>> > > >> > >> > > >> > code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > parameter
> > > >>> > > >> > >> > > >> > category
> > > >>> > > >> > >> > > >> > //         code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an
integer
> > > >>> specifying the
> > > >>> > > >> > >> parameter
> > > >>> > > >> > >> > > code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an
integer
> > > specifying
> > > >>> the
> > > >>> > > >> product
> > > >>> > > >> > >> > > >> definition
> > > >>> > > >> > >> > > >> > //         template (Table 4.0).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > generating
> > > >>> > > >> > >> > > >> > process
> > > >>> > > >> > >> > > >> > //         (Table 4.3).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an
integer
> > > >>> specifying the
> > > >>> > > >> > >> originating
> > > >>> > > >> > >> > > >> > center.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > ensemble
> > > >>> > > >> > >> > > >> type
> > > >>> > > >> > >> > > >> > //         (Table 4.6).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> derived
> > > >>> > > >> > >> > > >> > product
> > > >>> > > >> > >> > > >> > //         type (Table 4.7).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
> > > >>> > specifying
> > > >>> > > >> the
> > > >>> > > >> > >> > > >> statistical
> > > >>> > > >> > >> > > >> > //         processing type (Table
4.10).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index"
and
> > > >>> > > "GRIB2_ipdtmpl_val"
> > > >>> > > >> > >> entries
> > > >>> > > >> > >> > > are
> > > >>> > > >> > >> > > >> > arrays
> > > >>> > > >> > >> > > >> > //         of integers which specify
the
> product
> > > >>> > > description
> > > >>> > > >> > >> > template
> > > >>> > > >> > >> > > >> > values to
> > > >>> > > >> > >> > > >> > //         be used. The indices are 0-
based.
> For
> > > >>> > example,
> > > >>> > > >> use
> > > >>> > > >> > the
> > > >>> > > >> > >> > > >> > following to
> > > >>> > > >> > >> > > >> > //         request a GRIB2 record whose
9-th
> and
> > > >>> 27-th
> > > >>> > > >> product
> > > >>> > > >> > >> > > >> description
> > > >>> > > >> > >> > > >> > //         template values are 1 and 2,
> > > >>> respectively:
> > > >>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8,
26];
> > > >>> > > >> > GRIB2_ipdtmpl_val=[1,
> > > >>> > > >> > >> 2];
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > While I don't have any ECMWF data on
hand for
> > > >>> testing, I
> > > >>> > > do
> > > >>> > > >> > have
> > > >>> > > >> > >> a
> > > >>> > > >> > >> > > file
> > > >>> > > >> > >> > > >> > from the Air Force from which wgrib2
prints a
> > > >>> similar
> > > >>> > > >> result:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var
discipline=0
> > > >>> center=57
> > > >>> > > >> > >> local_table=1
> > > >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour
fcst:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > So I used that for testing. I ran the
> following
> > > >>> > > >> plot_data_plane
> > > >>> > > >> > >> > > command
> > > >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > plot_data_plane
GALWEM_20170610_CY00_FH18.GR2
> > > >>> plot.ps
> > > >>> > > >> > >> > 'GRIB2_disc=0;
> > > >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
> > > >>> > > >> GRIB2_parm=192;
> > > >>> > > >> > >> > > >> level="Z0";'
> > > >>> > > >> > >> > > >> > -v 3
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And then I looked closely at the log
message
> to
> > > >>> confirm
> > > >>> > > >> that it
> > > >>> > > >> > >> used
> > > >>> > > >> > >> > > >> > record 549, as expected:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane()
->
> Found
> > > >>> exact
> > > >>> > > match
> > > >>> > > >> > for
> > > >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1
of
> GRIB2
> > > file
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I
defined to
> do
> > a
> > > >>> table
> > > >>> > > >> lookup
> > > >>> > > >> > >> and
> > > >>> > > >> > >> > > find
> > > >>> > > >> > >> > > >> a
> > > >>> > > >> > >> > > >> > match in the MET GRIB tables:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >>
> > > >>> > >
> > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And it determined that that field
should be
> > called
> > > >>> > > "CTSTM".
> > > >>> > > >> > >> > Hopefully,
> > > >>> > > >> > >> > > >> the
> > > >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined
there
> in
> > > >>> MET. If
> > > >>> > > >> not, we
> > > >>> > > >> > >> can
> > > >>> > > >> > >> > > >> always
> > > >>> > > >> > >> > > >> > add them.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > Please try running plot_data_plane at
> verbosity
> > > >>> level 3
> > > >>> > on
> > > >>> > > >> one
> > > >>> > > >> > of
> > > >>> > > >> > >> > > these
> > > >>> > > >> > >> > > >> > files first. And make sure it runs
without
> error
> > > >>> and the
> > > >>> > > log
> > > >>> > > >> > >> message
> > > >>> > > >> > >> > > >> prints
> > > >>> > > >> > >> > > >> > out the "name" it found for that data.
If so,
> > you
> > > >>> can
> > > >>> > just
> > > >>> > > >> skip
> > > >>> > > >> > >> > using
> > > >>> > > >> > >> > > >> all
> > > >>> > > >> > >> > > >> > those integers and revert back to using
the
> name
> > > MET
> > > >>> > > knows.
> > > >>> > > >> If
> > > >>> > > >> > >> not,
> > > >>> > > >> > >> > > >> then we
> > > >>> > > >> > >> > > >> > may need to define a custom GRIB table
for
> this
> > > >>> ECMWF
> > > >>> > > GRIB2
> > > >>> > > >> > data.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > Thanks,
> > > >>> > > >> > >> > > >> > John
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> --
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin Zhou
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> 5830 University Research Ct.
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> College Park, MD 20740
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> 301-683-3683
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> --
> > > >>> > > >> > >> > > >>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >>> > > >> > >> > > >> Tara Jensen
> > > >>> > > >> > >> > > >> Project Manager II
> > > >>> > > >> > >> > > >> NCAR RAL and DTC
> > > >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > > >>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > --
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > Binbin Zhou
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > 5830 University Research Ct.
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > College Park, MD 20740
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > Binbin.Zhou at noaa.gov
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > 301-683-3683
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> > >>
> > > >>> > > >> > >
> > > >>> > > >> > > --
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin Zhou
> > > >>> > > >> > >
> > > >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >
> > > >>> > > >> > > 5830 University Research Ct.
> > > >>> > > >> > >
> > > >>> > > >> > > College Park, MD 20740
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin.Zhou at noaa.gov
> > > >>> > > >> > >
> > > >>> > > >> > > 301-683-3683
> > > >>> > > >> > >
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >> > --
> > > >>> > > >> >
> > > >>> > > >> > Binbin Zhou
> > > >>> > > >> >
> > > >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> >
> > > >>> > > >> > 5830 University Research Ct.
> > > >>> > > >> >
> > > >>> > > >> > College Park, MD 20740
> > > >>> > > >> >
> > > >>> > > >> > Binbin.Zhou at noaa.gov
> > > >>> > > >> >
> > > >>> > > >> > 301-683-3683
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > > >
> > > >>> > > > --
> > > >>> > > >
> > > >>> > > > Binbin Zhou
> > > >>> > > >
> > > >>> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >
> > > >>> > > > 5830 University Research Ct.
> > > >>> > > >
> > > >>> > > > College Park, MD 20740
> > > >>> > > >
> > > >>> > > > Binbin.Zhou at noaa.gov
> > > >>> > > >
> > > >>> > > > 301-683-3683
> > > >>> > > >
> > > >>> > >
> > > >>> > >
> > > >>> > > --
> > > >>> > >
> > > >>> > > Binbin Zhou
> > > >>> > >
> > > >>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > >
> > > >>> > > 5830 University Research Ct.
> > > >>> > >
> > > >>> > > College Park, MD 20740
> > > >>> > >
> > > >>> > > Binbin.Zhou at noaa.gov
> > > >>> > >
> > > >>> > > 301-683-3683
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Binbin Zhou
> > > >>>
> > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > >>>
> > > >>> 5830 University Research Ct.
> > > >>>
> > > >>> College Park, MD 20740
> > > >>>
> > > >>> Binbin.Zhou at noaa.gov
> > > >>>
> > > >>> 301-683-3683
> > > >>>
> > > >>>
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day.  Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Thu Dec 10 13:17:08 2020

Julie,

  I'll test it, Thanks!

Binbin

On Thu, Dec 10, 2020 at 1:59 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

> Thanks, John.  That worked great.
>
> Binbin, this version of MET is now accessible to you by running the
> following on venus:
>
> module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> module load met/ens_config_test
>
> Please let us know if you encounter any problems.
>
> Julie
>
> On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > Julie,
> >
> > Building the tarfile is very easy. I just ran this script from the
MET
> repo
> > and specified the branch to be built:
> >    MET/scripts/met_checkout_and_build.sh branch
feature_1604_ens_config
> >
> > And that created a tarfile which I posted to RAL's anonymous ftp
server:
> >
> >
> >
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
> >
> > I assume that'll work for you.
> >
> > Thanks,
> > John
> >
> > On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >
> > > John,
> > >
> > > I am able to help with this install on WCOSS.
> > >
> > > Would it be easier to use git to checkout the repo on WCOSS? Or
would
> it
> > be
> > > > easier if I built a tarfile for this version of the code?
> > > >
> > > The last time I tried to build from git on WCOSS (a while ago),
I
> > > couldn't.  I don't remember all of the details, but I think I
couldn't
> > run
> > > bootstrap and maybe it had to do with the version of autoconf
being too
> > > old?  I'm happy to try it again if you'd like though.
Otherwise, I
> know
> > a
> > > tarfile would work, but I know that is extra work for you.
> > >
> > > Julie
> > >
> > > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Julie,
> > > >
> > > > I have created a bugfix branch for main_v9.1 named
> > > > "feature_1604_ens_config" and it's up on GitHub.
> > > >
> > > > Only a single file has changed in it. But I need to get this
version
> > > > compiled on WCOSS with which Binbin can test.
> > > >
> > > > Are you to help me with this? Would it be easier to use git to
> checkout
> > > > the repo on WCOSS? Or would it be easier if I built a tarfile
for
> this
> > > > version of the code?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > > >
> > > >> Binbin,
> > > >>
> > > >> In order for you to test this change out, we'd need to
recompile a
> > > >> version of the code on WCOSS. I wrote this up as a GitHub
issue:
> > > >> https://github.com/dtcenter/MET/issues/1604
> > > >>
> > > >> And I'll create a branch from main_v9.1 named
> "bugfix_1604_ens_config"
> > > >> and commit the change there. Then we can checkout that code
on
> WCOSS,
> > > >> compile it, and you can test it.
> > > >>
> > > >> Yes, you're right, if the fcst fields differ from the climo
fields,
> > > you'd
> > > >> need to specify them separately. That's pretty easy in the
> > Ensemble-Stat
> > > >> config file... but are you asking how to control that via
METplus?
> > > >>
> > > >> John
> > > >>
> > > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >>>
> > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > > >>>
> > > >>> John,
> > > >>>
> > > >>>   Great! Let me test it at first before make change in
MET9.1
> > > >>> Another question:
> > > >>>    Since climatology data is still in NCEP's grib2 format,
In the
> > > >>> ensemble_stat
> > > >>> config file,  climo  entry settings  should be different
from those
> > of
> > > >>> fcst, obs entries?
> > > >>> How to set climo (name, level, etc.)?
> > > >>>
> > > >>> Binbin
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> > > >>> met_help at ucar.edu>
> > > >>> wrote:
> > > >>>
> > > >>> > Binbin,
> > > >>> >
> > > >>> > Success! When I recompile with that one line commented
out,
> > > >>> plot_data_plane
> > > >>> > is able to filter down all those records to the single one
that
> we
> > > >>> want.
> > > >>> > Note in the example below that I used the "lead_time" and
> > "GRIB_ens"
> > > >>> > options to filter the input records down.
> > > >>> >
> > > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps '*name="HGT";
> > > >>> level="P500";
> > > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found
exact
> match
> > > for
> > > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> > > >>> "ensposte.t00z.z500hr".*
> > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
> > records
> > > >>> > matching VarInfo "HGT/P500" in GRIB file
"ensposte.t00z.z500hr".
> > > >>> > DEBUG 1: Creating postscript file: plot.ps
> > > >>> >
> > > >>> > Also note that this selects out record number 26.
> > > >>> >
> > > >>> > Shall I make this change as a "bugfix" for met-9.1...which
could
> be
> > > >>> > included in a met-9.1.2 bugfix release? Or shall I just
include
> it
> > in
> > > >>> > development for the next release (planned for March 2021)?
> > > >>> >
> > > >>> > Thanks,
> > > >>> > John
> > > >>> >
> > > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via RT
<
> > > >>> > met_help at ucar.edu> wrote:
> > > >>> >
> > > >>> > >
> > > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> > > >>> > >
> > > >>> > > John,
> > > >>> > >
> > > >>> > >   If this removing impacts other applications, or takes
some
> time
> > > >>> > > to be effective, then won't try.  I will use another
approach,
> > such
> > > >>> as
> > > >>> > > borrow GEFS's grib2 data structure as tank to save EC
ensemble
> > > >>> > > data.
> > > >>> > >
> > > >>> > > Binbin
> > > >>> > >
> > > >>> > >
> > > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
Affiliate <
> > > >>> > > binbin.zhou at noaa.gov> wrote:
> > > >>> > >
> > > >>> > > > John,
> > > >>> > > >
> > > >>> > > >   Yes, please try to use this approach.
> > > >>> > > >
> > > >>> > > > Thanks!
> > > >>> > > >
> > > >>> > > > Binbin
> > > >>> > > >
> > > >>> > > > Binbin
> > > >>> > > >
> > > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway via
RT <
> > > >>> > > > met_help at ucar.edu> wrote:
> > > >>> > > >
> > > >>> > > >> Binbin,
> > > >>> > > >>
> > > >>> > > >> I used the "-PDS10" option for wgrib to dump those
records:
> > > >>> > > >>
> > > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > > >>> > > >> | egrep  "^11:|^26:|^43:"
> > > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12 6
0 0 1
> 0
> > 0
> > > 1
> > > >>> 0 0
> > > >>> > 0
> > > >>> > > 21
> > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > > >>> > > >>
> > > >>> > > >> Looking VERY closely, there is a difference in the
42-nd
> > column
> > > >>> which
> > > >>> > is
> > > >>> > > >> the 4-th from the last... records 11, 26, and 43 have
values
> > of
> > > >>> 3, 1,
> > > >>> > > and
> > > >>> > > >> 2
> > > >>> > > >> respectively.
> > > >>> > > >>
> > > >>> > > >> On this page (
> > > >>> > >
https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > > >>> > > >> ),
> > > >>> > > >> byte 42 = 1 means "unperturbed control forecast".
> > > >>> > > >>
> > > >>> > > >> So we *should* be able to use the "GRIB_ens" config
option
> to
> > > >>> > > >> differentiate
> > > >>> > > >> between those records. BUT when I try, it doesn't
work. And
> > > that's
> > > >>> > > because
> > > >>> > > >> of this line:
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > > >>> > > >>
> > > >>> > > >> We only apply those filtering options to GRIB files
created
> by
> > > >>> NCEP.
> > > >>> > If
> > > >>> > > >> you'd like, I could try removing that condition to
see if we
> > can
> > > >>> pick
> > > >>> > > out
> > > >>> > > >> the single record of interest.
> > > >>> > > >>
> > > >>> > > >> John
> > > >>> > > >>
> > > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov
via RT
> <
> > > >>> > > >> met_help at ucar.edu> wrote:
> > > >>> > > >>
> > > >>> > > >> >
> > > >>> > > >> > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >
> > > >>> > > >> >
> > > >>> > > >> > John,
> > > >>> > > >> >
> > > >>> > > >> >   Look at my
> > > >>> > > >> > directory
> > > >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is
only for
> 500
> > > >>> HGT.
> > > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
anl:type=analysis
> > > >>> > > >> > I get 3 records:
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > >>> > > >> >
> > > >>> > > >> > From its grib2 file, we know only  record#26 is
analysis
> > > low-res
> > > >>> > > >> control.
> > > >>> > > >> > If you have some way to determine which record is
analysis
> > > >>> low-res
> > > >>> > > >> control
> > > >>> > > >> > from the grib1 file,
> > > >>> > > >> > then we can use grib1 directly (don't consider the
size
> > > issue).
> > > >>> > > >> >
> > > >>> > > >> > Binbin
> > > >>> > > >> >
> > > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
> Affiliate
> > <
> > > >>> > > >> > binbin.zhou at noaa.gov> wrote:
> > > >>> > > >> >
> > > >>> > > >> > > John,
> > > >>> > > >> > >
> > > >>> > > >> > >   Grib1 data is too large, another issue is the
low-res
> > > >>> control
> > > >>> > > >> analysis
> > > >>> > > >> > > data can not be retrieved
> > > >>> > > >> > > from the grib1 ecmwf data files. If using wgrib,
we can
> > see
> > > 3
> > > >>> > > analysis
> > > >>> > > >> > > fields, but don't know
> > > >>> > > >> > > which is low-res control. After convert to grib2,
wgrib2
> > can
> > > >>> show
> > > >>> > > >> which
> > > >>> > > >> > > record is low-res control.
> > > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set
on
> WCOSS.
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin
> > > >>> > > >> > >
> > > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley Gotway
via
> RT <
> > > >>> > > >> > > met_help at ucar.edu> wrote:
> > > >>> > > >> > >
> > > >>> > > >> > >> Binbin,
> > > >>> > > >> > >>
> > > >>> > > >> > >> Sounds like there's a lot of moving parts there.
How
> > would
> > > >>> you
> > > >>> > like
> > > >>> > > >> to
> > > >>> > > >> > >> proceed?
> > > >>> > > >> > >>
> > > >>> > > >> > >> Should we check to see if MET can read those
ECMWF
> GRIB1
> > > >>> files
> > > >>> > > >> directly
> > > >>> > > >> > >> without needing to reformat?
> > > >>> > > >> > >> Or do you just want to create a custom GRIB
table for
> the
> > > >>> result
> > > >>> > of
> > > >>> > > >> your
> > > >>> > > >> > >> pre-processing and conversion steps?
> > > >>> > > >> > >>
> > > >>> > > >> > >> John
> > > >>> > > >> > >>
> > > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
Binbin.Zhou at noaa.gov
> via
> > > RT <
> > > >>> > > >> > >> met_help at ucar.edu> wrote:
> > > >>> > > >> > >>
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > <URL:
> > > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >>> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > John,
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
converted
> > from
> > > >>> its
> > > >>> > > grib1
> > > >>> > > >> > >> format
> > > >>> > > >> > >> > data.
> > > >>> > > >> > >> > Before conversion, the parameter names are
there, but
> > > after
> > > >>> > > >> > conversion,
> > > >>> > > >> > >> the
> > > >>> > > >> > >> > names
> > > >>> > > >> > >> > are gone. It is because of unknown tables. The
grib1
> > data
> > > >>> is
> > > >>> > > still
> > > >>> > > >> > >> > original. The original
> > > >>> > > >> > >> > grib1 data files are very strange data format
we can
> > not
> > > >>> read.
> > > >>> > So
> > > >>> > > >> we
> > > >>> > > >> > >> have
> > > >>> > > >> > >> > to use
> > > >>> > > >> > >> > a Frotran code to re-arrange the fields and
save them
> > as
> > > >>> > > NCEP-like
> > > >>> > > >> > grib1
> > > >>> > > >> > >> > data files.
> > > >>> > > >> > >> > So after several processes, some important
info like
> > > tables
> > > >>> > could
> > > >>> > > >> be
> > > >>> > > >> > >> lost.
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >    I'll test your first suggestion since the
field
> > record
> > > >>> order
> > > >>> > > is
> > > >>> > > >> > >> fixed.
> > > >>> > > >> > >> > Currently we still
> > > >>> > > >> > >> > obtained grib1 data from ECWMF, eventually, we
will
> get
> > > >>> grib2
> > > >>> > > data
> > > >>> > > >> > >> > directly. So before
> > > >>> > > >> > >> > we can get original grib2 files from ECMWF, we
just
> use
> > > >>> this
> > > >>> > > >> temporal
> > > >>> > > >> > >> > solution.
> > > >>> > > >> > >> > '
> > > >>> > > >> > >> > Thanks!
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley
Gotway via
> > RT
> > > <
> > > >>> > > >> > >> > met_help at ucar.edu>
> > > >>> > > >> > >> > wrote:
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > > Binbin,
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > OK, working on Venus.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > You have a typo in the command you sent.
> > > >>> > > >> > >> > >  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> a.ps
> > > >>> > > >> > >> > >
> > > >>> > > >> > >>
> > > >>> > > >>
> > > >>> >
> > > >>>
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> > >> > -v
> > > >>> > > >> > >> > > 3
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
> > > "GRIB2_parm_cat".
> > > >>> > After
> > > >>> > > >> > >> > correcting
> > > >>> > > >> > >> > > that I did still get an error but it's a
very
> useful
> > > one!
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > module use
> > > >>> > > >> > >>
> > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > >>> > > >> > >> > > module load met/9.1
> > > >>> > > >> > >> > > module load grib_util/1.1.1
> > > >>> > > >> > >> > > cd
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > plot.ps
> > > >>> > > >> > >> > 'GRIB2_disc=255;
> > > >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3;
GRIB2_parm=5;
> > > >>> level="P500";'
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > *ERROR  : ERROR  : VarInfoGrib2::set_dict()
-> no
> > > >>> parameter
> > > >>> > > found
> > > >>> > > >> > with
> > > >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat (3)
> > GRIB2_parm
> > > >>> (5).
> > > >>> > > Use
> > > >>> > > >> the
> > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
define
> custom
> > > >>> GRIB
> > > >>> > > >> > >> tables.ERROR
> > > >>> > > >> > >> > :
> > > >>> > > >> > >> > > *
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > So this table is not yet defined in MET. I
checked
> > > their
> > > >>> > > website:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > But I don't see an entry defined for
GRIB2_parm=5;
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > 3 things...
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (1) As a crude temporary solution, you could
refer
> to
> > > it
> > > >>> by
> > > >>> > > it's
> > > >>> > > >> > >> record
> > > >>> > > >> > >> > > number. Let's assume you know that record 1
has
> > > >>> temperature
> > > >>> > > data:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > plot.ps
> > > >>> > > >> > 'name="TMP";
> > > >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > Here I'm reading record 1, naming it TMP and
the
> > using
> > > >>> > > >> > set_attr_level
> > > >>> > > >> > >> > write
> > > >>> > > >> > >> > > the output level as P500 instead of R1.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (2) You could create a custom GRIB table
yourself
> > > >>> locally,
> > > >>> > and
> > > >>> > > >> then
> > > >>> > > >> > >> set
> > > >>> > > >> > >> > the
> > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
point MET
> to
> > > it.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > (3) The best solution is contacting ECMWF
and
> > > requesting
> > > >>> they
> > > >>> > > >> send
> > > >>> > > >> > >> over
> > > >>> > > >> > >> > the
> > > >>> > > >> > >> > > GRIB2 table we should be using to decode
this data.
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > Thanks,
> > > >>> > > >> > >> > > John
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley
Gotway <
> > > >>> > > >> johnhg at ucar.edu>
> > > >>> > > >> > >> > wrote:
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > > > We might already have them. I'm not sure.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > But there's an additional required piece
of the
> > > puzzle.
> > > >>> > > Binbin
> > > >>> > > >> > runs
> > > >>> > > >> > >> > > > "wgrib2" to figure out what "name" he
should use
> in
> > > >>> the MET
> > > >>> > > >> config
> > > >>> > > >> > >> > files.
> > > >>> > > >> > >> > > > If wgrib2 doesn't tell him the right
"name", the
> > > >>> problem
> > > >>> > will
> > > >>> > > >> > >> remain.
> > > >>> > > >> > >> > > > wgrib2 can be configured to define
additional
> > tables
> > > as
> > > >>> > well,
> > > >>> > > >> but
> > > >>> > > >> > >> for
> > > >>> > > >> > >> > > this
> > > >>> > > >> > >> > > > to be easy for Binbin, that'd mean
updating the
> > > script
> > > >>> > that's
> > > >>> > > >> run
> > > >>> > > >> > >> for
> > > >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And I'm
> guessing
> > > >>> that's
> > > >>> > > not
> > > >>> > > >> > very
> > > >>> > > >> > >> > easy.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > Yes, I do agree that the best solution is
getting
> > the
> > > >>> ECMWF
> > > >>> > > >> GRIB
> > > >>> > > >> > >> tables
> > > >>> > > >> > >> > > > and adding them for both wgrib2 and MET.
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > John
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara Jensen
<
> > > >>> > jensen at ucar.edu>
> > > >>> > > >> > wrote:
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > > >> John,
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> It seems to me that it would be
worthwhile to
> just
> > > ask
> > > >>> > ECMWF
> > > >>> > > >> for
> > > >>> > > >> > >> their
> > > >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.
And other
> > > >>> > operational
> > > >>> > > >> > >> centers
> > > >>> > > >> > >> > for
> > > >>> > > >> > >> > > >> that matter.  Do you agree this would be
a good
> > > >>> solution?
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Thanks, Tara
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> ---------- Forwarded message ---------
> > > >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769]
urgent
> > > >>> question for
> > > >>> > > >> > reading
> > > >>> > > >> > >> > ECMWF
> > > >>> > > >> > >> > > >> ensmeble data
> > > >>> > > >> > >> > > >> To:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> <URL:
> > > >>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > >>> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> John,
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>   I tested using following command:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > > a.ps
> > > >>> > > >> > >> > > >>
> > > >>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> -v3
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> ERROR  :
> > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either name
> > or
> > > >>> > > >> GRIB2_disc,
> > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified
> in
> > > >>> field
> > > >>> > > >> > >> information
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > > a.ps
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> >
> > > >>> > > >> >
> > > >>> > >
> > > >>>
> > >
>
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > >>> > > >> > >> > > >> -v
> > > >>> > > >> > >> > > >> 3
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Still got error
> > > >>> > > >> > >> > > >> ERROR  :
> > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either name
> > or
> > > >>> > > >> GRIB2_disc,
> > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
specified
> in
> > > >>> field
> > > >>> > > >> > >> information
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Looks like discipline=255 is not
recognized
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John
Halley
> Gotway
> > > >>> via RT
> > > >>> > <
> > > >>> > > >> > >> > > >> met_help at ucar.edu>
> > > >>> > > >> > >> > > >> wrote:
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> > Binbin,
> > > >>> > > >> > >> > > >> > You can do something similar with MET.
Please
> go
> > > to
> > > >>> this
> > > >>> > > >> link:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >>
> > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and
you'll
> > find
> > > >>> the
> > > >>> > > >> content
> > > >>> > > >> > >> > > >> > copied/pasted below. You can specify
these
> > INSTEAD
> > > >>> OF
> > > >>> > > >> > specifying
> > > >>> > > >> > >> a
> > > >>> > > >> > >> > > >> > requested name.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an
integer
> > > >>> specifying the
> > > >>> > > >> master
> > > >>> > > >> > >> > table
> > > >>> > > >> > >> > > >> > number.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an
integer
> > > >>> specifying the
> > > >>> > > >> local
> > > >>> > > >> > >> table
> > > >>> > > >> > >> > > >> > number.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an
integer
> > > >>> specifying the
> > > >>> > > >> GRIB2
> > > >>> > > >> > >> > > >> discipline
> > > >>> > > >> > >> > > >> > code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > parameter
> > > >>> > > >> > >> > > >> > category
> > > >>> > > >> > >> > > >> > //         code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an
integer
> > > >>> specifying the
> > > >>> > > >> > >> parameter
> > > >>> > > >> > >> > > code.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an
integer
> > > specifying
> > > >>> the
> > > >>> > > >> product
> > > >>> > > >> > >> > > >> definition
> > > >>> > > >> > >> > > >> > //         template (Table 4.0).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > generating
> > > >>> > > >> > >> > > >> > process
> > > >>> > > >> > >> > > >> > //         (Table 4.3).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an
integer
> > > >>> specifying the
> > > >>> > > >> > >> originating
> > > >>> > > >> > >> > > >> > center.
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> > ensemble
> > > >>> > > >> > >> > > >> type
> > > >>> > > >> > >> > > >> > //         (Table 4.6).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an
integer
> > > >>> specifying
> > > >>> > > the
> > > >>> > > >> > >> derived
> > > >>> > > >> > >> > > >> > product
> > > >>> > > >> > >> > > >> > //         type (Table 4.7).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is an
integer
> > > >>> > specifying
> > > >>> > > >> the
> > > >>> > > >> > >> > > >> statistical
> > > >>> > > >> > >> > > >> > //         processing type (Table
4.10).
> > > >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index"
and
> > > >>> > > "GRIB2_ipdtmpl_val"
> > > >>> > > >> > >> entries
> > > >>> > > >> > >> > > are
> > > >>> > > >> > >> > > >> > arrays
> > > >>> > > >> > >> > > >> > //         of integers which specify
the
> product
> > > >>> > > description
> > > >>> > > >> > >> > template
> > > >>> > > >> > >> > > >> > values to
> > > >>> > > >> > >> > > >> > //         be used. The indices are 0-
based.
> For
> > > >>> > example,
> > > >>> > > >> use
> > > >>> > > >> > the
> > > >>> > > >> > >> > > >> > following to
> > > >>> > > >> > >> > > >> > //         request a GRIB2 record whose
9-th
> and
> > > >>> 27-th
> > > >>> > > >> product
> > > >>> > > >> > >> > > >> description
> > > >>> > > >> > >> > > >> > //         template values are 1 and 2,
> > > >>> respectively:
> > > >>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8,
26];
> > > >>> > > >> > GRIB2_ipdtmpl_val=[1,
> > > >>> > > >> > >> 2];
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > While I don't have any ECMWF data on
hand for
> > > >>> testing, I
> > > >>> > > do
> > > >>> > > >> > have
> > > >>> > > >> > >> a
> > > >>> > > >> > >> > > file
> > > >>> > > >> > >> > > >> > from the Air Force from which wgrib2
prints a
> > > >>> similar
> > > >>> > > >> result:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var
discipline=0
> > > >>> center=57
> > > >>> > > >> > >> local_table=1
> > > >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour
fcst:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > So I used that for testing. I ran the
> following
> > > >>> > > >> plot_data_plane
> > > >>> > > >> > >> > > command
> > > >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > plot_data_plane
GALWEM_20170610_CY00_FH18.GR2
> > > >>> plot.ps
> > > >>> > > >> > >> > 'GRIB2_disc=0;
> > > >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
GRIB2_parm_cat=1;
> > > >>> > > >> GRIB2_parm=192;
> > > >>> > > >> > >> > > >> level="Z0";'
> > > >>> > > >> > >> > > >> > -v 3
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And then I looked closely at the log
message
> to
> > > >>> confirm
> > > >>> > > >> that it
> > > >>> > > >> > >> used
> > > >>> > > >> > >> > > >> > record 549, as expected:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > DEBUG 3: MetGrib2DataFile::data_plane()
->
> Found
> > > >>> exact
> > > >>> > > match
> > > >>> > > >> > for
> > > >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field 1
of
> GRIB2
> > > file
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I
defined to
> do
> > a
> > > >>> table
> > > >>> > > >> lookup
> > > >>> > > >> > >> and
> > > >>> > > >> > >> > > find
> > > >>> > > >> > >> > > >> a
> > > >>> > > >> > >> > > >> > match in the MET GRIB tables:
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >>
> > > >>> > >
> > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > And it determined that that field
should be
> > called
> > > >>> > > "CTSTM".
> > > >>> > > >> > >> > Hopefully,
> > > >>> > > >> > >> > > >> the
> > > >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already defined
there
> in
> > > >>> MET. If
> > > >>> > > >> not, we
> > > >>> > > >> > >> can
> > > >>> > > >> > >> > > >> always
> > > >>> > > >> > >> > > >> > add them.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > Please try running plot_data_plane at
> verbosity
> > > >>> level 3
> > > >>> > on
> > > >>> > > >> one
> > > >>> > > >> > of
> > > >>> > > >> > >> > > these
> > > >>> > > >> > >> > > >> > files first. And make sure it runs
without
> error
> > > >>> and the
> > > >>> > > log
> > > >>> > > >> > >> message
> > > >>> > > >> > >> > > >> prints
> > > >>> > > >> > >> > > >> > out the "name" it found for that data.
If so,
> > you
> > > >>> can
> > > >>> > just
> > > >>> > > >> skip
> > > >>> > > >> > >> > using
> > > >>> > > >> > >> > > >> all
> > > >>> > > >> > >> > > >> > those integers and revert back to using
the
> name
> > > MET
> > > >>> > > knows.
> > > >>> > > >> If
> > > >>> > > >> > >> not,
> > > >>> > > >> > >> > > >> then we
> > > >>> > > >> > >> > > >> > may need to define a custom GRIB table
for
> this
> > > >>> ECMWF
> > > >>> > > GRIB2
> > > >>> > > >> > data.
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >> > Thanks,
> > > >>> > > >> > >> > > >> > John
> > > >>> > > >> > >> > > >> >
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> --
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin Zhou
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> 5830 University Research Ct.
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> College Park, MD 20740
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> 301-683-3683
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >> --
> > > >>> > > >> > >> > > >>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >>> > > >> > >> > > >> Tara Jensen
> > > >>> > > >> > >> > > >> Project Manager II
> > > >>> > > >> > >> > > >> NCAR RAL and DTC
> > > >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307 USA
> > > >>> > > >> > >> > > >> +1 303-497-8479          jensen at ucar.edu
> > > >>> > > >> > >> > > >>
> > > >>> > > >> > >> > > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> > >
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > --
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > Binbin Zhou
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > 5830 University Research Ct.
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > College Park, MD 20740
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > Binbin.Zhou at noaa.gov
> > > >>> > > >> > >> >
> > > >>> > > >> > >> > 301-683-3683
> > > >>> > > >> > >> >
> > > >>> > > >> > >> >
> > > >>> > > >> > >>
> > > >>> > > >> > >>
> > > >>> > > >> > >
> > > >>> > > >> > > --
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin Zhou
> > > >>> > > >> > >
> > > >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> > >
> > > >>> > > >> > > 5830 University Research Ct.
> > > >>> > > >> > >
> > > >>> > > >> > > College Park, MD 20740
> > > >>> > > >> > >
> > > >>> > > >> > > Binbin.Zhou at noaa.gov
> > > >>> > > >> > >
> > > >>> > > >> > > 301-683-3683
> > > >>> > > >> > >
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >> > --
> > > >>> > > >> >
> > > >>> > > >> > Binbin Zhou
> > > >>> > > >> >
> > > >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >> >
> > > >>> > > >> > 5830 University Research Ct.
> > > >>> > > >> >
> > > >>> > > >> > College Park, MD 20740
> > > >>> > > >> >
> > > >>> > > >> > Binbin.Zhou at noaa.gov
> > > >>> > > >> >
> > > >>> > > >> > 301-683-3683
> > > >>> > > >> >
> > > >>> > > >> >
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > > >
> > > >>> > > > --
> > > >>> > > >
> > > >>> > > > Binbin Zhou
> > > >>> > > >
> > > >>> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > > >
> > > >>> > > > 5830 University Research Ct.
> > > >>> > > >
> > > >>> > > > College Park, MD 20740
> > > >>> > > >
> > > >>> > > > Binbin.Zhou at noaa.gov
> > > >>> > > >
> > > >>> > > > 301-683-3683
> > > >>> > > >
> > > >>> > >
> > > >>> > >
> > > >>> > > --
> > > >>> > >
> > > >>> > > Binbin Zhou
> > > >>> > >
> > > >>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>> > >
> > > >>> > > 5830 University Research Ct.
> > > >>> > >
> > > >>> > > College Park, MD 20740
> > > >>> > >
> > > >>> > > Binbin.Zhou at noaa.gov
> > > >>> > >
> > > >>> > > 301-683-3683
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Binbin Zhou
> > > >>>
> > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > >>>
> > > >>> 5830 University Research Ct.
> > > >>>
> > > >>> College Park, MD 20740
> > > >>>
> > > >>> Binbin.Zhou at noaa.gov
> > > >>>
> > > >>> 301-683-3683
> > > >>>
> > > >>>
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day.  Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Thu Dec 10 13:18:00 2020

John,

  Thanks!

Binbin

On Thu, Dec 10, 2020 at 2:06 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Thanks Julie.
>
> And Binbin, here's an EnsembleStatConfig file where climos are used:
>
>
>
https://github.com/dtcenter/MET/blob/main_v9.1/test/config/EnsembleStatConfig_climo
>
> Since the list of forecast fields is exactly the same as the climo
fields,
> this config file just copies them over using:
>    climo_mean = fcst;
>
> In your case, when they differ, you'd just specify the "field" array
inside
> the "climo_mean" dictionary. For example:
>
> climo_mean = {
>
>    field = [
>       { name  = "TMP"; level = "Z2";   },
>       { name  = "TMP"; level = "P850"; }
>    ];
>    ...
> }
>
> On Thu, Dec 10, 2020 at 12:00 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >
> > Thanks, John.  That worked great.
> >
> > Binbin, this version of MET is now accessible to you by running
the
> > following on venus:
> >
> > module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > module load met/ens_config_test
> >
> > Please let us know if you encounter any problems.
> >
> > Julie
> >
> > On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > >
> > > Julie,
> > >
> > > Building the tarfile is very easy. I just ran this script from
the MET
> > repo
> > > and specified the branch to be built:
> > >    MET/scripts/met_checkout_and_build.sh branch
feature_1604_ens_config
> > >
> > > And that created a tarfile which I posted to RAL's anonymous ftp
> server:
> > >
> > >
> > >
> >
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
> > >
> > > I assume that'll work for you.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> > > >
> > > > John,
> > > >
> > > > I am able to help with this install on WCOSS.
> > > >
> > > > Would it be easier to use git to checkout the repo on WCOSS?
Or would
> > it
> > > be
> > > > > easier if I built a tarfile for this version of the code?
> > > > >
> > > > The last time I tried to build from git on WCOSS (a while
ago), I
> > > > couldn't.  I don't remember all of the details, but I think I
> couldn't
> > > run
> > > > bootstrap and maybe it had to do with the version of autoconf
being
> too
> > > > old?  I'm happy to try it again if you'd like though.
Otherwise, I
> > know
> > > a
> > > > tarfile would work, but I know that is extra work for you.
> > > >
> > > > Julie
> > > >
> > > > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > > >
> > > > > Julie,
> > > > >
> > > > > I have created a bugfix branch for main_v9.1 named
> > > > > "feature_1604_ens_config" and it's up on GitHub.
> > > > >
> > > > > Only a single file has changed in it. But I need to get this
> version
> > > > > compiled on WCOSS with which Binbin can test.
> > > > >
> > > > > Are you to help me with this? Would it be easier to use git
to
> > checkout
> > > > > the repo on WCOSS? Or would it be easier if I built a
tarfile for
> > this
> > > > > version of the code?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > >> Binbin,
> > > > >>
> > > > >> In order for you to test this change out, we'd need to
recompile a
> > > > >> version of the code on WCOSS. I wrote this up as a GitHub
issue:
> > > > >> https://github.com/dtcenter/MET/issues/1604
> > > > >>
> > > > >> And I'll create a branch from main_v9.1 named
> > "bugfix_1604_ens_config"
> > > > >> and commit the change there. Then we can checkout that code
on
> > WCOSS,
> > > > >> compile it, and you can test it.
> > > > >>
> > > > >> Yes, you're right, if the fcst fields differ from the climo
> fields,
> > > > you'd
> > > > >> need to specify them separately. That's pretty easy in the
> > > Ensemble-Stat
> > > > >> config file... but are you asking how to control that via
METplus?
> > > > >>
> > > > >> John
> > > > >>
> > > > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT
<
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >>>
> > > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> > > > >>>
> > > > >>> John,
> > > > >>>
> > > > >>>   Great! Let me test it at first before make change in
MET9.1
> > > > >>> Another question:
> > > > >>>    Since climatology data is still in NCEP's grib2 format,
In the
> > > > >>> ensemble_stat
> > > > >>> config file,  climo  entry settings  should be different
from
> those
> > > of
> > > > >>> fcst, obs entries?
> > > > >>> How to set climo (name, level, etc.)?
> > > > >>>
> > > > >>> Binbin
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT <
> > > > >>> met_help at ucar.edu>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Binbin,
> > > > >>> >
> > > > >>> > Success! When I recompile with that one line commented
out,
> > > > >>> plot_data_plane
> > > > >>> > is able to filter down all those records to the single
one that
> > we
> > > > >>> want.
> > > > >>> > Note in the example below that I used the "lead_time"
and
> > > "GRIB_ens"
> > > > >>> > options to filter the input records down.
> > > > >>> >
> > > > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps
'*name="HGT";
> > > > >>> level="P500";
> > > > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> > > > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found
exact
> > match
> > > > for
> > > > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> > > > >>> "ensposte.t00z.z500hr".*
> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1
GRIB
> > > records
> > > > >>> > matching VarInfo "HGT/P500" in GRIB file
> "ensposte.t00z.z500hr".
> > > > >>> > DEBUG 1: Creating postscript file: plot.ps
> > > > >>> >
> > > > >>> > Also note that this selects out record number 26.
> > > > >>> >
> > > > >>> > Shall I make this change as a "bugfix" for met-
9.1...which
> could
> > be
> > > > >>> > included in a met-9.1.2 bugfix release? Or shall I just
include
> > it
> > > in
> > > > >>> > development for the next release (planned for March
2021)?
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> > John
> > > > >>> >
> > > > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via
RT <
> > > > >>> > met_help at ucar.edu> wrote:
> > > > >>> >
> > > > >>> > >
> > > > >>> > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > >
> > > > >>> > >
> > > > >>> > > John,
> > > > >>> > >
> > > > >>> > >   If this removing impacts other applications, or
takes some
> > time
> > > > >>> > > to be effective, then won't try.  I will use another
> approach,
> > > such
> > > > >>> as
> > > > >>> > > borrow GEFS's grib2 data structure as tank to save EC
> ensemble
> > > > >>> > > data.
> > > > >>> > >
> > > > >>> > > Binbin
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
Affiliate <
> > > > >>> > > binbin.zhou at noaa.gov> wrote:
> > > > >>> > >
> > > > >>> > > > John,
> > > > >>> > > >
> > > > >>> > > >   Yes, please try to use this approach.
> > > > >>> > > >
> > > > >>> > > > Thanks!
> > > > >>> > > >
> > > > >>> > > > Binbin
> > > > >>> > > >
> > > > >>> > > > Binbin
> > > > >>> > > >
> > > > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway
via RT <
> > > > >>> > > > met_help at ucar.edu> wrote:
> > > > >>> > > >
> > > > >>> > > >> Binbin,
> > > > >>> > > >>
> > > > >>> > > >> I used the "-PDS10" option for wgrib to dump those
> records:
> > > > >>> > > >>
> > > > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> > > > >>> > > >>
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> > > > >>> > > >> | egrep  "^11:|^26:|^43:"
> > > > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12
6 0 0
> 1
> > 0
> > > 0
> > > > 1
> > > > >>> 0 0
> > > > >>> > 0
> > > > >>> > > 21
> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> > > > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12
6 0 0
> 1
> > 0
> > > 0
> > > > 1
> > > > >>> 0 0
> > > > >>> > 0
> > > > >>> > > 21
> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> > > > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20 12
6 0 0
> 1
> > 0
> > > 0
> > > > 1
> > > > >>> 0 0
> > > > >>> > 0
> > > > >>> > > 21
> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> > > > >>> > > >>
> > > > >>> > > >> Looking VERY closely, there is a difference in the
42-nd
> > > column
> > > > >>> which
> > > > >>> > is
> > > > >>> > > >> the 4-th from the last... records 11, 26, and 43
have
> values
> > > of
> > > > >>> 3, 1,
> > > > >>> > > and
> > > > >>> > > >> 2
> > > > >>> > > >> respectively.
> > > > >>> > > >>
> > > > >>> > > >> On this page (
> > > > >>> > >
https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> > > > >>> > > >> ),
> > > > >>> > > >> byte 42 = 1 means "unperturbed control forecast".
> > > > >>> > > >>
> > > > >>> > > >> So we *should* be able to use the "GRIB_ens" config
option
> > to
> > > > >>> > > >> differentiate
> > > > >>> > > >> between those records. BUT when I try, it doesn't
work.
> And
> > > > that's
> > > > >>> > > because
> > > > >>> > > >> of this line:
> > > > >>> > > >>
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> > > > >>> > > >>
> > > > >>> > > >> We only apply those filtering options to GRIB files
> created
> > by
> > > > >>> NCEP.
> > > > >>> > If
> > > > >>> > > >> you'd like, I could try removing that condition to
see if
> we
> > > can
> > > > >>> pick
> > > > >>> > > out
> > > > >>> > > >> the single record of interest.
> > > > >>> > > >>
> > > > >>> > > >> John
> > > > >>> > > >>
> > > > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM Binbin.Zhou at noaa.gov
via
> RT
> > <
> > > > >>> > > >> met_help at ucar.edu> wrote:
> > > > >>> > > >>
> > > > >>> > > >> >
> > > > >>> > > >> > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > > >
> > > > >>> > > >> >
> > > > >>> > > >> > John,
> > > > >>> > > >> >
> > > > >>> > > >> >   Look at my
> > > > >>> > > >> > directory
> > > > >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> > > > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is
only for
> > 500
> > > > >>> HGT.
> > > > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
> anl:type=analysis
> > > > >>> > > >> > I get 3 records:
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> > > > >>> > > >> >
> > > > >>> > > >> > From its grib2 file, we know only  record#26 is
analysis
> > > > low-res
> > > > >>> > > >> control.
> > > > >>> > > >> > If you have some way to determine which record is
> analysis
> > > > >>> low-res
> > > > >>> > > >> control
> > > > >>> > > >> > from the grib1 file,
> > > > >>> > > >> > then we can use grib1 directly (don't consider
the size
> > > > issue).
> > > > >>> > > >> >
> > > > >>> > > >> > Binbin
> > > > >>> > > >> >
> > > > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou - NOAA
> > Affiliate
> > > <
> > > > >>> > > >> > binbin.zhou at noaa.gov> wrote:
> > > > >>> > > >> >
> > > > >>> > > >> > > John,
> > > > >>> > > >> > >
> > > > >>> > > >> > >   Grib1 data is too large, another issue is the
> low-res
> > > > >>> control
> > > > >>> > > >> analysis
> > > > >>> > > >> > > data can not be retrieved
> > > > >>> > > >> > > from the grib1 ecmwf data files. If using
wgrib, we
> can
> > > see
> > > > 3
> > > > >>> > > analysis
> > > > >>> > > >> > > fields, but don't know
> > > > >>> > > >> > > which is low-res control. After convert to
grib2,
> wgrib2
> > > can
> > > > >>> show
> > > > >>> > > >> which
> > > > >>> > > >> > > record is low-res control.
> > > > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data set
on
> > WCOSS.
> > > > >>> > > >> > >
> > > > >>> > > >> > > Binbin
> > > > >>> > > >> > >
> > > > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley
Gotway via
> > RT <
> > > > >>> > > >> > > met_help at ucar.edu> wrote:
> > > > >>> > > >> > >
> > > > >>> > > >> > >> Binbin,
> > > > >>> > > >> > >>
> > > > >>> > > >> > >> Sounds like there's a lot of moving parts
there. How
> > > would
> > > > >>> you
> > > > >>> > like
> > > > >>> > > >> to
> > > > >>> > > >> > >> proceed?
> > > > >>> > > >> > >>
> > > > >>> > > >> > >> Should we check to see if MET can read those
ECMWF
> > GRIB1
> > > > >>> files
> > > > >>> > > >> directly
> > > > >>> > > >> > >> without needing to reformat?
> > > > >>> > > >> > >> Or do you just want to create a custom GRIB
table for
> > the
> > > > >>> result
> > > > >>> > of
> > > > >>> > > >> your
> > > > >>> > > >> > >> pre-processing and conversion steps?
> > > > >>> > > >> > >>
> > > > >>> > > >> > >> John
> > > > >>> > > >> > >>
> > > > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
Binbin.Zhou at noaa.gov
> > via
> > > > RT <
> > > > >>> > > >> > >> met_help at ucar.edu> wrote:
> > > > >>> > > >> > >>
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > <URL:
> > > > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > > >>> > >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > John,
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
converted
> > > from
> > > > >>> its
> > > > >>> > > grib1
> > > > >>> > > >> > >> format
> > > > >>> > > >> > >> > data.
> > > > >>> > > >> > >> > Before conversion, the parameter names are
there,
> but
> > > > after
> > > > >>> > > >> > conversion,
> > > > >>> > > >> > >> the
> > > > >>> > > >> > >> > names
> > > > >>> > > >> > >> > are gone. It is because of unknown tables.
The
> grib1
> > > data
> > > > >>> is
> > > > >>> > > still
> > > > >>> > > >> > >> > original. The original
> > > > >>> > > >> > >> > grib1 data files are very strange data
format we
> can
> > > not
> > > > >>> read.
> > > > >>> > So
> > > > >>> > > >> we
> > > > >>> > > >> > >> have
> > > > >>> > > >> > >> > to use
> > > > >>> > > >> > >> > a Frotran code to re-arrange the fields and
save
> them
> > > as
> > > > >>> > > NCEP-like
> > > > >>> > > >> > grib1
> > > > >>> > > >> > >> > data files.
> > > > >>> > > >> > >> > So after several processes, some important
info
> like
> > > > tables
> > > > >>> > could
> > > > >>> > > >> be
> > > > >>> > > >> > >> lost.
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> >    I'll test your first suggestion since the
field
> > > record
> > > > >>> order
> > > > >>> > > is
> > > > >>> > > >> > >> fixed.
> > > > >>> > > >> > >> > Currently we still
> > > > >>> > > >> > >> > obtained grib1 data from ECWMF, eventually,
we will
> > get
> > > > >>> grib2
> > > > >>> > > data
> > > > >>> > > >> > >> > directly. So before
> > > > >>> > > >> > >> > we can get original grib2 files from ECMWF,
we just
> > use
> > > > >>> this
> > > > >>> > > >> temporal
> > > > >>> > > >> > >> > solution.
> > > > >>> > > >> > >> > '
> > > > >>> > > >> > >> > Thanks!
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley
Gotway
> via
> > > RT
> > > > <
> > > > >>> > > >> > >> > met_help at ucar.edu>
> > > > >>> > > >> > >> > wrote:
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > > Binbin,
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > OK, working on Venus.
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > You have a typo in the command you sent.
> > > > >>> > > >> > >> > >  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
> > a.ps
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >>
> > > > >>> > > >>
> > > > >>> >
> > > > >>>
> > > >
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >>> > > >> > >> > -v
> > > > >>> > > >> > >> > > 3
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
> > > > "GRIB2_parm_cat".
> > > > >>> > After
> > > > >>> > > >> > >> > correcting
> > > > >>> > > >> > >> > > that I did still get an error but it's a
very
> > useful
> > > > one!
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > module use
> > > > >>> > > >> > >>
> > > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> > > > >>> > > >> > >> > > module load met/9.1
> > > > >>> > > >> > >> > > module load grib_util/1.1.1
> > > > >>> > > >> > >> > > cd
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >>
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
> > > > >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
> > > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > > plot.ps
> > > > >>> > > >> > >> > 'GRIB2_disc=255;
> > > > >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3;
GRIB2_parm=5;
> > > > >>> level="P500";'
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > *ERROR  : ERROR  :
VarInfoGrib2::set_dict() -> no
> > > > >>> parameter
> > > > >>> > > found
> > > > >>> > > >> > with
> > > > >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat
(3)
> > > GRIB2_parm
> > > > >>> (5).
> > > > >>> > > Use
> > > > >>> > > >> the
> > > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
define
> > custom
> > > > >>> GRIB
> > > > >>> > > >> > >> tables.ERROR
> > > > >>> > > >> > >> > :
> > > > >>> > > >> > >> > > *
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > So this table is not yet defined in MET. I
> checked
> > > > their
> > > > >>> > > website:
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >>
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > But I don't see an entry defined for
> GRIB2_parm=5;
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > 3 things...
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > (1) As a crude temporary solution, you
could
> refer
> > to
> > > > it
> > > > >>> by
> > > > >>> > > it's
> > > > >>> > > >> > >> record
> > > > >>> > > >> > >> > > number. Let's assume you know that record
1 has
> > > > >>> temperature
> > > > >>> > > data:
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2 ~/
> > > > plot.ps
> > > > >>> > > >> > 'name="TMP";
> > > > >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > Here I'm reading record 1, naming it TMP
and the
> > > using
> > > > >>> > > >> > set_attr_level
> > > > >>> > > >> > >> > write
> > > > >>> > > >> > >> > > the output level as P500 instead of R1.
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > (2) You could create a custom GRIB table
yourself
> > > > >>> locally,
> > > > >>> > and
> > > > >>> > > >> then
> > > > >>> > > >> > >> set
> > > > >>> > > >> > >> > the
> > > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
point MET
> > to
> > > > it.
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > (3) The best solution is contacting ECMWF
and
> > > > requesting
> > > > >>> they
> > > > >>> > > >> send
> > > > >>> > > >> > >> over
> > > > >>> > > >> > >> > the
> > > > >>> > > >> > >> > > GRIB2 table we should be using to decode
this
> data.
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > Thanks,
> > > > >>> > > >> > >> > > John
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John Halley
> Gotway <
> > > > >>> > > >> johnhg at ucar.edu>
> > > > >>> > > >> > >> > wrote:
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > > > We might already have them. I'm not
sure.
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > > > But there's an additional required piece
of the
> > > > puzzle.
> > > > >>> > > Binbin
> > > > >>> > > >> > runs
> > > > >>> > > >> > >> > > > "wgrib2" to figure out what "name" he
should
> use
> > in
> > > > >>> the MET
> > > > >>> > > >> config
> > > > >>> > > >> > >> > files.
> > > > >>> > > >> > >> > > > If wgrib2 doesn't tell him the right
"name",
> the
> > > > >>> problem
> > > > >>> > will
> > > > >>> > > >> > >> remain.
> > > > >>> > > >> > >> > > > wgrib2 can be configured to define
additional
> > > tables
> > > > as
> > > > >>> > well,
> > > > >>> > > >> but
> > > > >>> > > >> > >> for
> > > > >>> > > >> > >> > > this
> > > > >>> > > >> > >> > > > to be easy for Binbin, that'd mean
updating the
> > > > script
> > > > >>> > that's
> > > > >>> > > >> run
> > > > >>> > > >> > >> for
> > > > >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And
I'm
> > guessing
> > > > >>> that's
> > > > >>> > > not
> > > > >>> > > >> > very
> > > > >>> > > >> > >> > easy.
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > > > Yes, I do agree that the best solution
is
> getting
> > > the
> > > > >>> ECMWF
> > > > >>> > > >> GRIB
> > > > >>> > > >> > >> tables
> > > > >>> > > >> > >> > > > and adding them for both wgrib2 and MET.
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > > > John
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara
Jensen <
> > > > >>> > jensen at ucar.edu>
> > > > >>> > > >> > wrote:
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > > >> John,
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> It seems to me that it would be
worthwhile to
> > just
> > > > ask
> > > > >>> > ECMWF
> > > > >>> > > >> for
> > > > >>> > > >> > >> their
> > > > >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.
And
> other
> > > > >>> > operational
> > > > >>> > > >> > >> centers
> > > > >>> > > >> > >> > for
> > > > >>> > > >> > >> > > >> that matter.  Do you agree this would
be a
> good
> > > > >>> solution?
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Thanks, Tara
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> ---------- Forwarded message ---------
> > > > >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
> > > > met_help at ucar.edu>
> > > > >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
> > > > >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769]
urgent
> > > > >>> question for
> > > > >>> > > >> > reading
> > > > >>> > > >> > >> > ECMWF
> > > > >>> > > >> > >> > > >> ensmeble data
> > > > >>> > > >> > >> > > >> To:
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> <URL:
> > > > >>> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> > > > >>> > > >> >
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> John,
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>   I tested using following command:
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>  plot_data_plane
> ecme.ens01.t00z.grd3.f12.grib2
> > > > a.ps
> > > > >>> > > >> > >> > > >>
> > > > >>> 'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >>> > > >> -v3
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> ERROR  :
> > > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either
> name
> > > or
> > > > >>> > > >> GRIB2_disc,
> > > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
> specified
> > in
> > > > >>> field
> > > > >>> > > >> > >> information
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>  plot_data_plane
> ecme.ens01.t00z.grd3.f12.grib2
> > > > a.ps
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> >
> > > > >>> > > >> >
> > > > >>> > >
> > > > >>>
> > > >
> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
> > > > >>> > > >> > >> > > >> -v
> > > > >>> > > >> > >> > > >> 3
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Still got error
> > > > >>> > > >> > >> > > >> ERROR  :
> > > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either
> name
> > > or
> > > > >>> > > >> GRIB2_disc,
> > > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
> specified
> > in
> > > > >>> field
> > > > >>> > > >> > >> information
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Looks like discipline=255 is not
recognized
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Binbin
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John
Halley
> > Gotway
> > > > >>> via RT
> > > > >>> > <
> > > > >>> > > >> > >> > > >> met_help at ucar.edu>
> > > > >>> > > >> > >> > > >> wrote:
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> > Binbin,
> > > > >>> > > >> > >> > > >> > You can do something similar with
MET.
> Please
> > go
> > > > to
> > > > >>> this
> > > > >>> > > >> link:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >>
> > > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > And then search for "GRIB2_mtab", and
you'll
> > > find
> > > > >>> the
> > > > >>> > > >> content
> > > > >>> > > >> > >> > > >> > copied/pasted below. You can specify
these
> > > INSTEAD
> > > > >>> OF
> > > > >>> > > >> > specifying
> > > > >>> > > >> > >> a
> > > > >>> > > >> > >> > > >> > requested name.
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an
integer
> > > > >>> specifying the
> > > > >>> > > >> master
> > > > >>> > > >> > >> > table
> > > > >>> > > >> > >> > > >> > number.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an
integer
> > > > >>> specifying the
> > > > >>> > > >> local
> > > > >>> > > >> > >> table
> > > > >>> > > >> > >> > > >> > number.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an
integer
> > > > >>> specifying the
> > > > >>> > > >> GRIB2
> > > > >>> > > >> > >> > > >> discipline
> > > > >>> > > >> > >> > > >> > code.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is an
> integer
> > > > >>> specifying
> > > > >>> > > the
> > > > >>> > > >> > >> > parameter
> > > > >>> > > >> > >> > > >> > category
> > > > >>> > > >> > >> > > >> > //         code.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an
integer
> > > > >>> specifying the
> > > > >>> > > >> > >> parameter
> > > > >>> > > >> > >> > > code.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an
integer
> > > > specifying
> > > > >>> the
> > > > >>> > > >> product
> > > > >>> > > >> > >> > > >> definition
> > > > >>> > > >> > >> > > >> > //         template (Table 4.0).
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
integer
> > > > >>> specifying
> > > > >>> > > the
> > > > >>> > > >> > >> > generating
> > > > >>> > > >> > >> > > >> > process
> > > > >>> > > >> > >> > > >> > //         (Table 4.3).
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an
integer
> > > > >>> specifying the
> > > > >>> > > >> > >> originating
> > > > >>> > > >> > >> > > >> > center.
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is an
> integer
> > > > >>> specifying
> > > > >>> > > the
> > > > >>> > > >> > >> > ensemble
> > > > >>> > > >> > >> > > >> type
> > > > >>> > > >> > >> > > >> > //         (Table 4.6).
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is an
> integer
> > > > >>> specifying
> > > > >>> > > the
> > > > >>> > > >> > >> derived
> > > > >>> > > >> > >> > > >> > product
> > > > >>> > > >> > >> > > >> > //         type (Table 4.7).
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is
an
> integer
> > > > >>> > specifying
> > > > >>> > > >> the
> > > > >>> > > >> > >> > > >> statistical
> > > > >>> > > >> > >> > > >> > //         processing type (Table
4.10).
> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index"
and
> > > > >>> > > "GRIB2_ipdtmpl_val"
> > > > >>> > > >> > >> entries
> > > > >>> > > >> > >> > > are
> > > > >>> > > >> > >> > > >> > arrays
> > > > >>> > > >> > >> > > >> > //         of integers which specify
the
> > product
> > > > >>> > > description
> > > > >>> > > >> > >> > template
> > > > >>> > > >> > >> > > >> > values to
> > > > >>> > > >> > >> > > >> > //         be used. The indices are
0-based.
> > For
> > > > >>> > example,
> > > > >>> > > >> use
> > > > >>> > > >> > the
> > > > >>> > > >> > >> > > >> > following to
> > > > >>> > > >> > >> > > >> > //         request a GRIB2 record
whose 9-th
> > and
> > > > >>> 27-th
> > > > >>> > > >> product
> > > > >>> > > >> > >> > > >> description
> > > > >>> > > >> > >> > > >> > //         template values are 1 and
2,
> > > > >>> respectively:
> > > > >>> > > >> > >> > > >> > //            GRIB2_ipdtmpl_index=[8,
26];
> > > > >>> > > >> > GRIB2_ipdtmpl_val=[1,
> > > > >>> > > >> > >> 2];
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > While I don't have any ECMWF data on
hand
> for
> > > > >>> testing, I
> > > > >>> > > do
> > > > >>> > > >> > have
> > > > >>> > > >> > >> a
> > > > >>> > > >> > >> > > file
> > > > >>> > > >> > >> > > >> > from the Air Force from which wgrib2
prints
> a
> > > > >>> similar
> > > > >>> > > >> result:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var
discipline=0
> > > > >>> center=57
> > > > >>> > > >> > >> local_table=1
> > > > >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour
fcst:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > So I used that for testing. I ran the
> > following
> > > > >>> > > >> plot_data_plane
> > > > >>> > > >> > >> > > command
> > > > >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > plot_data_plane
> GALWEM_20170610_CY00_FH18.GR2
> > > > >>> plot.ps
> > > > >>> > > >> > >> > 'GRIB2_disc=0;
> > > > >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
> GRIB2_parm_cat=1;
> > > > >>> > > >> GRIB2_parm=192;
> > > > >>> > > >> > >> > > >> level="Z0";'
> > > > >>> > > >> > >> > > >> > -v 3
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > And then I looked closely at the log
message
> > to
> > > > >>> confirm
> > > > >>> > > >> that it
> > > > >>> > > >> > >> used
> > > > >>> > > >> > >> > > >> > record 549, as expected:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > DEBUG 3:
MetGrib2DataFile::data_plane() ->
> > Found
> > > > >>> exact
> > > > >>> > > match
> > > > >>> > > >> > for
> > > > >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field
1 of
> > GRIB2
> > > > file
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >>
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I
defined to
> > do
> > > a
> > > > >>> table
> > > > >>> > > >> lookup
> > > > >>> > > >> > >> and
> > > > >>> > > >> > >> > > find
> > > > >>> > > >> > >> > > >> a
> > > > >>> > > >> > >> > > >> > match in the MET GRIB tables:
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >>
> > > > >>> > >
> > > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > And it determined that that field
should be
> > > called
> > > > >>> > > "CTSTM".
> > > > >>> > > >> > >> > Hopefully,
> > > > >>> > > >> > >> > > >> the
> > > > >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already
defined there
> > in
> > > > >>> MET. If
> > > > >>> > > >> not, we
> > > > >>> > > >> > >> can
> > > > >>> > > >> > >> > > >> always
> > > > >>> > > >> > >> > > >> > add them.
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > Please try running plot_data_plane at
> > verbosity
> > > > >>> level 3
> > > > >>> > on
> > > > >>> > > >> one
> > > > >>> > > >> > of
> > > > >>> > > >> > >> > > these
> > > > >>> > > >> > >> > > >> > files first. And make sure it runs
without
> > error
> > > > >>> and the
> > > > >>> > > log
> > > > >>> > > >> > >> message
> > > > >>> > > >> > >> > > >> prints
> > > > >>> > > >> > >> > > >> > out the "name" it found for that
data. If
> so,
> > > you
> > > > >>> can
> > > > >>> > just
> > > > >>> > > >> skip
> > > > >>> > > >> > >> > using
> > > > >>> > > >> > >> > > >> all
> > > > >>> > > >> > >> > > >> > those integers and revert back to
using the
> > name
> > > > MET
> > > > >>> > > knows.
> > > > >>> > > >> If
> > > > >>> > > >> > >> not,
> > > > >>> > > >> > >> > > >> then we
> > > > >>> > > >> > >> > > >> > may need to define a custom GRIB
table for
> > this
> > > > >>> ECMWF
> > > > >>> > > GRIB2
> > > > >>> > > >> > data.
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >> > Thanks,
> > > > >>> > > >> > >> > > >> > John
> > > > >>> > > >> > >> > > >> >
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> --
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Binbin Zhou
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> 5830 University Research Ct.
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> College Park, MD 20740
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> 301-683-3683
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >> --
> > > > >>> > > >> > >> > > >>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > >>> > > >> > >> > > >> Tara Jensen
> > > > >>> > > >> > >> > > >> Project Manager II
> > > > >>> > > >> > >> > > >> NCAR RAL and DTC
> > > > >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307
USA
> > > > >>> > > >> > >> > > >> +1 303-497-8479
jensen at ucar.edu
> > > > >>> > > >> > >> > > >>
> > > > >>> > > >> > >> > > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> > >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > --
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > Binbin Zhou
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > 5830 University Research Ct.
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > College Park, MD 20740
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > Binbin.Zhou at noaa.gov
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> > 301-683-3683
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >> >
> > > > >>> > > >> > >>
> > > > >>> > > >> > >>
> > > > >>> > > >> > >
> > > > >>> > > >> > > --
> > > > >>> > > >> > >
> > > > >>> > > >> > > Binbin Zhou
> > > > >>> > > >> > >
> > > > >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > > >> > >
> > > > >>> > > >> > > 5830 University Research Ct.
> > > > >>> > > >> > >
> > > > >>> > > >> > > College Park, MD 20740
> > > > >>> > > >> > >
> > > > >>> > > >> > > Binbin.Zhou at noaa.gov
> > > > >>> > > >> > >
> > > > >>> > > >> > > 301-683-3683
> > > > >>> > > >> > >
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >> > --
> > > > >>> > > >> >
> > > > >>> > > >> > Binbin Zhou
> > > > >>> > > >> >
> > > > >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > > >> >
> > > > >>> > > >> > 5830 University Research Ct.
> > > > >>> > > >> >
> > > > >>> > > >> > College Park, MD 20740
> > > > >>> > > >> >
> > > > >>> > > >> > Binbin.Zhou at noaa.gov
> > > > >>> > > >> >
> > > > >>> > > >> > 301-683-3683
> > > > >>> > > >> >
> > > > >>> > > >> >
> > > > >>> > > >>
> > > > >>> > > >>
> > > > >>> > > >
> > > > >>> > > > --
> > > > >>> > > >
> > > > >>> > > > Binbin Zhou
> > > > >>> > > >
> > > > >>> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > > >
> > > > >>> > > > 5830 University Research Ct.
> > > > >>> > > >
> > > > >>> > > > College Park, MD 20740
> > > > >>> > > >
> > > > >>> > > > Binbin.Zhou at noaa.gov
> > > > >>> > > >
> > > > >>> > > > 301-683-3683
> > > > >>> > > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > --
> > > > >>> > >
> > > > >>> > > Binbin Zhou
> > > > >>> > >
> > > > >>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>> > >
> > > > >>> > > 5830 University Research Ct.
> > > > >>> > >
> > > > >>> > > College Park, MD 20740
> > > > >>> > >
> > > > >>> > > Binbin.Zhou at noaa.gov
> > > > >>> > >
> > > > >>> > > 301-683-3683
> > > > >>> > >
> > > > >>> > >
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > > >>> --
> > > > >>>
> > > > >>> Binbin Zhou
> > > > >>>
> > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > >>>
> > > > >>> 5830 University Research Ct.
> > > > >>>
> > > > >>> College Park, MD 20740
> > > > >>>
> > > > >>> Binbin.Zhou at noaa.gov
> > > > >>>
> > > > >>> 301-683-3683
> > > > >>>
> > > > >>>
> > > >
> > > > --
> > > > Julie Prestopnik (she/her)
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day.  Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: Binbin.Zhou at noaa.gov
Time: Fri Dec 11 10:20:46 2020

John,

  As I tested following plotting

plot_data_plane ecmanl.t12z.grid3.f00.grib1 plot.ps '*name="HGT";
level="P500"; lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3

the error message is

ERROR  :
ERROR  : yyerror() -> syntax error in file "/tmp/met_config_9350_0"
ERROR  :
ERROR  :    line   = 1
ERROR  :
ERROR  :    column = 1
ERROR  :
ERROR  :    text   = "*"
ERROR  :
ERROR  :
ERROR  : *name="HGT"; level="P500"; lead_time="00";
GRIB_ens="low_res_ctl";*
ERROR  :
^__________________________________________________________________
ERROR  :

After I removed "*"
  plot_data_plane ecmanl.t12z.grid3.f00.grib1 plot.ps 'name="HGT";
level="P500"; lead_time="00"; GRIB_ens="low_res_ctl";' -v 3
It shows 3 records matched

DEBUG 1: Opening data file: ecmanl.t12z.grid3.f00.grib1
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match for
VarInfo "HGT/P500" in GRIB record 1 of GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match for
VarInfo "HGT/P500" in GRIB record 2 of GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match for
VarInfo "HGT/P500" in GRIB record 3 of GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB records
matching VarInfo "HGT/P500" in GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 1: Creating postscript file: plot.ps

My testing directory on Venus
is /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0_grib1
test script is ploting.sh

Can you look at it?

Thanks!
Binbin


On Thu, Dec 10, 2020 at 3:17 PM Binbin Zhou - NOAA Affiliate <
binbin.zhou at noaa.gov> wrote:

> John,
>
>   Thanks!
>
> Binbin
>
> On Thu, Dec 10, 2020 at 2:06 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Thanks Julie.
>>
>> And Binbin, here's an EnsembleStatConfig file where climos are
used:
>>
>>
>>
https://github.com/dtcenter/MET/blob/main_v9.1/test/config/EnsembleStatConfig_climo
>>
>> Since the list of forecast fields is exactly the same as the climo
fields,
>> this config file just copies them over using:
>>    climo_mean = fcst;
>>
>> In your case, when they differ, you'd just specify the "field"
array
>> inside
>> the "climo_mean" dictionary. For example:
>>
>> climo_mean = {
>>
>>    field = [
>>       { name  = "TMP"; level = "Z2";   },
>>       { name  = "TMP"; level = "P850"; }
>>    ];
>>    ...
>> }
>>
>> On Thu, Dec 10, 2020 at 12:00 PM Julie Prestopnik via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> >
>> > Thanks, John.  That worked great.
>> >
>> > Binbin, this version of MET is now accessible to you by running
the
>> > following on venus:
>> >
>> > module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>> > module load met/ens_config_test
>> >
>> > Please let us know if you encounter any problems.
>> >
>> > Julie
>> >
>> > On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > >
>> > > Julie,
>> > >
>> > > Building the tarfile is very easy. I just ran this script from
the MET
>> > repo
>> > > and specified the branch to be built:
>> > >    MET/scripts/met_checkout_and_build.sh branch
>> feature_1604_ens_config
>> > >
>> > > And that created a tarfile which I posted to RAL's anonymous
ftp
>> server:
>> > >
>> > >
>> > >
>> >
>> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
>> > >
>> > > I assume that'll work for you.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT <
>> > met_help at ucar.edu
>> > > >
>> > > wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
>> > > >
>> > > > John,
>> > > >
>> > > > I am able to help with this install on WCOSS.
>> > > >
>> > > > Would it be easier to use git to checkout the repo on WCOSS?
Or
>> would
>> > it
>> > > be
>> > > > > easier if I built a tarfile for this version of the code?
>> > > > >
>> > > > The last time I tried to build from git on WCOSS (a while
ago), I
>> > > > couldn't.  I don't remember all of the details, but I think I
>> couldn't
>> > > run
>> > > > bootstrap and maybe it had to do with the version of autoconf
being
>> too
>> > > > old?  I'm happy to try it again if you'd like though.
Otherwise, I
>> > know
>> > > a
>> > > > tarfile would work, but I know that is extra work for you.
>> > > >
>> > > > Julie
>> > > >
>> > > > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway
<johnhg at ucar.edu>
>> > > wrote:
>> > > >
>> > > > > Julie,
>> > > > >
>> > > > > I have created a bugfix branch for main_v9.1 named
>> > > > > "feature_1604_ens_config" and it's up on GitHub.
>> > > > >
>> > > > > Only a single file has changed in it. But I need to get
this
>> version
>> > > > > compiled on WCOSS with which Binbin can test.
>> > > > >
>> > > > > Are you to help me with this? Would it be easier to use git
to
>> > checkout
>> > > > > the repo on WCOSS? Or would it be easier if I built a
tarfile for
>> > this
>> > > > > version of the code?
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway <
>> johnhg at ucar.edu>
>> > > > wrote:
>> > > > >
>> > > > >> Binbin,
>> > > > >>
>> > > > >> In order for you to test this change out, we'd need to
recompile
>> a
>> > > > >> version of the code on WCOSS. I wrote this up as a GitHub
issue:
>> > > > >> https://github.com/dtcenter/MET/issues/1604
>> > > > >>
>> > > > >> And I'll create a branch from main_v9.1 named
>> > "bugfix_1604_ens_config"
>> > > > >> and commit the change there. Then we can checkout that
code on
>> > WCOSS,
>> > > > >> compile it, and you can test it.
>> > > > >>
>> > > > >> Yes, you're right, if the fcst fields differ from the
climo
>> fields,
>> > > > you'd
>> > > > >> need to specify them separately. That's pretty easy in the
>> > > Ensemble-Stat
>> > > > >> config file... but are you asking how to control that via
>> METplus?
>> > > > >>
>> > > > >> John
>> > > > >>
>> > > > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via RT
<
>> > > > >> met_help at ucar.edu> wrote:
>> > > > >>
>> > > > >>>
>> > > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>> > > > >>>
>> > > > >>> John,
>> > > > >>>
>> > > > >>>   Great! Let me test it at first before make change in
MET9.1
>> > > > >>> Another question:
>> > > > >>>    Since climatology data is still in NCEP's grib2
format, In
>> the
>> > > > >>> ensemble_stat
>> > > > >>> config file,  climo  entry settings  should be different
from
>> those
>> > > of
>> > > > >>> fcst, obs entries?
>> > > > >>> How to set climo (name, level, etc.)?
>> > > > >>>
>> > > > >>> Binbin
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via RT
<
>> > > > >>> met_help at ucar.edu>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>> > Binbin,
>> > > > >>> >
>> > > > >>> > Success! When I recompile with that one line commented
out,
>> > > > >>> plot_data_plane
>> > > > >>> > is able to filter down all those records to the single
one
>> that
>> > we
>> > > > >>> want.
>> > > > >>> > Note in the example below that I used the "lead_time"
and
>> > > "GRIB_ens"
>> > > > >>> > options to filter the input records down.
>> > > > >>> >
>> > > > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps
'*name="HGT";
>> > > > >>> level="P500";
>> > > > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
>> > > > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
>> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> *Found
exact
>> > match
>> > > > for
>> > > > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
>> > > > >>> "ensposte.t00z.z500hr".*
>> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found
1 GRIB
>> > > records
>> > > > >>> > matching VarInfo "HGT/P500" in GRIB file
>> "ensposte.t00z.z500hr".
>> > > > >>> > DEBUG 1: Creating postscript file: plot.ps
>> > > > >>> >
>> > > > >>> > Also note that this selects out record number 26.
>> > > > >>> >
>> > > > >>> > Shall I make this change as a "bugfix" for met-
9.1...which
>> could
>> > be
>> > > > >>> > included in a met-9.1.2 bugfix release? Or shall I just
>> include
>> > it
>> > > in
>> > > > >>> > development for the next release (planned for March
2021)?
>> > > > >>> >
>> > > > >>> > Thanks,
>> > > > >>> > John
>> > > > >>> >
>> > > > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov via
RT <
>> > > > >>> > met_help at ucar.edu> wrote:
>> > > > >>> >
>> > > > >>> > >
>> > > > >>> > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > >
>> > > > >>> > >
>> > > > >>> > > John,
>> > > > >>> > >
>> > > > >>> > >   If this removing impacts other applications, or
takes some
>> > time
>> > > > >>> > > to be effective, then won't try.  I will use another
>> approach,
>> > > such
>> > > > >>> as
>> > > > >>> > > borrow GEFS's grib2 data structure as tank to save EC
>> ensemble
>> > > > >>> > > data.
>> > > > >>> > >
>> > > > >>> > > Binbin
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
Affiliate
>> <
>> > > > >>> > > binbin.zhou at noaa.gov> wrote:
>> > > > >>> > >
>> > > > >>> > > > John,
>> > > > >>> > > >
>> > > > >>> > > >   Yes, please try to use this approach.
>> > > > >>> > > >
>> > > > >>> > > > Thanks!
>> > > > >>> > > >
>> > > > >>> > > > Binbin
>> > > > >>> > > >
>> > > > >>> > > > Binbin
>> > > > >>> > > >
>> > > > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway
via RT <
>> > > > >>> > > > met_help at ucar.edu> wrote:
>> > > > >>> > > >
>> > > > >>> > > >> Binbin,
>> > > > >>> > > >>
>> > > > >>> > > >> I used the "-PDS10" option for wgrib to dump those
>> records:
>> > > > >>> > > >>
>> > > > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
>> > > > >>> > > >>
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
>> > > > >>> > > >> | egrep  "^11:|^26:|^43:"
>> > > > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
>> 0 1
>> > 0
>> > > 0
>> > > > 1
>> > > > >>> 0 0
>> > > > >>> > 0
>> > > > >>> > > 21
>> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
>> > > > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
>> 0 1
>> > 0
>> > > 0
>> > > > 1
>> > > > >>> 0 0
>> > > > >>> > 0
>> > > > >>> > > 21
>> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
>> > > > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
>> 0 1
>> > 0
>> > > 0
>> > > > 1
>> > > > >>> 0 0
>> > > > >>> > 0
>> > > > >>> > > 21
>> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
>> > > > >>> > > >>
>> > > > >>> > > >> Looking VERY closely, there is a difference in the
42-nd
>> > > column
>> > > > >>> which
>> > > > >>> > is
>> > > > >>> > > >> the 4-th from the last... records 11, 26, and 43
have
>> values
>> > > of
>> > > > >>> 3, 1,
>> > > > >>> > > and
>> > > > >>> > > >> 2
>> > > > >>> > > >> respectively.
>> > > > >>> > > >>
>> > > > >>> > > >> On this page (
>> > > > >>> > >
https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
>> > > > >>> > > >> ),
>> > > > >>> > > >> byte 42 = 1 means "unperturbed control forecast".
>> > > > >>> > > >>
>> > > > >>> > > >> So we *should* be able to use the "GRIB_ens"
config
>> option
>> > to
>> > > > >>> > > >> differentiate
>> > > > >>> > > >> between those records. BUT when I try, it doesn't
work.
>> And
>> > > > that's
>> > > > >>> > > because
>> > > > >>> > > >> of this line:
>> > > > >>> > > >>
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
>> > > > >>> > > >>
>> > > > >>> > > >> We only apply those filtering options to GRIB
files
>> created
>> > by
>> > > > >>> NCEP.
>> > > > >>> > If
>> > > > >>> > > >> you'd like, I could try removing that condition to
see
>> if we
>> > > can
>> > > > >>> pick
>> > > > >>> > > out
>> > > > >>> > > >> the single record of interest.
>> > > > >>> > > >>
>> > > > >>> > > >> John
>> > > > >>> > > >>
>> > > > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM
Binbin.Zhou at noaa.gov via
>> RT
>> > <
>> > > > >>> > > >> met_help at ucar.edu> wrote:
>> > > > >>> > > >>
>> > > > >>> > > >> >
>> > > > >>> > > >> > <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > > > >
>> > > > >>> > > >> >
>> > > > >>> > > >> > John,
>> > > > >>> > > >> >
>> > > > >>> > > >> >   Look at my
>> > > > >>> > > >> > directory
>> > > > >>> /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
>> > > > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this is
only
>> for
>> > 500
>> > > > >>> HGT.
>> > > > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
>> anl:type=analysis
>> > > > >>> > > >> > I get 3 records:
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
>> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
>> > > > >>> > > >> >
>> > > > >>> > > >> > From its grib2 file, we know only  record#26 is
>> analysis
>> > > > low-res
>> > > > >>> > > >> control.
>> > > > >>> > > >> > If you have some way to determine which record
is
>> analysis
>> > > > >>> low-res
>> > > > >>> > > >> control
>> > > > >>> > > >> > from the grib1 file,
>> > > > >>> > > >> > then we can use grib1 directly (don't consider
the size
>> > > > issue).
>> > > > >>> > > >> >
>> > > > >>> > > >> > Binbin
>> > > > >>> > > >> >
>> > > > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou -
NOAA
>> > Affiliate
>> > > <
>> > > > >>> > > >> > binbin.zhou at noaa.gov> wrote:
>> > > > >>> > > >> >
>> > > > >>> > > >> > > John,
>> > > > >>> > > >> > >
>> > > > >>> > > >> > >   Grib1 data is too large, another issue is
the
>> low-res
>> > > > >>> control
>> > > > >>> > > >> analysis
>> > > > >>> > > >> > > data can not be retrieved
>> > > > >>> > > >> > > from the grib1 ecmwf data files. If using
wgrib, we
>> can
>> > > see
>> > > > 3
>> > > > >>> > > analysis
>> > > > >>> > > >> > > fields, but don't know
>> > > > >>> > > >> > > which is low-res control. After convert to
grib2,
>> wgrib2
>> > > can
>> > > > >>> show
>> > > > >>> > > >> which
>> > > > >>> > > >> > > record is low-res control.
>> > > > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data
set on
>> > WCOSS.
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > Binbin
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley
Gotway via
>> > RT <
>> > > > >>> > > >> > > met_help at ucar.edu> wrote:
>> > > > >>> > > >> > >
>> > > > >>> > > >> > >> Binbin,
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >> Sounds like there's a lot of moving parts
there. How
>> > > would
>> > > > >>> you
>> > > > >>> > like
>> > > > >>> > > >> to
>> > > > >>> > > >> > >> proceed?
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >> Should we check to see if MET can read those
ECMWF
>> > GRIB1
>> > > > >>> files
>> > > > >>> > > >> directly
>> > > > >>> > > >> > >> without needing to reformat?
>> > > > >>> > > >> > >> Or do you just want to create a custom GRIB
table
>> for
>> > the
>> > > > >>> result
>> > > > >>> > of
>> > > > >>> > > >> your
>> > > > >>> > > >> > >> pre-processing and conversion steps?
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >> John
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
Binbin.Zhou at noaa.gov
>> > via
>> > > > RT <
>> > > > >>> > > >> > >> met_help at ucar.edu> wrote:
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > <URL:
>> > > > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > > > >>> > >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > John,
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
>> converted
>> > > from
>> > > > >>> its
>> > > > >>> > > grib1
>> > > > >>> > > >> > >> format
>> > > > >>> > > >> > >> > data.
>> > > > >>> > > >> > >> > Before conversion, the parameter names are
there,
>> but
>> > > > after
>> > > > >>> > > >> > conversion,
>> > > > >>> > > >> > >> the
>> > > > >>> > > >> > >> > names
>> > > > >>> > > >> > >> > are gone. It is because of unknown tables.
The
>> grib1
>> > > data
>> > > > >>> is
>> > > > >>> > > still
>> > > > >>> > > >> > >> > original. The original
>> > > > >>> > > >> > >> > grib1 data files are very strange data
format we
>> can
>> > > not
>> > > > >>> read.
>> > > > >>> > So
>> > > > >>> > > >> we
>> > > > >>> > > >> > >> have
>> > > > >>> > > >> > >> > to use
>> > > > >>> > > >> > >> > a Frotran code to re-arrange the fields and
save
>> them
>> > > as
>> > > > >>> > > NCEP-like
>> > > > >>> > > >> > grib1
>> > > > >>> > > >> > >> > data files.
>> > > > >>> > > >> > >> > So after several processes, some important
info
>> like
>> > > > tables
>> > > > >>> > could
>> > > > >>> > > >> be
>> > > > >>> > > >> > >> lost.
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> >    I'll test your first suggestion since
the field
>> > > record
>> > > > >>> order
>> > > > >>> > > is
>> > > > >>> > > >> > >> fixed.
>> > > > >>> > > >> > >> > Currently we still
>> > > > >>> > > >> > >> > obtained grib1 data from ECWMF, eventually,
we
>> will
>> > get
>> > > > >>> grib2
>> > > > >>> > > data
>> > > > >>> > > >> > >> > directly. So before
>> > > > >>> > > >> > >> > we can get original grib2 files from ECMWF,
we
>> just
>> > use
>> > > > >>> this
>> > > > >>> > > >> temporal
>> > > > >>> > > >> > >> > solution.
>> > > > >>> > > >> > >> > '
>> > > > >>> > > >> > >> > Thanks!
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > On Fri, Dec 4, 2020 at 5:48 PM John Halley
Gotway
>> via
>> > > RT
>> > > > <
>> > > > >>> > > >> > >> > met_help at ucar.edu>
>> > > > >>> > > >> > >> > wrote:
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > > Binbin,
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > OK, working on Venus.
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > You have a typo in the command you sent.
>> > > > >>> > > >> > >> > >  plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
>> > a.ps
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >>
>> > > > >>> > > >>
>> > > > >>> >
>> > > > >>>
>> > > >
>> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > > >>> > > >> > >> > -v
>> > > > >>> > > >> > >> > > 3
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > You have "RIB2_parm_cat" that should be
>> > > > "GRIB2_parm_cat".
>> > > > >>> > After
>> > > > >>> > > >> > >> > correcting
>> > > > >>> > > >> > >> > > that I did still get an error but it's a
very
>> > useful
>> > > > one!
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > module use
>> > > > >>> > > >> > >>
>> > > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
>> > > > >>> > > >> > >> > > module load met/9.1
>> > > > >>> > > >> > >> > > module load grib_util/1.1.1
>> > > > >>> > > >> > >> > > cd
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >>
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/g2g/com/verf/dev/test_ecmanl/grib2
>> > > > >>> > > >> > >> > > $WGRIB2 ensposte.t00z.z500.grd3.grib2
>> > > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
>> ~/
>> > > > plot.ps
>> > > > >>> > > >> > >> > 'GRIB2_disc=255;
>> > > > >>> > > >> > >> > > GRIB2_mtab=2; GRIB2_parm_cat=3;
GRIB2_parm=5;
>> > > > >>> level="P500";'
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > *ERROR  : ERROR  :
VarInfoGrib2::set_dict() ->
>> no
>> > > > >>> parameter
>> > > > >>> > > found
>> > > > >>> > > >> > with
>> > > > >>> > > >> > >> > > matching GRIB2_disc (255) GRIB2_parm_cat
(3)
>> > > GRIB2_parm
>> > > > >>> (5).
>> > > > >>> > > Use
>> > > > >>> > > >> the
>> > > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
define
>> > custom
>> > > > >>> GRIB
>> > > > >>> > > >> > >> tables.ERROR
>> > > > >>> > > >> > >> > :
>> > > > >>> > > >> > >> > > *
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > So this table is not yet defined in MET.
I
>> checked
>> > > > their
>> > > > >>> > > website:
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >>
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>> https://apps.ecmwf.int/codes/grib/param-
db?&discipline=255&category=3&filter=grib2
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > But I don't see an entry defined for
>> GRIB2_parm=5;
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > 3 things...
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > (1) As a crude temporary solution, you
could
>> refer
>> > to
>> > > > it
>> > > > >>> by
>> > > > >>> > > it's
>> > > > >>> > > >> > >> record
>> > > > >>> > > >> > >> > > number. Let's assume you know that record
1 has
>> > > > >>> temperature
>> > > > >>> > > data:
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > plot_data_plane
ecme.ens01.t00z.grd3.f12.grib2
>> ~/
>> > > > plot.ps
>> > > > >>> > > >> > 'name="TMP";
>> > > > >>> > > >> > >> > > level="R1"; set_attr_level="P500";' -v 5
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > Here I'm reading record 1, naming it TMP
and the
>> > > using
>> > > > >>> > > >> > set_attr_level
>> > > > >>> > > >> > >> > write
>> > > > >>> > > >> > >> > > the output level as P500 instead of R1.
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > (2) You could create a custom GRIB table
>> yourself
>> > > > >>> locally,
>> > > > >>> > and
>> > > > >>> > > >> then
>> > > > >>> > > >> > >> set
>> > > > >>> > > >> > >> > the
>> > > > >>> > > >> > >> > > MET_GRIB_TABLES environment variable to
point
>> MET
>> > to
>> > > > it.
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > (3) The best solution is contacting ECMWF
and
>> > > > requesting
>> > > > >>> they
>> > > > >>> > > >> send
>> > > > >>> > > >> > >> over
>> > > > >>> > > >> > >> > the
>> > > > >>> > > >> > >> > > GRIB2 table we should be using to decode
this
>> data.
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > Thanks,
>> > > > >>> > > >> > >> > > John
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > On Fri, Dec 4, 2020 at 3:30 PM John
Halley
>> Gotway <
>> > > > >>> > > >> johnhg at ucar.edu>
>> > > > >>> > > >> > >> > wrote:
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > > > We might already have them. I'm not
sure.
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > > > But there's an additional required
piece of
>> the
>> > > > puzzle.
>> > > > >>> > > Binbin
>> > > > >>> > > >> > runs
>> > > > >>> > > >> > >> > > > "wgrib2" to figure out what "name" he
should
>> use
>> > in
>> > > > >>> the MET
>> > > > >>> > > >> config
>> > > > >>> > > >> > >> > files.
>> > > > >>> > > >> > >> > > > If wgrib2 doesn't tell him the right
"name",
>> the
>> > > > >>> problem
>> > > > >>> > will
>> > > > >>> > > >> > >> remain.
>> > > > >>> > > >> > >> > > > wgrib2 can be configured to define
additional
>> > > tables
>> > > > as
>> > > > >>> > well,
>> > > > >>> > > >> but
>> > > > >>> > > >> > >> for
>> > > > >>> > > >> > >> > > this
>> > > > >>> > > >> > >> > > > to be easy for Binbin, that'd mean
updating
>> the
>> > > > script
>> > > > >>> > that's
>> > > > >>> > > >> run
>> > > > >>> > > >> > >> for
>> > > > >>> > > >> > >> > > > "module load grib_utils" on WCOSS. And
I'm
>> > guessing
>> > > > >>> that's
>> > > > >>> > > not
>> > > > >>> > > >> > very
>> > > > >>> > > >> > >> > easy.
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > > > Yes, I do agree that the best solution
is
>> getting
>> > > the
>> > > > >>> ECMWF
>> > > > >>> > > >> GRIB
>> > > > >>> > > >> > >> tables
>> > > > >>> > > >> > >> > > > and adding them for both wgrib2 and
MET.
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > > > John
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > > > On Fri, Dec 4, 2020 at 2:32 PM Tara
Jensen <
>> > > > >>> > jensen at ucar.edu>
>> > > > >>> > > >> > wrote:
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > > >> John,
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> It seems to me that it would be
worthwhile to
>> > just
>> > > > ask
>> > > > >>> > ECMWF
>> > > > >>> > > >> for
>> > > > >>> > > >> > >> their
>> > > > >>> > > >> > >> > > >> Grib2 Table for their Ensemble files.
And
>> other
>> > > > >>> > operational
>> > > > >>> > > >> > >> centers
>> > > > >>> > > >> > >> > for
>> > > > >>> > > >> > >> > > >> that matter.  Do you agree this would
be a
>> good
>> > > > >>> solution?
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Thanks, Tara
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> ---------- Forwarded message ---------
>> > > > >>> > > >> > >> > > >> From: Binbin.Zhou at noaa.gov via RT <
>> > > > met_help at ucar.edu>
>> > > > >>> > > >> > >> > > >> Date: Fri, Dec 4, 2020 at 11:06 AM
>> > > > >>> > > >> > >> > > >> Subject: Re: [rt.rap.ucar.edu #97769]
urgent
>> > > > >>> question for
>> > > > >>> > > >> > reading
>> > > > >>> > > >> > >> > ECMWF
>> > > > >>> > > >> > >> > > >> ensmeble data
>> > > > >>> > > >> > >> > > >> To:
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> <URL:
>> > > > >>> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>> > > > >>> > > >> >
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> John,
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>   I tested using following command:
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>  plot_data_plane
>> ecme.ens01.t00z.grd3.f12.grib2
>> > > > a.ps
>> > > > >>> > > >> > >> > > >>
>> > > > >>>
'GRIB2_mtab=2;GRIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > > >>> > > >> -v3
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> ERROR  :
>> > > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either
>> name
>> > > or
>> > > > >>> > > >> GRIB2_disc,
>> > > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
>> specified
>> > in
>> > > > >>> field
>> > > > >>> > > >> > >> information
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>  plot_data_plane
>> ecme.ens01.t00z.grd3.f12.grib2
>> > > > a.ps
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> >
>> > > > >>> > >
>> > > > >>>
>> > > >
>> >
'GRIB2_mtab=2;GRIB2_disc=255;RIB2_parm_cat=3;GRIB2_parm=5;level="P500";'
>> > > > >>> > > >> > >> > > >> -v
>> > > > >>> > > >> > >> > > >> 3
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Still got error
>> > > > >>> > > >> > >> > > >> ERROR  :
>> > > > >>> > > >> > >> > > >> ERROR  : VarInfoGrib2::set_dict() ->
either
>> name
>> > > or
>> > > > >>> > > >> GRIB2_disc,
>> > > > >>> > > >> > >> > > >> GRIB2_parm_cat and GRIB2_parm must be
>> specified
>> > in
>> > > > >>> field
>> > > > >>> > > >> > >> information
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Looks like discipline=255 is not
recognized
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Binbin
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> On Fri, Dec 4, 2020 at 11:57 AM John
Halley
>> > Gotway
>> > > > >>> via RT
>> > > > >>> > <
>> > > > >>> > > >> > >> > > >> met_help at ucar.edu>
>> > > > >>> > > >> > >> > > >> wrote:
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> > Binbin,
>> > > > >>> > > >> > >> > > >> > You can do something similar with
MET.
>> Please
>> > go
>> > > > to
>> > > > >>> this
>> > > > >>> > > >> link:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >>
>> > > https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > And then search for "GRIB2_mtab",
and
>> you'll
>> > > find
>> > > > >>> the
>> > > > >>> > > >> content
>> > > > >>> > > >> > >> > > >> > copied/pasted below. You can specify
these
>> > > INSTEAD
>> > > > >>> OF
>> > > > >>> > > >> > specifying
>> > > > >>> > > >> > >> a
>> > > > >>> > > >> > >> > > >> > requested name.
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_mtab" is an
integer
>> > > > >>> specifying the
>> > > > >>> > > >> master
>> > > > >>> > > >> > >> > table
>> > > > >>> > > >> > >> > > >> > number.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ltab" is an
integer
>> > > > >>> specifying the
>> > > > >>> > > >> local
>> > > > >>> > > >> > >> table
>> > > > >>> > > >> > >> > > >> > number.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_disc" is an
integer
>> > > > >>> specifying the
>> > > > >>> > > >> GRIB2
>> > > > >>> > > >> > >> > > >> discipline
>> > > > >>> > > >> > >> > > >> > code.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm_cat" is
an
>> integer
>> > > > >>> specifying
>> > > > >>> > > the
>> > > > >>> > > >> > >> > parameter
>> > > > >>> > > >> > >> > > >> > category
>> > > > >>> > > >> > >> > > >> > //         code.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_parm" is an
integer
>> > > > >>> specifying the
>> > > > >>> > > >> > >> parameter
>> > > > >>> > > >> > >> > > code.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_pdt" is an
integer
>> > > > specifying
>> > > > >>> the
>> > > > >>> > > >> product
>> > > > >>> > > >> > >> > > >> definition
>> > > > >>> > > >> > >> > > >> > //         template (Table 4.0).
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_process" is an
>> integer
>> > > > >>> specifying
>> > > > >>> > > the
>> > > > >>> > > >> > >> > generating
>> > > > >>> > > >> > >> > > >> > process
>> > > > >>> > > >> > >> > > >> > //         (Table 4.3).
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_cntr" is an
integer
>> > > > >>> specifying the
>> > > > >>> > > >> > >> originating
>> > > > >>> > > >> > >> > > >> > center.
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ens_type" is
an
>> integer
>> > > > >>> specifying
>> > > > >>> > > the
>> > > > >>> > > >> > >> > ensemble
>> > > > >>> > > >> > >> > > >> type
>> > > > >>> > > >> > >> > > >> > //         (Table 4.6).
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_der_type" is
an
>> integer
>> > > > >>> specifying
>> > > > >>> > > the
>> > > > >>> > > >> > >> derived
>> > > > >>> > > >> > >> > > >> > product
>> > > > >>> > > >> > >> > > >> > //         type (Table 4.7).
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_stat_type" is
an
>> integer
>> > > > >>> > specifying
>> > > > >>> > > >> the
>> > > > >>> > > >> > >> > > >> statistical
>> > > > >>> > > >> > >> > > >> > //         processing type (Table
4.10).
>> > > > >>> > > >> > >> > > >> > //       - The "GRIB2_ipdtmpl_index"
and
>> > > > >>> > > "GRIB2_ipdtmpl_val"
>> > > > >>> > > >> > >> entries
>> > > > >>> > > >> > >> > > are
>> > > > >>> > > >> > >> > > >> > arrays
>> > > > >>> > > >> > >> > > >> > //         of integers which specify
the
>> > product
>> > > > >>> > > description
>> > > > >>> > > >> > >> > template
>> > > > >>> > > >> > >> > > >> > values to
>> > > > >>> > > >> > >> > > >> > //         be used. The indices are
>> 0-based.
>> > For
>> > > > >>> > example,
>> > > > >>> > > >> use
>> > > > >>> > > >> > the
>> > > > >>> > > >> > >> > > >> > following to
>> > > > >>> > > >> > >> > > >> > //         request a GRIB2 record
whose
>> 9-th
>> > and
>> > > > >>> 27-th
>> > > > >>> > > >> product
>> > > > >>> > > >> > >> > > >> description
>> > > > >>> > > >> > >> > > >> > //         template values are 1 and
2,
>> > > > >>> respectively:
>> > > > >>> > > >> > >> > > >> > //
GRIB2_ipdtmpl_index=[8, 26];
>> > > > >>> > > >> > GRIB2_ipdtmpl_val=[1,
>> > > > >>> > > >> > >> 2];
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > While I don't have any ECMWF data on
hand
>> for
>> > > > >>> testing, I
>> > > > >>> > > do
>> > > > >>> > > >> > have
>> > > > >>> > > >> > >> a
>> > > > >>> > > >> > >> > > file
>> > > > >>> > > >> > >> > > >> > from the Air Force from which wgrib2
>> prints a
>> > > > >>> similar
>> > > > >>> > > >> result:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > 549:687031550:d=2017061000:var
discipline=0
>> > > > >>> center=57
>> > > > >>> > > >> > >> local_table=1
>> > > > >>> > > >> > >> > > >> > parmcat=1 parm=192:surface:18 hour
fcst:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > So I used that for testing. I ran
the
>> > following
>> > > > >>> > > >> plot_data_plane
>> > > > >>> > > >> > >> > > command
>> > > > >>> > > >> > >> > > >> > WITHOUT specifying a requested name:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > plot_data_plane
>> GALWEM_20170610_CY00_FH18.GR2
>> > > > >>> plot.ps
>> > > > >>> > > >> > >> > 'GRIB2_disc=0;
>> > > > >>> > > >> > >> > > >> > GRIB2_cntr=57; GRIB2_ltab=1;
>> GRIB2_parm_cat=1;
>> > > > >>> > > >> GRIB2_parm=192;
>> > > > >>> > > >> > >> > > >> level="Z0";'
>> > > > >>> > > >> > >> > > >> > -v 3
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > And then I looked closely at the log
>> message
>> > to
>> > > > >>> confirm
>> > > > >>> > > >> that it
>> > > > >>> > > >> > >> used
>> > > > >>> > > >> > >> > > >> > record 549, as expected:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > DEBUG 3:
MetGrib2DataFile::data_plane() ->
>> > Found
>> > > > >>> exact
>> > > > >>> > > match
>> > > > >>> > > >> > for
>> > > > >>> > > >> > >> > > >> > 'CTSTM/Z0' in GRIB2 record 549 field
1 of
>> > GRIB2
>> > > > file
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >>
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>>
'/Volumes/d1/projects/MET/MET_unit_test/MET_test_input/model_data/grib2/galwem/GALWEM_20170610_CY00_FH18.GR2'
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > So MET used these GRIB2 numbers I
defined
>> to
>> > do
>> > > a
>> > > > >>> table
>> > > > >>> > > >> lookup
>> > > > >>> > > >> > >> and
>> > > > >>> > > >> > >> > > find
>> > > > >>> > > >> > >> > > >> a
>> > > > >>> > > >> > >> > > >> > match in the MET GRIB tables:
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >>
>> > > > >>> > >
>> > > >
https://github.com/dtcenter/MET/tree/main_v9.1/met/data/table_files
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > And it determined that that field
should be
>> > > called
>> > > > >>> > > "CTSTM".
>> > > > >>> > > >> > >> > Hopefully,
>> > > > >>> > > >> > >> > > >> the
>> > > > >>> > > >> > >> > > >> > ECMWF GRIB2 tables are already
defined
>> there
>> > in
>> > > > >>> MET. If
>> > > > >>> > > >> not, we
>> > > > >>> > > >> > >> can
>> > > > >>> > > >> > >> > > >> always
>> > > > >>> > > >> > >> > > >> > add them.
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > Please try running plot_data_plane
at
>> > verbosity
>> > > > >>> level 3
>> > > > >>> > on
>> > > > >>> > > >> one
>> > > > >>> > > >> > of
>> > > > >>> > > >> > >> > > these
>> > > > >>> > > >> > >> > > >> > files first. And make sure it runs
without
>> > error
>> > > > >>> and the
>> > > > >>> > > log
>> > > > >>> > > >> > >> message
>> > > > >>> > > >> > >> > > >> prints
>> > > > >>> > > >> > >> > > >> > out the "name" it found for that
data. If
>> so,
>> > > you
>> > > > >>> can
>> > > > >>> > just
>> > > > >>> > > >> skip
>> > > > >>> > > >> > >> > using
>> > > > >>> > > >> > >> > > >> all
>> > > > >>> > > >> > >> > > >> > those integers and revert back to
using the
>> > name
>> > > > MET
>> > > > >>> > > knows.
>> > > > >>> > > >> If
>> > > > >>> > > >> > >> not,
>> > > > >>> > > >> > >> > > >> then we
>> > > > >>> > > >> > >> > > >> > may need to define a custom GRIB
table for
>> > this
>> > > > >>> ECMWF
>> > > > >>> > > GRIB2
>> > > > >>> > > >> > data.
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >> > Thanks,
>> > > > >>> > > >> > >> > > >> > John
>> > > > >>> > > >> > >> > > >> >
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> --
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Binbin Zhou
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> 5830 University Research Ct.
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> College Park, MD 20740
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> Binbin.Zhou at noaa.gov
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> 301-683-3683
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >> --
>> > > > >>> > > >> > >> > > >>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > > > >>> > > >> > >> > > >> Tara Jensen
>> > > > >>> > > >> > >> > > >> Project Manager II
>> > > > >>> > > >> > >> > > >> NCAR RAL and DTC
>> > > > >>> > > >> > >> > > >> PO Box 3000, Boulder, Colorado 80307
USA
>> > > > >>> > > >> > >> > > >> +1 303-497-8479
jensen at ucar.edu
>> > > > >>> > > >> > >> > > >>
>> > > > >>> > > >> > >> > > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> > >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > --
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > Binbin Zhou
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > 5830 University Research Ct.
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > College Park, MD 20740
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > Binbin.Zhou at noaa.gov
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> > 301-683-3683
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >> >
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >>
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > --
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > Binbin Zhou
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > 5830 University Research Ct.
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > College Park, MD 20740
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > Binbin.Zhou at noaa.gov
>> > > > >>> > > >> > >
>> > > > >>> > > >> > > 301-683-3683
>> > > > >>> > > >> > >
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >> > --
>> > > > >>> > > >> >
>> > > > >>> > > >> > Binbin Zhou
>> > > > >>> > > >> >
>> > > > >>> > > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > > >> >
>> > > > >>> > > >> > 5830 University Research Ct.
>> > > > >>> > > >> >
>> > > > >>> > > >> > College Park, MD 20740
>> > > > >>> > > >> >
>> > > > >>> > > >> > Binbin.Zhou at noaa.gov
>> > > > >>> > > >> >
>> > > > >>> > > >> > 301-683-3683
>> > > > >>> > > >> >
>> > > > >>> > > >> >
>> > > > >>> > > >>
>> > > > >>> > > >>
>> > > > >>> > > >
>> > > > >>> > > > --
>> > > > >>> > > >
>> > > > >>> > > > Binbin Zhou
>> > > > >>> > > >
>> > > > >>> > > > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > > >
>> > > > >>> > > > 5830 University Research Ct.
>> > > > >>> > > >
>> > > > >>> > > > College Park, MD 20740
>> > > > >>> > > >
>> > > > >>> > > > Binbin.Zhou at noaa.gov
>> > > > >>> > > >
>> > > > >>> > > > 301-683-3683
>> > > > >>> > > >
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > --
>> > > > >>> > >
>> > > > >>> > > Binbin Zhou
>> > > > >>> > >
>> > > > >>> > > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>> > >
>> > > > >>> > > 5830 University Research Ct.
>> > > > >>> > >
>> > > > >>> > > College Park, MD 20740
>> > > > >>> > >
>> > > > >>> > > Binbin.Zhou at noaa.gov
>> > > > >>> > >
>> > > > >>> > > 301-683-3683
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> >
>> > > > >>> >
>> > > > >>>
>> > > > >>> --
>> > > > >>>
>> > > > >>> Binbin Zhou
>> > > > >>>
>> > > > >>> IMSG at NOAA/NWS/NCEP/EMC
>> > > > >>>
>> > > > >>> 5830 University Research Ct.
>> > > > >>>
>> > > > >>> College Park, MD 20740
>> > > > >>>
>> > > > >>> Binbin.Zhou at noaa.gov
>> > > > >>>
>> > > > >>> 301-683-3683
>> > > > >>>
>> > > > >>>
>> > > >
>> > > > --
>> > > > Julie Prestopnik (she/her)
>> > > > Software Engineer
>> > > > National Center for Atmospheric Research
>> > > > Research Applications Laboratory
>> > > > Email: jpresto at ucar.edu
>> > > >
>> > > > My working day may not be your working day.  Please do not
feel
>> obliged
>> > > to
>> > > > reply to this email outside of your normal working hours.
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Julie Prestopnik (she/her)
>> > Software Engineer
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > Email: jpresto at ucar.edu
>> >
>> > My working day may not be your working day.  Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
>
> Binbin Zhou
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct.
>
> College Park, MD 20740
>
> Binbin.Zhou at noaa.gov
>
> 301-683-3683
>


--

Binbin Zhou

IMSG at NOAA/NWS/NCEP/EMC

5830 University Research Ct.

College Park, MD 20740

Binbin.Zhou at noaa.gov

301-683-3683

------------------------------------------------
Subject: urgent question for reading ECMWF ensmeble data
From: John Halley Gotway
Time: Fri Dec 11 10:47:46 2020

Binbin,

Those * characters don't belong in the plot_data_plane command. That's
the
way the mail system formats bold text. To be totally clear, here are
the
exact commands I ran after logging on to Venus to test:

cd /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0_grib1
module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
module load met/ens_config_test
plot_data_plane ecmanl.t12z.grid3.f00.grib1 ~/plot.ps 'name="HGT";
level="P500"; lead_time="00"; GRIB_ens="low_res_ctl";' -v 3

DEBUG 1: Opening data file: ecmanl.t12z.grid3.f00.grib1
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match for
VarInfo "HGT/P500" in GRIB record 3 of GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 1 GRIB records
matching VarInfo "HGT/P500" in GRIB file
"ecmanl.t12z.grid3.f00.grib1".
DEBUG 1: Creating postscript file: /u/John.H.Gotway/plot.ps

That matches a single record, namely the 3rd record in that GRIB file.
Do
you see different behavior? Perhaps you failed to load the
"ens_config_test" version of the code before testing?

John

On Fri, Dec 11, 2020 at 10:20 AM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
>
> John,
>
>   As I tested following plotting
>
> plot_data_plane ecmanl.t12z.grid3.f00.grib1 plot.ps '*name="HGT";
> level="P500"; lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
>
> the error message is
>
> ERROR  :
> ERROR  : yyerror() -> syntax error in file "/tmp/met_config_9350_0"
> ERROR  :
> ERROR  :    line   = 1
> ERROR  :
> ERROR  :    column = 1
> ERROR  :
> ERROR  :    text   = "*"
> ERROR  :
> ERROR  :
> ERROR  : *name="HGT"; level="P500"; lead_time="00";
> GRIB_ens="low_res_ctl";*
> ERROR  :
> ^__________________________________________________________________
> ERROR  :
>
> After I removed "*"
>   plot_data_plane ecmanl.t12z.grid3.f00.grib1 plot.ps 'name="HGT";
> level="P500"; lead_time="00"; GRIB_ens="low_res_ctl";' -v 3
> It shows 3 records matched
>
> DEBUG 1: Opening data file: ecmanl.t12z.grid3.f00.grib1
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match
for
> VarInfo "HGT/P500" in GRIB record 1 of GRIB file
> "ecmanl.t12z.grid3.f00.grib1".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match
for
> VarInfo "HGT/P500" in GRIB record 2 of GRIB file
> "ecmanl.t12z.grid3.f00.grib1".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found exact match
for
> VarInfo "HGT/P500" in GRIB record 3 of GRIB file
> "ecmanl.t12z.grid3.f00.grib1".
> DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found 3 GRIB
records
> matching VarInfo "HGT/P500" in GRIB file
"ecmanl.t12z.grid3.f00.grib1".
> DEBUG 1: Creating postscript file: plot.ps
>
> My testing directory on Venus
> is /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0_grib1
> test script is ploting.sh
>
> Can you look at it?
>
> Thanks!
> Binbin
>
>
> On Thu, Dec 10, 2020 at 3:17 PM Binbin Zhou - NOAA Affiliate <
> binbin.zhou at noaa.gov> wrote:
>
> > John,
> >
> >   Thanks!
> >
> > Binbin
> >
> > On Thu, Dec 10, 2020 at 2:06 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Thanks Julie.
> >>
> >> And Binbin, here's an EnsembleStatConfig file where climos are
used:
> >>
> >>
> >>
>
https://github.com/dtcenter/MET/blob/main_v9.1/test/config/EnsembleStatConfig_climo
> >>
> >> Since the list of forecast fields is exactly the same as the
climo
> fields,
> >> this config file just copies them over using:
> >>    climo_mean = fcst;
> >>
> >> In your case, when they differ, you'd just specify the "field"
array
> >> inside
> >> the "climo_mean" dictionary. For example:
> >>
> >> climo_mean = {
> >>
> >>    field = [
> >>       { name  = "TMP"; level = "Z2";   },
> >>       { name  = "TMP"; level = "P850"; }
> >>    ];
> >>    ...
> >> }
> >>
> >> On Thu, Dec 10, 2020 at 12:00 PM Julie Prestopnik via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> >
> >> > Thanks, John.  That worked great.
> >> >
> >> > Binbin, this version of MET is now accessible to you by running
the
> >> > following on venus:
> >> >
> >> > module use
> /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
> >> > module load met/ens_config_test
> >> >
> >> > Please let us know if you encounter any problems.
> >> >
> >> > Julie
> >> >
> >> > On Thu, Dec 10, 2020 at 10:05 AM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
>
> >> > >
> >> > > Julie,
> >> > >
> >> > > Building the tarfile is very easy. I just ran this script
from the
> MET
> >> > repo
> >> > > and specified the branch to be built:
> >> > >    MET/scripts/met_checkout_and_build.sh branch
> >> feature_1604_ens_config
> >> > >
> >> > > And that created a tarfile which I posted to RAL's anonymous
ftp
> >> server:
> >> > >
> >> > >
> >> > >
> >> >
> >>
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
feature_1604_ens_config.20201210.tar.gz
> >> > >
> >> > > I assume that'll work for you.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Thu, Dec 10, 2020 at 9:25 AM Julie Prestopnik via RT <
> >> > met_help at ucar.edu
> >> > > >
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769 >
> >> > > >
> >> > > > John,
> >> > > >
> >> > > > I am able to help with this install on WCOSS.
> >> > > >
> >> > > > Would it be easier to use git to checkout the repo on
WCOSS? Or
> >> would
> >> > it
> >> > > be
> >> > > > > easier if I built a tarfile for this version of the code?
> >> > > > >
> >> > > > The last time I tried to build from git on WCOSS (a while
ago), I
> >> > > > couldn't.  I don't remember all of the details, but I think
I
> >> couldn't
> >> > > run
> >> > > > bootstrap and maybe it had to do with the version of
autoconf
> being
> >> too
> >> > > > old?  I'm happy to try it again if you'd like though.
Otherwise,
> I
> >> > know
> >> > > a
> >> > > > tarfile would work, but I know that is extra work for you.
> >> > > >
> >> > > > Julie
> >> > > >
> >> > > > On Wed, Dec 9, 2020 at 5:18 PM John Halley Gotway <
> johnhg at ucar.edu>
> >> > > wrote:
> >> > > >
> >> > > > > Julie,
> >> > > > >
> >> > > > > I have created a bugfix branch for main_v9.1 named
> >> > > > > "feature_1604_ens_config" and it's up on GitHub.
> >> > > > >
> >> > > > > Only a single file has changed in it. But I need to get
this
> >> version
> >> > > > > compiled on WCOSS with which Binbin can test.
> >> > > > >
> >> > > > > Are you to help me with this? Would it be easier to use
git to
> >> > checkout
> >> > > > > the repo on WCOSS? Or would it be easier if I built a
tarfile
> for
> >> > this
> >> > > > > version of the code?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Wed, Dec 9, 2020 at 4:52 PM John Halley Gotway <
> >> johnhg at ucar.edu>
> >> > > > wrote:
> >> > > > >
> >> > > > >> Binbin,
> >> > > > >>
> >> > > > >> In order for you to test this change out, we'd need to
> recompile
> >> a
> >> > > > >> version of the code on WCOSS. I wrote this up as a
GitHub
> issue:
> >> > > > >> https://github.com/dtcenter/MET/issues/1604
> >> > > > >>
> >> > > > >> And I'll create a branch from main_v9.1 named
> >> > "bugfix_1604_ens_config"
> >> > > > >> and commit the change there. Then we can checkout that
code on
> >> > WCOSS,
> >> > > > >> compile it, and you can test it.
> >> > > > >>
> >> > > > >> Yes, you're right, if the fcst fields differ from the
climo
> >> fields,
> >> > > > you'd
> >> > > > >> need to specify them separately. That's pretty easy in
the
> >> > > Ensemble-Stat
> >> > > > >> config file... but are you asking how to control that
via
> >> METplus?
> >> > > > >>
> >> > > > >> John
> >> > > > >>
> >> > > > >> On Wed, Dec 9, 2020 at 2:29 PM Binbin.Zhou at noaa.gov via
RT <
> >> > > > >> met_help at ucar.edu> wrote:
> >> > > > >>
> >> > > > >>>
> >> > > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >
> >> > > > >>>
> >> > > > >>> John,
> >> > > > >>>
> >> > > > >>>   Great! Let me test it at first before make change in
MET9.1
> >> > > > >>> Another question:
> >> > > > >>>    Since climatology data is still in NCEP's grib2
format, In
> >> the
> >> > > > >>> ensemble_stat
> >> > > > >>> config file,  climo  entry settings  should be
different from
> >> those
> >> > > of
> >> > > > >>> fcst, obs entries?
> >> > > > >>> How to set climo (name, level, etc.)?
> >> > > > >>>
> >> > > > >>> Binbin
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Wed, Dec 9, 2020 at 4:14 PM John Halley Gotway via
RT <
> >> > > > >>> met_help at ucar.edu>
> >> > > > >>> wrote:
> >> > > > >>>
> >> > > > >>> > Binbin,
> >> > > > >>> >
> >> > > > >>> > Success! When I recompile with that one line
commented out,
> >> > > > >>> plot_data_plane
> >> > > > >>> > is able to filter down all those records to the
single one
> >> that
> >> > we
> >> > > > >>> want.
> >> > > > >>> > Note in the example below that I used the "lead_time"
and
> >> > > "GRIB_ens"
> >> > > > >>> > options to filter the input records down.
> >> > > > >>> >
> >> > > > >>> > plot_data_plane ensposte.t00z.z500hr plot.ps
'*name="HGT";
> >> > > > >>> level="P500";
> >> > > > >>> > lead_time="00"; GRIB_ens="low_res_ctl";*' -v 3
> >> > > > >>> > DEBUG 1: Opening data file: ensposte.t00z.z500hr
> >> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() ->
*Found
> exact
> >> > match
> >> > > > for
> >> > > > >>> > VarInfo "HGT/P500" in GRIB record 26 of GRIB file
> >> > > > >>> "ensposte.t00z.z500hr".*
> >> > > > >>> > DEBUG 3: MetGrib1DataFile::data_plane_array() ->
Found 1
> GRIB
> >> > > records
> >> > > > >>> > matching VarInfo "HGT/P500" in GRIB file
> >> "ensposte.t00z.z500hr".
> >> > > > >>> > DEBUG 1: Creating postscript file: plot.ps
> >> > > > >>> >
> >> > > > >>> > Also note that this selects out record number 26.
> >> > > > >>> >
> >> > > > >>> > Shall I make this change as a "bugfix" for met-
9.1...which
> >> could
> >> > be
> >> > > > >>> > included in a met-9.1.2 bugfix release? Or shall I
just
> >> include
> >> > it
> >> > > in
> >> > > > >>> > development for the next release (planned for March
2021)?
> >> > > > >>> >
> >> > > > >>> > Thanks,
> >> > > > >>> > John
> >> > > > >>> >
> >> > > > >>> > On Tue, Dec 8, 2020 at 5:27 PM Binbin.Zhou at noaa.gov
via RT
> <
> >> > > > >>> > met_help at ucar.edu> wrote:
> >> > > > >>> >
> >> > > > >>> > >
> >> > > > >>> > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >> > >
> >> > > > >>> > >
> >> > > > >>> > > John,
> >> > > > >>> > >
> >> > > > >>> > >   If this removing impacts other applications, or
takes
> some
> >> > time
> >> > > > >>> > > to be effective, then won't try.  I will use
another
> >> approach,
> >> > > such
> >> > > > >>> as
> >> > > > >>> > > borrow GEFS's grib2 data structure as tank to save
EC
> >> ensemble
> >> > > > >>> > > data.
> >> > > > >>> > >
> >> > > > >>> > > Binbin
> >> > > > >>> > >
> >> > > > >>> > >
> >> > > > >>> > > On Tue, Dec 8, 2020 at 6:46 PM Binbin Zhou - NOAA
> Affiliate
> >> <
> >> > > > >>> > > binbin.zhou at noaa.gov> wrote:
> >> > > > >>> > >
> >> > > > >>> > > > John,
> >> > > > >>> > > >
> >> > > > >>> > > >   Yes, please try to use this approach.
> >> > > > >>> > > >
> >> > > > >>> > > > Thanks!
> >> > > > >>> > > >
> >> > > > >>> > > > Binbin
> >> > > > >>> > > >
> >> > > > >>> > > > Binbin
> >> > > > >>> > > >
> >> > > > >>> > > > On Tue, Dec 8, 2020 at 5:43 PM John Halley Gotway
via
> RT <
> >> > > > >>> > > > met_help at ucar.edu> wrote:
> >> > > > >>> > > >
> >> > > > >>> > > >> Binbin,
> >> > > > >>> > > >>
> >> > > > >>> > > >> I used the "-PDS10" option for wgrib to dump
those
> >> records:
> >> > > > >>> > > >>
> >> > > > >>> > > >> [John.H.Gotway at v71a1 ~]$ $WGRIB -PDS10
> >> > > > >>> > > >>
> >> > > > >>> > > >>
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0/ensposte.t00z.z500hr
> >> > > > >>> > > >> | egrep  "^11:|^26:|^43:"
> >> > > > >>> > > >> 11:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
> >> 0 1
> >> > 0
> >> > > 0
> >> > > > 1
> >> > > > >>> 0 0
> >> > > > >>> > 0
> >> > > > >>> > > 21
> >> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 2 1 255
> >> > > > >>> > > >> 26:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
> >> 0 1
> >> > 0
> >> > > 0
> >> > > > 1
> >> > > > >>> 0 0
> >> > > > >>> > 0
> >> > > > >>> > > 21
> >> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 1 255
> >> > > > >>> > > >> 43:PDS10= 0 0 45 2 98 151 255 128 7 100 1 244 20
12 6 0
> >> 0 1
> >> > 0
> >> > > 0
> >> > > > 1
> >> > > > >>> 0 0
> >> > > > >>> > 0
> >> > > > >>> > > 21
> >> > > > >>> > > >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 255
> >> > > > >>> > > >>
> >> > > > >>> > > >> Looking VERY closely, there is a difference in
the
> 42-nd
> >> > > column
> >> > > > >>> which
> >> > > > >>> > is
> >> > > > >>> > > >> the 4-th from the last... records 11, 26, and 43
have
> >> values
> >> > > of
> >> > > > >>> 3, 1,
> >> > > > >>> > > and
> >> > > > >>> > > >> 2
> >> > > > >>> > > >> respectively.
> >> > > > >>> > > >>
> >> > > > >>> > > >> On this page (
> >> > > > >>> > >
> https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html
> >> > > > >>> > > >> ),
> >> > > > >>> > > >> byte 42 = 1 means "unperturbed control
forecast".
> >> > > > >>> > > >>
> >> > > > >>> > > >> So we *should* be able to use the "GRIB_ens"
config
> >> option
> >> > to
> >> > > > >>> > > >> differentiate
> >> > > > >>> > > >> between those records. BUT when I try, it
doesn't work.
> >> And
> >> > > > that's
> >> > > > >>> > > because
> >> > > > >>> > > >> of this line:
> >> > > > >>> > > >>
> >> > > > >>> > > >>
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_data2d_grib/data2d_grib_utils.cc#L89
> >> > > > >>> > > >>
> >> > > > >>> > > >> We only apply those filtering options to GRIB
files
> >> created
> >> > by
> >> > > > >>> NCEP.
> >> > > > >>> > If
> >> > > > >>> > > >> you'd like, I could try removing that condition
to see
> >> if we
> >> > > can
> >> > > > >>> pick
> >> > > > >>> > > out
> >> > > > >>> > > >> the single record of interest.
> >> > > > >>> > > >>
> >> > > > >>> > > >> John
> >> > > > >>> > > >>
> >> > > > >>> > > >> On Tue, Dec 8, 2020 at 2:51 PM
Binbin.Zhou at noaa.gov
> via
> >> RT
> >> > <
> >> > > > >>> > > >> met_help at ucar.edu> wrote:
> >> > > > >>> > > >>
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >> > > > >
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > John,
> >> > > > >>> > > >> >
> >> > > > >>> > > >> >   Look at my
> >> > > > >>> > > >> > directory
> >> > > > >>>
/gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/ecme1p0
> >> > > > >>> > > >> > The grib1 file is ensposte.t00z.z500hr, this
is only
> >> for
> >> > 500
> >> > > > >>> HGT.
> >> > > > >>> > > >> > wgrib ensposte.t00z.z500hr|grep anl|grep
> >> anl:type=analysis
> >> > > > >>> > > >> > I get 3 records:
> >> > > > >>> > > >> >
> >> > > > >>> > > >> >
> >> > > > >>> > > >> >
> >> > > > >>> > > >>
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
>
11:896950:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> >> > > > >>> > > >> >
> >> > > > >>> > > >> >
> >> > > > >>> > > >>
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
>
26:2242375:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> >> > > > >>> > > >> >
> >> > > > >>> > > >> >
> >> > > > >>> > > >>
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
>
43:3767190:d=20120600:HGT:kpds5=7:kpds6=100:kpds7=500:TR=1:P1=0:P2=0:TimeU=1:500
> >> > > > >>> > > >> > mb:anl:type=analysis:NAve=0
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > From its grib2 file, we know only  record#26
is
> >> analysis
> >> > > > low-res
> >> > > > >>> > > >> control.
> >> > > > >>> > > >> > If you have some way to determine which record
is
> >> analysis
> >> > > > >>> low-res
> >> > > > >>> > > >> control
> >> > > > >>> > > >> > from the grib1 file,
> >> > > > >>> > > >> > then we can use grib1 directly (don't consider
the
> size
> >> > > > issue).
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > Binbin
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > On Tue, Dec 8, 2020 at 4:36 PM Binbin Zhou -
NOAA
> >> > Affiliate
> >> > > <
> >> > > > >>> > > >> > binbin.zhou at noaa.gov> wrote:
> >> > > > >>> > > >> >
> >> > > > >>> > > >> > > John,
> >> > > > >>> > > >> > >
> >> > > > >>> > > >> > >   Grib1 data is too large, another issue is
the
> >> low-res
> >> > > > >>> control
> >> > > > >>> > > >> analysis
> >> > > > >>> > > >> > > data can not be retrieved
> >> > > > >>> > > >> > > from the grib1 ecmwf data files. If using
wgrib, we
> >> can
> >> > > see
> >> > > > 3
> >> > > > >>> > > analysis
> >> > > > >>> > > >> > > fields, but don't know
> >> > > > >>> > > >> > > which is low-res control. After convert to
grib2,
> >> wgrib2
> >> > > can
> >> > > > >>> show
> >> > > > >>> > > >> which
> >> > > > >>> > > >> > > record is low-res control.
> >> > > > >>> > > >> > > Tomorrow I'll show you the ecmwf grib1 data
set on
> >> > WCOSS.
> >> > > > >>> > > >> > >
> >> > > > >>> > > >> > > Binbin
> >> > > > >>> > > >> > >
> >> > > > >>> > > >> > > On Tue, Dec 8, 2020 at 3:57 PM John Halley
Gotway
> via
> >> > RT <
> >> > > > >>> > > >> > > met_help at ucar.edu> wrote:
> >> > > > >>> > > >> > >
> >> > > > >>> > > >> > >> Binbin,
> >> > > > >>> > > >> > >>
> >> > > > >>> > > >> > >> Sounds like there's a lot of moving parts
there.
> How
> >> > > would
> >> > > > >>> you
> >> > > > >>> > like
> >> > > > >>> > > >> to
> >> > > > >>> > > >> > >> proceed?
> >> > > > >>> > > >> > >>
> >> > > > >>> > > >> > >> Should we check to see if MET can read
those ECMWF
> >> > GRIB1
> >> > > > >>> files
> >> > > > >>> > > >> directly
> >> > > > >>> > > >> > >> without needing to reformat?
> >> > > > >>> > > >> > >> Or do you just want to create a custom GRIB
table
> >> for
> >> > the
> >> > > > >>> result
> >> > > > >>> > of
> >> > > > >>> > > >> your
> >> > > > >>> > > >> > >> pre-processing and conversion steps?
> >> > > > >>> > > >> > >>
> >> > > > >>> > > >> > >> John
> >> > > > >>> > > >> > >>
> >> > > > >>> > > >> > >> On Mon, Dec 7, 2020 at 2:49 PM
> Binbin.Zhou at noaa.gov
> >> > via
> >> > > > RT <
> >> > > > >>> > > >> > >> met_help at ucar.edu> wrote:
> >> > > > >>> > > >> > >>
> >> > > > >>> > > >> > >> >
> >> > > > >>> > > >> > >> > <URL:
> >> > > > >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97769
> >> > > > >>> > >
> >> > > > >>> > > >> > >> >
> >> > > > >>> > > >> > >> > John,
> >> > > > >>> > > >> > >> >
> >> > > > >>> > > >> > >> >   Our ECMWF grib2 ensemble data files are
> >> converted
> >> > > from
> >> > > > >>> its
> >> > > > >>> > > grib1
> >> > > > >>> > > >> > >> format
> >> > > > >>> > > >> > >> > data.
><