[Met_help] [rt.rap.ucar.edu #86714] History for Trouble Reading a Model Data File

Thu Aug 23 11:26:23 MDT 2018

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

We have been tasked to verify half-degree resolution ensemble data.  One of the three models, NAVGEM, appears to be in a GriB2 format MET can't read.  I get a "Trouble Reading..." error.

I'll be sending the file in question via AMRDEC.  This is the exact file as it comes into the building here; I haven't modified it in any way.

I found a workaround: extract the individual GriB2 records and read in smaller files with just those records.

Can you take a look and see if there's a way I can read this file in MET?

Thanks,
Matt

Matthew C. Sittel, Meteorologist
University Corporation for Atmospheric Research
Matthew.Sittel.ctr at us.af.mil

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Trouble Reading a Model Data File
From: John Halley Gotway
Time: Wed Aug 22 15:44:00 2018

Matt,

Thanks for sending the sample data file.  That helped a lot in
debugging.
I pulled it down and tried to use met-6.0 to plot the first record,
which
appears to be CAPE:

/usr/local/met-6.0/bin/plot_data_plane
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000 cape.ps
'name="CAPE";
level="L0";'
DEBUG 1: Opening data file:
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000
ERROR  :
ERROR  : plot_data_plane -> file
"ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000" not a valid data
file
ERROR  :

I got a similar (but not identical) error message.  So then I tried
adding
"file_type=GRIB2;" and that did the trick:

/usr/local/met-6.0/bin/plot_data_plane
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000 cape.ps
'name="CAPE";
level="L0"; file_type=GRIB2;'
DEBUG 1: Opening data file:
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000
DEBUG 1: Creating postscript file: cape.ps

Here's how the file type checking logic in MET works...
(1) If the "file_type" is specified by the user, process the file with
the
specified library code.
(2) If not, check the file extension to figure out how to interpret
it.
For example, ".grb2" or ".grib2" suffixes are processed as GRIB2
files.
(3) If no known suffix was used, check for the "magic cookie".  For
example, the first few bytes of a GRIB file include the letters
"GRIB"...
and then MET knows how to process it.

Obviously the logic in (1) and (2) don't apply to your initial run,
but why
didn't the logic in (3) work?  I ran "more" on that file and saw the
following:

*HTTP/1.1 200 OKServer: CagipsCacheMgr/2.2Transfer-Encoding:
chunkedContent-Type:
application/binary;FileName="Bundle.1534745290";CustomerId="0";loginName="Rule.server at fnmoc.navy.mil
<Rule.server at fnmoc.navy.mil>";serverGMT="1534745290";subscriptionId="NULL";subscriptionName="NULL";productCount="NULL";mapText="NULL";aoiName="NULL";bounding_box_left="NULL";bounding_box_top="NULL";bounding_box_right="NULL";bounding_box_bottom="NULL";1DAEFGRIB--
More--(0%)*

For some reason, this file includes a header with "HTTP..." info in
it.
That breaks the magic cookie checking in step (3).

Next, I used wgrib2 to subset this file, select out only the first
record,
and write it to a file named "rec1":
   wgrib2 -d 1 -grib_out rec1
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000

Running "more" on that file reveals:
*GRIB*

So the magic cookie checking of (3) should now work on this subset
file.

Long story short, your options are:
(1) Use the "file_type=GRIB2;" option to explicitly tell MET how to
interpret these files.
(2) Or rename the files by adding a ".grb2" or ".grib2" suffix and MET
will
know what to do.
(3) Or do something else to strip off that HTTP header that's showing
up.
The first 4 bytes of each GRIB record should be "GRIB".

Thanks,
John

On Wed, Aug 22, 2018 at 9:15 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> Wed Aug 22 09:15:54 2018: Request 86714 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>        Queue: met_help
>      Subject: Trouble Reading a Model Data File
>        Owner: Nobody
>   Requestors: matthew.sittel.ctr at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86714 >
>
>
> Hi,
>
> We have been tasked to verify half-degree resolution ensemble data.
One
> of the three models, NAVGEM, appears to be in a GriB2 format MET
can't
> read.  I get a "Trouble Reading..." error.
>
> I'll be sending the file in question via AMRDEC.  This is the exact
file
> as it comes into the building here; I haven't modified it in any
way.
>
> I found a workaround: extract the individual GriB2 records and read
in
> smaller files with just those records.
>
> Can you take a look and see if there's a way I can read this file in
MET?
>
> Thanks,
> Matt
>
> Matthew C. Sittel, Meteorologist
> University Corporation for Atmospheric Research
> Matthew.Sittel.ctr at us.af.mil
>
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86714] Trouble Reading a Model Data File
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Aug 23 11:23:50 2018

Thanks John!  That worked.  I appreciate your help!

Matt

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, August 22, 2018 4:44 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86714] Trouble Reading
a Model Data File

Matt,

Thanks for sending the sample data file.  That helped a lot in
debugging.
I pulled it down and tried to use met-6.0 to plot the first record,
which appears to be CAPE:

/usr/local/met-6.0/bin/plot_data_plane
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000 cape.ps
'name="CAPE"; level="L0";'
DEBUG 1: Opening data file:
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000
ERROR  :
ERROR  : plot_data_plane -> file
"ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000" not a valid data
file ERROR  :

I got a similar (but not identical) error message.  So then I tried
adding "file_type=GRIB2;" and that did the trick:

/usr/local/met-6.0/bin/plot_data_plane
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000 cape.ps
'name="CAPE"; level="L0"; file_type=GRIB2;'
DEBUG 1: Opening data file:
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000
DEBUG 1: Creating postscript file: cape.ps

Here's how the file type checking logic in MET works...
(1) If the "file_type" is specified by the user, process the file with
the specified library code.
(2) If not, check the file extension to figure out how to interpret
it.
For example, ".grb2" or ".grib2" suffixes are processed as GRIB2
files.
(3) If no known suffix was used, check for the "magic cookie".  For
example, the first few bytes of a GRIB file include the letters
"GRIB"...
and then MET knows how to process it.

Obviously the logic in (1) and (2) don't apply to your initial run,
but why didn't the logic in (3) work?  I ran "more" on that file and
saw the
following:

*HTTP/1.1 200 OKServer: CagipsCacheMgr/2.2Transfer-Encoding:
chunkedContent-Type:
application/binary;FileName="Bundle.1534745290";CustomerId="0";loginName="Rule.server at fnmoc.navy.mil
<Rule.server at fnmoc.navy.mil>";serverGMT="1534745290";subscriptionId="NULL";subscriptionName="NULL";productCount="NULL";mapText="NULL";aoiName="NULL";bounding_box_left="NULL";bounding_box_top="NULL";bounding_box_right="NULL";bounding_box_bottom="NULL";1DAEFGRIB--
More--(0%)*

For some reason, this file includes a header with "HTTP..." info in
it.
That breaks the magic cookie checking in step (3).

Next, I used wgrib2 to subset this file, select out only the first
record, and write it to a file named "rec1":
   wgrib2 -d 1 -grib_out rec1
ENSEMBLE.halfDegree.MET.fcst_et010.036.2018082000

Running "more" on that file reveals:
*GRIB*

So the magic cookie checking of (3) should now work on this subset
file.

Long story short, your options are:
(1) Use the "file_type=GRIB2;" option to explicitly tell MET how to
interpret these files.
(2) Or rename the files by adding a ".grb2" or ".grib2" suffix and MET
will know what to do.
(3) Or do something else to strip off that HTTP header that's showing
up.
The first 4 bytes of each GRIB record should be "GRIB".

Thanks,
John

On Wed, Aug 22, 2018 at 9:15 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> Wed Aug 22 09:15:54 2018: Request 86714 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>        Queue: met_help
>      Subject: Trouble Reading a Model Data File
>        Owner: Nobody
>   Requestors: matthew.sittel.ctr at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86714
> >
>
>
> Hi,
>
> We have been tasked to verify half-degree resolution ensemble data.
> One of the three models, NAVGEM, appears to be in a GriB2 format MET
> can't read.  I get a "Trouble Reading..." error.
>
> I'll be sending the file in question via AMRDEC.  This is the exact
> file as it comes into the building here; I haven't modified it in
any way.
>
> I found a workaround: extract the individual GriB2 records and read
in
> smaller files with just those records.
>
> Can you take a look and see if there's a way I can read this file in
MET?
>
> Thanks,
> Matt
>
> Matthew C. Sittel, Meteorologist
> University Corporation for Atmospheric Research
> Matthew.Sittel.ctr at us.af.mil
>
>
>
>
>

------------------------------------------------