[ncl-talk] grib2 file read

Debasish Hazra debasish.hazra5 at gmail.com
Wed Jul 27 13:50:51 MDT 2016


Thanks Dave and David. Lets see how more information can be retrieved.

Debasish

On Wed, Jul 27, 2016 at 1:58 PM, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> This file is fortran unformatted binary sequential, big endian, with
> varying record lengths.  There are mixed data types, at least character and
> integer, probably also some floats, if I am not mistaken.  The first record
> begins with the string "NEMSIO  GFS     bin4" which is likely a format
> identifier.
>
> This is a custom made fortran file.  This will be impossible to decode
> without some source documentation.  NCL may have problems with the varying
> length records and mixed data types.  This file was made for fortran
> access, not for NCL or other common climate software.
>
> --Dave
>
>
> On Tue, Jul 26, 2016 at 7:40 PM, David Brown <dbrown at ucar.edu> wrote:
>
>> I don't know how this file is organized. I do know that the 'strings'
>> tool outputs some stuff that looks like variable names at the
>> beginning:
>>
>> strings ngac.t00z.aerf24 |more
>>
>> 0NEMSIO  GFS     bin4
>>
>> bccmass         bcdp001         bcdp002         bcem001
>> bcem002         bceman          bcembb          bcembf
>> bchyphil        bcsd001         bcsd002         bcsmass
>> bcsv001         bcsv002         bcwt001         bcwt002
>> dmscmass        dmssmass        ducmass25       ducmass
>> dudp001         dudp002         dudp003         dudp004
>> dudp005         duem001         duem002         duem003
>> duem004         duem005         dusd001         dusd002
>> dusd003         dusd004         dusd005         dusmass25
>> dusmass         dusv001         dusv002         dusv003
>> dusv004         dusv005         duwt001         duwt002
>> duwt003         duwt004         duwt005         occmass
>> ocdp001         ocdp002         ocem001         ocem002         oceman
>>          ocembb          ocembf          ocembg          ochyphil
>>   ocsd001         ocsd002         ocsmass         ocsv001
>> ocsv002         ocwt001         ocwt002         so2cmass
>> so2eman         so2embb         so2emve         so2emvn
>> so2smass        so4cmass        so4eman         so4smass
>> sscmass25       sscmass         ssdp001         ssdp002
>> ssdp003         ssdp004         ssdp005         ssem001
>> ssem002         ssem003         ssem004         ssem005
>> sssd001         sssd002         sssd003         sssd004
>> sssd005         sssmass25       sssmass         sssv001
>> sssv002         sssv003         sssv004         sssv005
>> sswt001         sswt002         sswt003         sswt004
>> sswt005         sudp001         sudp002         sudp003
>> sudp004         suem001         suem002         suem003
>> suem004         supmsa          supso2          supso4aq
>> supso4g         supso4wt        susd001         susd002
>> susd003         susd004         susv001         susv002
>> susv003         susv004         suwt001         suwt002
>> suwt003         suwt004
>>
>> atmos col       atmos col       atmos col       atmos col       atmos
>> col       atmos col       atmos col       atmos col       atmos col
>>    atmos col       atmos col       atmos col       atmos col
>> atmos col       atmos col       atmos col       atmos col       atmos
>> col       atmos col       atmos col
>>
>> .....
>>
>> The "bin4" at the beginning suggests maybe that it is 4 byte floats,
>> but other than than I have no idea.
>>
>> I think you will need to find some explanation of the contents of this
>> file from the people who created it.
>>  -dave
>>
>>
>> On Tue, Jul 26, 2016 at 7:16 PM, Debasish Hazra
>> <debasish.hazra5 at gmail.com> wrote:
>> > Thanks David. Do I need to read the file as "binary" ?
>> >
>> > Debasish
>> >
>> > On Tue, Jul 26, 2016 at 6:12 PM, David Brown <dbrown at ucar.edu> wrote:
>> >>
>> >> I am afraid there have been some misunderstandings.
>> >> First of all the file "ngac.t00z.aerf24.grib2" (actually called
>> >> ngac.t00z.aerf24 on the ftp site if anyone else wants to download it)
>> >> is
>> >> not a GRIB file at all. A simple test if you suspect something is not
>> >> GRIB:
>> >>
>> >> strings <filename> | grep GRIB
>> >>
>> >> If this produces nothing then it's definitely not a GRIB file (of
>> >> course the presence of "GRIB" does not prove it is a GRIB file).
>> >> To digress for a moment, if it is a GRIB file then the command,
>> >>
>> >> strings <filename> | egrep 'GRIB|7777'
>> >>
>> >> will produce alternating instances of 'GRIB' and '7777'. These strings
>> >> mark the beginning and end of each GRIB record. This is true for both
>> >> GRIB 1 and 2.
>> >>
>> >> strings ngac.t00z.aerf24 |grep GRIB
>> >> produces nothing so it is not GRIB.
>> >>
>> >> Second, the warning messages that are emitted when an ngac file that
>> >> is valid GRIB is read by NCL such as:
>> >>
>> >> warning: Entry (62006) not found in code table file
>> >> /Users/shea/devel/lib/ncarg/grib2_codetables/ncep/4/4.3.table
>> >>
>> >> refer to entries not found in table 4.3 which is "type of generating
>> >> process". This is not the same as table 4.233 "Aerosol Type"
>> >> and so the number 62006 in this context has nothing to do with 62006
>> >> ("Sulphate Dry") in the Aerosol Type table
>> >>
>> >> 62006 as a type of generating processing is not documented in the 4.3
>> >> table and hence the message. Since type of generating process is
>> >> usually of less concern to users I am considering eliminating the
>> >> error message in this case.
>> >>
>> >> It is true that NCL does not yet make use of the 4.233 table. That is
>> >> because it is an auxiliary table that needs to be used in the context
>> >> of certain GRIB2 product templates that address aerosols. It does not
>> >> contain actual parameters. Rather a parameter might be named, e.g.
>> >> "Aerosol optical thickness" and based on the template there is another
>> >> piece of metadata called "Aerosol Type" which is as documented in the
>> >> "Aerosol Types" table. A small extension to our GRIB2 data model is
>> >> required to handle these templates. We hope to implement it in time
>> >> for 6.4.0 but that is not yet certain.
>> >>  -dave
>> >>
>> >>
>> >> On Tue, Jul 26, 2016 at 12:23 PM, Debasish Hazra
>> >> <debasish.hazra5 at gmail.com> wrote:
>> >> > Hi Dennis,
>> >> >
>> >> > The variables in the range  62000 to 62010 in the updated grib2 table
>> >> > "http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml"
>> are
>> >> > "optical depth" and has no units associated with them.
>> >> >
>> >> > 62000 Total Aerosol
>> >> > 62001 Dust Dry
>> >> > 62006 Sulphate Dry
>> >> > 62008 Sea Salt Dry
>> >> > 62009 Black Carbon Dry
>> >> > 62010 Particulate Organic Matter Dry
>> >> >
>> >> > 62010-62016 are mixing ratios in kg/kg
>> >> >
>> >> > Debasish
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Jul 26, 2016 at 2:13 PM, Dennis Shea <shea at ucar.edu> wrote:
>> >> >>
>> >> >> Again, I will forward this. please send to ncl-talk. Otherwise, I am
>> >> >> just
>> >> >> a middleman forwarding the emails.
>> >> >>
>> >> >> Usually the GRIB tables have units:
>> >> >>
>> >> >> Examples:
>> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_141.htm
>> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ecmwf_132.htm
>> >> >>
>> >> >> NCL ... see GRIB section
>> >> >> See:
>> >> >>
>> >> >>
>> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml
>> >> >>
>> >> >> On Tue, Jul 26, 2016 at 12:05 PM, Debasish Hazra
>> >> >> <debasish.hazra5 at gmail.com> wrote:
>> >> >>>
>> >> >>> The link updated grib2 table is here :
>> >> >>>
>> >> >>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
>> >> >>>
>> >> >>> Thanks
>> >> >>> Debasish
>> >> >>>
>> >> >>> On Tue, Jul 26, 2016 at 1:33 PM, Dennis Shea <shea at ucar.edu>
>> wrote:
>> >> >>>>
>> >> >>>> The issue is that the NGAC grib tables are not publicly available.
>> >> >>>> If you know of where they are located, they can be included within
>> >> >>>> NCL
>> >> >>>>
>> >> >>>> In fact, there is a JIRA ticket on this:
>> >> >>>>
>> >> >>>> [JIRA] (NCL-2442) GRIB-2: warning: Entry (not found in code table
>> >> >>>> file
>> >> >>>> .../grib2_codetables/ncep/4/4.3.table
>> >> >>>>
>> >> >>>> Just yesterday, a comment was made on this ticket:
>> >> >>>>
>> >> >>>> ---------
>> >> >>>>
>> >> >>>> Unfortunately the "not found in code table file" errors
>> >> >>>> that Dennis reported in this ticket cannot be corrected
>> >> >>>> because the entries for that were not found are not documented
>> >> >>>> by the NCEP GRIB2 web pages. Table 4.3 contains the generating
>> >> >>>> process
>> >> >>>> mapping. The values found in the file, all in the range 62000 to
>> >> >>>> 62016,
>> >> >>>> probably refer to local "generating processes" related to the NGAC
>> >> >>>> model.
>> >> >>>> If documentation can be found to describe these I would be happy
>> to
>> >> >>>> add
>> >> >>>> them to the tables.
>> >> >>>>
>> >> >>>> The other option to get rid of these admittedly annoying
>> >> >>>> messages might be to just not print the warnings for this piece of
>> >> >>>> metadata
>> >> >>>> since it is not usually of that much interest to the data
>> consumer. I
>> >> >>>> would
>> >> >>>> be happy to consider this.
>> >> >>>>
>> >> >>>> ---------
>> >> >>>> Note: the *values* returned by NCL are correct. The meta data is
>> not
>> >> >>>> because the
>> >> >>>> NCEP GRIB2 web pages are not available.
>> >> >>>>
>> >> >>>> --------
>> >> >>>>
>> >> >>>> Again, if you know where the appropriate tables are located,
>> please
>> >> >>>> send
>> >> >>>> to ncl-talk.
>> >> >>>>
>> >> >>>> THX
>> >> >>>>
>> >> >>>> On Tue, Jul 26, 2016 at 11:21 AM, Debasish Hazra
>> >> >>>> <debasish.hazra5 at gmail.com> wrote:
>> >> >>>>>
>> >> >>>>> Hi,
>> >> >>>>>
>> >> >>>>> I am trying to read grib2 file with ncl (v6.3) and it shows the
>> >> >>>>> following error while trying to open it:
>> >> >>>>>
>> >> >>>>> fatal:An internal error has occurred. The file format requested
>> does
>> >> >>>>> not appear to be supported, could not open
>> (ngac.t00z.aerf24.grib2)
>> >> >>>>> Variable: f
>> >> >>>>> Type: file
>> >> >>>>> (0) File Missing Value : -1
>> >> >>>>>
>> >> >>>>> I kept the data file in ftp. Any help on this is appreciated.
>> >> >>>>>
>> >> >>>>> Thanks.
>> >> >>>>> Debasish
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160727/1b612ba7/attachment.html 


More information about the ncl-talk mailing list