[ncl-talk] grib2 file read
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Wed Jul 27 11:58:23 MDT 2016
This file is fortran unformatted binary sequential, big endian, with
varying record lengths. There are mixed data types, at least character and
integer, probably also some floats, if I am not mistaken. The first record
begins with the string "NEMSIO GFS bin4" which is likely a format
identifier.
This is a custom made fortran file. This will be impossible to decode
without some source documentation. NCL may have problems with the varying
length records and mixed data types. This file was made for fortran
access, not for NCL or other common climate software.
--Dave
On Tue, Jul 26, 2016 at 7:40 PM, David Brown <dbrown at ucar.edu> wrote:
> I don't know how this file is organized. I do know that the 'strings'
> tool outputs some stuff that looks like variable names at the
> beginning:
>
> strings ngac.t00z.aerf24 |more
>
> 0NEMSIO GFS bin4
>
> bccmass bcdp001 bcdp002 bcem001
> bcem002 bceman bcembb bcembf
> bchyphil bcsd001 bcsd002 bcsmass
> bcsv001 bcsv002 bcwt001 bcwt002
> dmscmass dmssmass ducmass25 ducmass
> dudp001 dudp002 dudp003 dudp004
> dudp005 duem001 duem002 duem003
> duem004 duem005 dusd001 dusd002
> dusd003 dusd004 dusd005 dusmass25
> dusmass dusv001 dusv002 dusv003
> dusv004 dusv005 duwt001 duwt002
> duwt003 duwt004 duwt005 occmass
> ocdp001 ocdp002 ocem001 ocem002 oceman
> ocembb ocembf ocembg ochyphil
> ocsd001 ocsd002 ocsmass ocsv001
> ocsv002 ocwt001 ocwt002 so2cmass
> so2eman so2embb so2emve so2emvn
> so2smass so4cmass so4eman so4smass
> sscmass25 sscmass ssdp001 ssdp002
> ssdp003 ssdp004 ssdp005 ssem001
> ssem002 ssem003 ssem004 ssem005
> sssd001 sssd002 sssd003 sssd004
> sssd005 sssmass25 sssmass sssv001
> sssv002 sssv003 sssv004 sssv005
> sswt001 sswt002 sswt003 sswt004
> sswt005 sudp001 sudp002 sudp003
> sudp004 suem001 suem002 suem003
> suem004 supmsa supso2 supso4aq
> supso4g supso4wt susd001 susd002
> susd003 susd004 susv001 susv002
> susv003 susv004 suwt001 suwt002
> suwt003 suwt004
>
> atmos col atmos col atmos col atmos col atmos
> col atmos col atmos col atmos col atmos col
> atmos col atmos col atmos col atmos col
> atmos col atmos col atmos col atmos col atmos
> col atmos col atmos col
>
> .....
>
> The "bin4" at the beginning suggests maybe that it is 4 byte floats,
> but other than than I have no idea.
>
> I think you will need to find some explanation of the contents of this
> file from the people who created it.
> -dave
>
>
> On Tue, Jul 26, 2016 at 7:16 PM, Debasish Hazra
> <debasish.hazra5 at gmail.com> wrote:
> > Thanks David. Do I need to read the file as "binary" ?
> >
> > Debasish
> >
> > On Tue, Jul 26, 2016 at 6:12 PM, David Brown <dbrown at ucar.edu> wrote:
> >>
> >> I am afraid there have been some misunderstandings.
> >> First of all the file "ngac.t00z.aerf24.grib2" (actually called
> >> ngac.t00z.aerf24 on the ftp site if anyone else wants to download it)
> >> is
> >> not a GRIB file at all. A simple test if you suspect something is not
> >> GRIB:
> >>
> >> strings <filename> | grep GRIB
> >>
> >> If this produces nothing then it's definitely not a GRIB file (of
> >> course the presence of "GRIB" does not prove it is a GRIB file).
> >> To digress for a moment, if it is a GRIB file then the command,
> >>
> >> strings <filename> | egrep 'GRIB|7777'
> >>
> >> will produce alternating instances of 'GRIB' and '7777'. These strings
> >> mark the beginning and end of each GRIB record. This is true for both
> >> GRIB 1 and 2.
> >>
> >> strings ngac.t00z.aerf24 |grep GRIB
> >> produces nothing so it is not GRIB.
> >>
> >> Second, the warning messages that are emitted when an ngac file that
> >> is valid GRIB is read by NCL such as:
> >>
> >> warning: Entry (62006) not found in code table file
> >> /Users/shea/devel/lib/ncarg/grib2_codetables/ncep/4/4.3.table
> >>
> >> refer to entries not found in table 4.3 which is "type of generating
> >> process". This is not the same as table 4.233 "Aerosol Type"
> >> and so the number 62006 in this context has nothing to do with 62006
> >> ("Sulphate Dry") in the Aerosol Type table
> >>
> >> 62006 as a type of generating processing is not documented in the 4.3
> >> table and hence the message. Since type of generating process is
> >> usually of less concern to users I am considering eliminating the
> >> error message in this case.
> >>
> >> It is true that NCL does not yet make use of the 4.233 table. That is
> >> because it is an auxiliary table that needs to be used in the context
> >> of certain GRIB2 product templates that address aerosols. It does not
> >> contain actual parameters. Rather a parameter might be named, e.g.
> >> "Aerosol optical thickness" and based on the template there is another
> >> piece of metadata called "Aerosol Type" which is as documented in the
> >> "Aerosol Types" table. A small extension to our GRIB2 data model is
> >> required to handle these templates. We hope to implement it in time
> >> for 6.4.0 but that is not yet certain.
> >> -dave
> >>
> >>
> >> On Tue, Jul 26, 2016 at 12:23 PM, Debasish Hazra
> >> <debasish.hazra5 at gmail.com> wrote:
> >> > Hi Dennis,
> >> >
> >> > The variables in the range 62000 to 62010 in the updated grib2 table
> >> > "http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml"
> are
> >> > "optical depth" and has no units associated with them.
> >> >
> >> > 62000 Total Aerosol
> >> > 62001 Dust Dry
> >> > 62006 Sulphate Dry
> >> > 62008 Sea Salt Dry
> >> > 62009 Black Carbon Dry
> >> > 62010 Particulate Organic Matter Dry
> >> >
> >> > 62010-62016 are mixing ratios in kg/kg
> >> >
> >> > Debasish
> >> >
> >> >
> >> >
> >> > On Tue, Jul 26, 2016 at 2:13 PM, Dennis Shea <shea at ucar.edu> wrote:
> >> >>
> >> >> Again, I will forward this. please send to ncl-talk. Otherwise, I am
> >> >> just
> >> >> a middleman forwarding the emails.
> >> >>
> >> >> Usually the GRIB tables have units:
> >> >>
> >> >> Examples:
> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_141.htm
> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ecmwf_132.htm
> >> >>
> >> >> NCL ... see GRIB section
> >> >> See:
> >> >>
> >> >>
> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml
> >> >>
> >> >> On Tue, Jul 26, 2016 at 12:05 PM, Debasish Hazra
> >> >> <debasish.hazra5 at gmail.com> wrote:
> >> >>>
> >> >>> The link updated grib2 table is here :
> >> >>>
> >> >>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
> >> >>>
> >> >>> Thanks
> >> >>> Debasish
> >> >>>
> >> >>> On Tue, Jul 26, 2016 at 1:33 PM, Dennis Shea <shea at ucar.edu> wrote:
> >> >>>>
> >> >>>> The issue is that the NGAC grib tables are not publicly available.
> >> >>>> If you know of where they are located, they can be included within
> >> >>>> NCL
> >> >>>>
> >> >>>> In fact, there is a JIRA ticket on this:
> >> >>>>
> >> >>>> [JIRA] (NCL-2442) GRIB-2: warning: Entry (not found in code table
> >> >>>> file
> >> >>>> .../grib2_codetables/ncep/4/4.3.table
> >> >>>>
> >> >>>> Just yesterday, a comment was made on this ticket:
> >> >>>>
> >> >>>> ---------
> >> >>>>
> >> >>>> Unfortunately the "not found in code table file" errors
> >> >>>> that Dennis reported in this ticket cannot be corrected
> >> >>>> because the entries for that were not found are not documented
> >> >>>> by the NCEP GRIB2 web pages. Table 4.3 contains the generating
> >> >>>> process
> >> >>>> mapping. The values found in the file, all in the range 62000 to
> >> >>>> 62016,
> >> >>>> probably refer to local "generating processes" related to the NGAC
> >> >>>> model.
> >> >>>> If documentation can be found to describe these I would be happy to
> >> >>>> add
> >> >>>> them to the tables.
> >> >>>>
> >> >>>> The other option to get rid of these admittedly annoying
> >> >>>> messages might be to just not print the warnings for this piece of
> >> >>>> metadata
> >> >>>> since it is not usually of that much interest to the data
> consumer. I
> >> >>>> would
> >> >>>> be happy to consider this.
> >> >>>>
> >> >>>> ---------
> >> >>>> Note: the *values* returned by NCL are correct. The meta data is
> not
> >> >>>> because the
> >> >>>> NCEP GRIB2 web pages are not available.
> >> >>>>
> >> >>>> --------
> >> >>>>
> >> >>>> Again, if you know where the appropriate tables are located, please
> >> >>>> send
> >> >>>> to ncl-talk.
> >> >>>>
> >> >>>> THX
> >> >>>>
> >> >>>> On Tue, Jul 26, 2016 at 11:21 AM, Debasish Hazra
> >> >>>> <debasish.hazra5 at gmail.com> wrote:
> >> >>>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> I am trying to read grib2 file with ncl (v6.3) and it shows the
> >> >>>>> following error while trying to open it:
> >> >>>>>
> >> >>>>> fatal:An internal error has occurred. The file format requested
> does
> >> >>>>> not appear to be supported, could not open
> (ngac.t00z.aerf24.grib2)
> >> >>>>> Variable: f
> >> >>>>> Type: file
> >> >>>>> (0) File Missing Value : -1
> >> >>>>>
> >> >>>>> I kept the data file in ftp. Any help on this is appreciated.
> >> >>>>>
> >> >>>>> Thanks.
> >> >>>>> Debasish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160727/b2a62e77/attachment.html
More information about the ncl-talk
mailing list