[ncl-talk] grib2 file read

David Brown dbrown at ucar.edu
Tue Jul 26 19:40:09 MDT 2016


I don't know how this file is organized. I do know that the 'strings'
tool outputs some stuff that looks like variable names at the
beginning:

strings ngac.t00z.aerf24 |more

0NEMSIO  GFS     bin4

bccmass         bcdp001         bcdp002         bcem001
bcem002         bceman          bcembb          bcembf
bchyphil        bcsd001         bcsd002         bcsmass
bcsv001         bcsv002         bcwt001         bcwt002
dmscmass        dmssmass        ducmass25       ducmass
dudp001         dudp002         dudp003         dudp004
dudp005         duem001         duem002         duem003
duem004         duem005         dusd001         dusd002
dusd003         dusd004         dusd005         dusmass25
dusmass         dusv001         dusv002         dusv003
dusv004         dusv005         duwt001         duwt002
duwt003         duwt004         duwt005         occmass
ocdp001         ocdp002         ocem001         ocem002         oceman
         ocembb          ocembf          ocembg          ochyphil
  ocsd001         ocsd002         ocsmass         ocsv001
ocsv002         ocwt001         ocwt002         so2cmass
so2eman         so2embb         so2emve         so2emvn
so2smass        so4cmass        so4eman         so4smass
sscmass25       sscmass         ssdp001         ssdp002
ssdp003         ssdp004         ssdp005         ssem001
ssem002         ssem003         ssem004         ssem005
sssd001         sssd002         sssd003         sssd004
sssd005         sssmass25       sssmass         sssv001
sssv002         sssv003         sssv004         sssv005
sswt001         sswt002         sswt003         sswt004
sswt005         sudp001         sudp002         sudp003
sudp004         suem001         suem002         suem003
suem004         supmsa          supso2          supso4aq
supso4g         supso4wt        susd001         susd002
susd003         susd004         susv001         susv002
susv003         susv004         suwt001         suwt002
suwt003         suwt004

atmos col       atmos col       atmos col       atmos col       atmos
col       atmos col       atmos col       atmos col       atmos col
   atmos col       atmos col       atmos col       atmos col
atmos col       atmos col       atmos col       atmos col       atmos
col       atmos col       atmos col

.....

The "bin4" at the beginning suggests maybe that it is 4 byte floats,
but other than than I have no idea.

I think you will need to find some explanation of the contents of this
file from the people who created it.
 -dave


On Tue, Jul 26, 2016 at 7:16 PM, Debasish Hazra
<debasish.hazra5 at gmail.com> wrote:
> Thanks David. Do I need to read the file as "binary" ?
>
> Debasish
>
> On Tue, Jul 26, 2016 at 6:12 PM, David Brown <dbrown at ucar.edu> wrote:
>>
>> I am afraid there have been some misunderstandings.
>> First of all the file "ngac.t00z.aerf24.grib2" (actually called
>> ngac.t00z.aerf24 on the ftp site if anyone else wants to download it)
>> is
>> not a GRIB file at all. A simple test if you suspect something is not
>> GRIB:
>>
>> strings <filename> | grep GRIB
>>
>> If this produces nothing then it's definitely not a GRIB file (of
>> course the presence of "GRIB" does not prove it is a GRIB file).
>> To digress for a moment, if it is a GRIB file then the command,
>>
>> strings <filename> | egrep 'GRIB|7777'
>>
>> will produce alternating instances of 'GRIB' and '7777'. These strings
>> mark the beginning and end of each GRIB record. This is true for both
>> GRIB 1 and 2.
>>
>> strings ngac.t00z.aerf24 |grep GRIB
>> produces nothing so it is not GRIB.
>>
>> Second, the warning messages that are emitted when an ngac file that
>> is valid GRIB is read by NCL such as:
>>
>> warning: Entry (62006) not found in code table file
>> /Users/shea/devel/lib/ncarg/grib2_codetables/ncep/4/4.3.table
>>
>> refer to entries not found in table 4.3 which is "type of generating
>> process". This is not the same as table 4.233 "Aerosol Type"
>> and so the number 62006 in this context has nothing to do with 62006
>> ("Sulphate Dry") in the Aerosol Type table
>>
>> 62006 as a type of generating processing is not documented in the 4.3
>> table and hence the message. Since type of generating process is
>> usually of less concern to users I am considering eliminating the
>> error message in this case.
>>
>> It is true that NCL does not yet make use of the 4.233 table. That is
>> because it is an auxiliary table that needs to be used in the context
>> of certain GRIB2 product templates that address aerosols. It does not
>> contain actual parameters. Rather a parameter might be named, e.g.
>> "Aerosol optical thickness" and based on the template there is another
>> piece of metadata called "Aerosol Type" which is as documented in the
>> "Aerosol Types" table. A small extension to our GRIB2 data model is
>> required to handle these templates. We hope to implement it in time
>> for 6.4.0 but that is not yet certain.
>>  -dave
>>
>>
>> On Tue, Jul 26, 2016 at 12:23 PM, Debasish Hazra
>> <debasish.hazra5 at gmail.com> wrote:
>> > Hi Dennis,
>> >
>> > The variables in the range  62000 to 62010 in the updated grib2 table
>> > "http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml" are
>> > "optical depth" and has no units associated with them.
>> >
>> > 62000 Total Aerosol
>> > 62001 Dust Dry
>> > 62006 Sulphate Dry
>> > 62008 Sea Salt Dry
>> > 62009 Black Carbon Dry
>> > 62010 Particulate Organic Matter Dry
>> >
>> > 62010-62016 are mixing ratios in kg/kg
>> >
>> > Debasish
>> >
>> >
>> >
>> > On Tue, Jul 26, 2016 at 2:13 PM, Dennis Shea <shea at ucar.edu> wrote:
>> >>
>> >> Again, I will forward this. please send to ncl-talk. Otherwise, I am
>> >> just
>> >> a middleman forwarding the emails.
>> >>
>> >> Usually the GRIB tables have units:
>> >>
>> >> Examples:
>> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_141.htm
>> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ecmwf_132.htm
>> >>
>> >> NCL ... see GRIB section
>> >> See:
>> >>
>> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml
>> >>
>> >> On Tue, Jul 26, 2016 at 12:05 PM, Debasish Hazra
>> >> <debasish.hazra5 at gmail.com> wrote:
>> >>>
>> >>> The link updated grib2 table is here :
>> >>>
>> >>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
>> >>>
>> >>> Thanks
>> >>> Debasish
>> >>>
>> >>> On Tue, Jul 26, 2016 at 1:33 PM, Dennis Shea <shea at ucar.edu> wrote:
>> >>>>
>> >>>> The issue is that the NGAC grib tables are not publicly available.
>> >>>> If you know of where they are located, they can be included within
>> >>>> NCL
>> >>>>
>> >>>> In fact, there is a JIRA ticket on this:
>> >>>>
>> >>>> [JIRA] (NCL-2442) GRIB-2: warning: Entry (not found in code table
>> >>>> file
>> >>>> .../grib2_codetables/ncep/4/4.3.table
>> >>>>
>> >>>> Just yesterday, a comment was made on this ticket:
>> >>>>
>> >>>> ---------
>> >>>>
>> >>>> Unfortunately the "not found in code table file" errors
>> >>>> that Dennis reported in this ticket cannot be corrected
>> >>>> because the entries for that were not found are not documented
>> >>>> by the NCEP GRIB2 web pages. Table 4.3 contains the generating
>> >>>> process
>> >>>> mapping. The values found in the file, all in the range 62000 to
>> >>>> 62016,
>> >>>> probably refer to local "generating processes" related to the NGAC
>> >>>> model.
>> >>>> If documentation can be found to describe these I would be happy to
>> >>>> add
>> >>>> them to the tables.
>> >>>>
>> >>>> The other option to get rid of these admittedly annoying
>> >>>> messages might be to just not print the warnings for this piece of
>> >>>> metadata
>> >>>> since it is not usually of that much interest to the data consumer. I
>> >>>> would
>> >>>> be happy to consider this.
>> >>>>
>> >>>> ---------
>> >>>> Note: the *values* returned by NCL are correct. The meta data is not
>> >>>> because the
>> >>>> NCEP GRIB2 web pages are not available.
>> >>>>
>> >>>> --------
>> >>>>
>> >>>> Again, if you know where the appropriate tables are located, please
>> >>>> send
>> >>>> to ncl-talk.
>> >>>>
>> >>>> THX
>> >>>>
>> >>>> On Tue, Jul 26, 2016 at 11:21 AM, Debasish Hazra
>> >>>> <debasish.hazra5 at gmail.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I am trying to read grib2 file with ncl (v6.3) and it shows the
>> >>>>> following error while trying to open it:
>> >>>>>
>> >>>>> fatal:An internal error has occurred. The file format requested does
>> >>>>> not appear to be supported, could not open (ngac.t00z.aerf24.grib2)
>> >>>>> Variable: f
>> >>>>> Type: file
>> >>>>> (0) File Missing Value : -1
>> >>>>>
>> >>>>> I kept the data file in ftp. Any help on this is appreciated.
>> >>>>>
>> >>>>> Thanks.
>> >>>>> Debasish
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> ncl-talk mailing list
>> >>>>> ncl-talk at ucar.edu
>> >>>>> List instructions, subscriber options, unsubscribe:
>> >>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> ncl-talk mailing list
>> >>> ncl-talk at ucar.edu
>> >>> List instructions, subscriber options, unsubscribe:
>> >>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > ncl-talk mailing list
>> > ncl-talk at ucar.edu
>> > List instructions, subscriber options, unsubscribe:
>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >
>
>


More information about the ncl-talk mailing list