[ncl-talk] grib2 file read
Debasish Hazra
debasish.hazra5 at gmail.com
Wed Jul 27 13:50:51 MDT 2016
Thanks Dave and David. Lets see how more information can be retrieved.
Debasish
On Wed, Jul 27, 2016 at 1:58 PM, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:
> This file is fortran unformatted binary sequential, big endian, with
> varying record lengths. There are mixed data types, at least character and
> integer, probably also some floats, if I am not mistaken. The first record
> begins with the string "NEMSIO GFS bin4" which is likely a format
> identifier.
>
> This is a custom made fortran file. This will be impossible to decode
> without some source documentation. NCL may have problems with the varying
> length records and mixed data types. This file was made for fortran
> access, not for NCL or other common climate software.
>
> --Dave
>
>
> On Tue, Jul 26, 2016 at 7:40 PM, David Brown <dbrown at ucar.edu> wrote:
>
>> I don't know how this file is organized. I do know that the 'strings'
>> tool outputs some stuff that looks like variable names at the
>> beginning:
>>
>> strings ngac.t00z.aerf24 |more
>>
>> 0NEMSIO GFS bin4
>>
>> bccmass bcdp001 bcdp002 bcem001
>> bcem002 bceman bcembb bcembf
>> bchyphil bcsd001 bcsd002 bcsmass
>> bcsv001 bcsv002 bcwt001 bcwt002
>> dmscmass dmssmass ducmass25 ducmass
>> dudp001 dudp002 dudp003 dudp004
>> dudp005 duem001 duem002 duem003
>> duem004 duem005 dusd001 dusd002
>> dusd003 dusd004 dusd005 dusmass25
>> dusmass dusv001 dusv002 dusv003
>> dusv004 dusv005 duwt001 duwt002
>> duwt003 duwt004 duwt005 occmass
>> ocdp001 ocdp002 ocem001 ocem002 oceman
>> ocembb ocembf ocembg ochyphil
>> ocsd001 ocsd002 ocsmass ocsv001
>> ocsv002 ocwt001 ocwt002 so2cmass
>> so2eman so2embb so2emve so2emvn
>> so2smass so4cmass so4eman so4smass
>> sscmass25 sscmass ssdp001 ssdp002
>> ssdp003 ssdp004 ssdp005 ssem001
>> ssem002 ssem003 ssem004 ssem005
>> sssd001 sssd002 sssd003 sssd004
>> sssd005 sssmass25 sssmass sssv001
>> sssv002 sssv003 sssv004 sssv005
>> sswt001 sswt002 sswt003 sswt004
>> sswt005 sudp001 sudp002 sudp003
>> sudp004 suem001 suem002 suem003
>> suem004 supmsa supso2 supso4aq
>> supso4g supso4wt susd001 susd002
>> susd003 susd004 susv001 susv002
>> susv003 susv004 suwt001 suwt002
>> suwt003 suwt004
>>
>> atmos col atmos col atmos col atmos col atmos
>> col atmos col atmos col atmos col atmos col
>> atmos col atmos col atmos col atmos col
>> atmos col atmos col atmos col atmos col atmos
>> col atmos col atmos col
>>
>> .....
>>
>> The "bin4" at the beginning suggests maybe that it is 4 byte floats,
>> but other than than I have no idea.
>>
>> I think you will need to find some explanation of the contents of this
>> file from the people who created it.
>> -dave
>>
>>
>> On Tue, Jul 26, 2016 at 7:16 PM, Debasish Hazra
>> <debasish.hazra5 at gmail.com> wrote:
>> > Thanks David. Do I need to read the file as "binary" ?
>> >
>> > Debasish
>> >
>> > On Tue, Jul 26, 2016 at 6:12 PM, David Brown <dbrown at ucar.edu> wrote:
>> >>
>> >> I am afraid there have been some misunderstandings.
>> >> First of all the file "ngac.t00z.aerf24.grib2" (actually called
>> >> ngac.t00z.aerf24 on the ftp site if anyone else wants to download it)
>> >> is
>> >> not a GRIB file at all. A simple test if you suspect something is not
>> >> GRIB:
>> >>
>> >> strings <filename> | grep GRIB
>> >>
>> >> If this produces nothing then it's definitely not a GRIB file (of
>> >> course the presence of "GRIB" does not prove it is a GRIB file).
>> >> To digress for a moment, if it is a GRIB file then the command,
>> >>
>> >> strings <filename> | egrep 'GRIB|7777'
>> >>
>> >> will produce alternating instances of 'GRIB' and '7777'. These strings
>> >> mark the beginning and end of each GRIB record. This is true for both
>> >> GRIB 1 and 2.
>> >>
>> >> strings ngac.t00z.aerf24 |grep GRIB
>> >> produces nothing so it is not GRIB.
>> >>
>> >> Second, the warning messages that are emitted when an ngac file that
>> >> is valid GRIB is read by NCL such as:
>> >>
>> >> warning: Entry (62006) not found in code table file
>> >> /Users/shea/devel/lib/ncarg/grib2_codetables/ncep/4/4.3.table
>> >>
>> >> refer to entries not found in table 4.3 which is "type of generating
>> >> process". This is not the same as table 4.233 "Aerosol Type"
>> >> and so the number 62006 in this context has nothing to do with 62006
>> >> ("Sulphate Dry") in the Aerosol Type table
>> >>
>> >> 62006 as a type of generating processing is not documented in the 4.3
>> >> table and hence the message. Since type of generating process is
>> >> usually of less concern to users I am considering eliminating the
>> >> error message in this case.
>> >>
>> >> It is true that NCL does not yet make use of the 4.233 table. That is
>> >> because it is an auxiliary table that needs to be used in the context
>> >> of certain GRIB2 product templates that address aerosols. It does not
>> >> contain actual parameters. Rather a parameter might be named, e.g.
>> >> "Aerosol optical thickness" and based on the template there is another
>> >> piece of metadata called "Aerosol Type" which is as documented in the
>> >> "Aerosol Types" table. A small extension to our GRIB2 data model is
>> >> required to handle these templates. We hope to implement it in time
>> >> for 6.4.0 but that is not yet certain.
>> >> -dave
>> >>
>> >>
>> >> On Tue, Jul 26, 2016 at 12:23 PM, Debasish Hazra
>> >> <debasish.hazra5 at gmail.com> wrote:
>> >> > Hi Dennis,
>> >> >
>> >> > The variables in the range 62000 to 62010 in the updated grib2 table
>> >> > "http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml"
>> are
>> >> > "optical depth" and has no units associated with them.
>> >> >
>> >> > 62000 Total Aerosol
>> >> > 62001 Dust Dry
>> >> > 62006 Sulphate Dry
>> >> > 62008 Sea Salt Dry
>> >> > 62009 Black Carbon Dry
>> >> > 62010 Particulate Organic Matter Dry
>> >> >
>> >> > 62010-62016 are mixing ratios in kg/kg
>> >> >
>> >> > Debasish
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Jul 26, 2016 at 2:13 PM, Dennis Shea <shea at ucar.edu> wrote:
>> >> >>
>> >> >> Again, I will forward this. please send to ncl-talk. Otherwise, I am
>> >> >> just
>> >> >> a middleman forwarding the emails.
>> >> >>
>> >> >> Usually the GRIB tables have units:
>> >> >>
>> >> >> Examples:
>> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ncep_141.htm
>> >> >> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/ecmwf_132.htm
>> >> >>
>> >> >> NCL ... see GRIB section
>> >> >> See:
>> >> >>
>> >> >>
>> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml
>> >> >>
>> >> >> On Tue, Jul 26, 2016 at 12:05 PM, Debasish Hazra
>> >> >> <debasish.hazra5 at gmail.com> wrote:
>> >> >>>
>> >> >>> The link updated grib2 table is here :
>> >> >>>
>> >> >>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-233.shtml
>> >> >>>
>> >> >>> Thanks
>> >> >>> Debasish
>> >> >>>
>> >> >>> On Tue, Jul 26, 2016 at 1:33 PM, Dennis Shea <shea at ucar.edu>
>> wrote:
>> >> >>>>
>> >> >>>> The issue is that the NGAC grib tables are not publicly available.
>> >> >>>> If you know of where they are located, they can be included within
>> >> >>>> NCL
>> >> >>>>
>> >> >>>> In fact, there is a JIRA ticket on this:
>> >> >>>>
>> >> >>>> [JIRA] (NCL-2442) GRIB-2: warning: Entry (not found in code table
>> >> >>>> file
>> >> >>>> .../grib2_codetables/ncep/4/4.3.table
>> >> >>>>
>> >> >>>> Just yesterday, a comment was made on this ticket:
>> >> >>>>
>> >> >>>> ---------
>> >> >>>>
>> >> >>>> Unfortunately the "not found in code table file" errors
>> >> >>>> that Dennis reported in this ticket cannot be corrected
>> >> >>>> because the entries for that were not found are not documented
>> >> >>>> by the NCEP GRIB2 web pages. Table 4.3 contains the generating
>> >> >>>> process
>> >> >>>> mapping. The values found in the file, all in the range 62000 to
>> >> >>>> 62016,
>> >> >>>> probably refer to local "generating processes" related to the NGAC
>> >> >>>> model.
>> >> >>>> If documentation can be found to describe these I would be happy
>> to
>> >> >>>> add
>> >> >>>> them to the tables.
>> >> >>>>
>> >> >>>> The other option to get rid of these admittedly annoying
>> >> >>>> messages might be to just not print the warnings for this piece of
>> >> >>>> metadata
>> >> >>>> since it is not usually of that much interest to the data
>> consumer. I
>> >> >>>> would
>> >> >>>> be happy to consider this.
>> >> >>>>
>> >> >>>> ---------
>> >> >>>> Note: the *values* returned by NCL are correct. The meta data is
>> not
>> >> >>>> because the
>> >> >>>> NCEP GRIB2 web pages are not available.
>> >> >>>>
>> >> >>>> --------
>> >> >>>>
>> >> >>>> Again, if you know where the appropriate tables are located,
>> please
>> >> >>>> send
>> >> >>>> to ncl-talk.
>> >> >>>>
>> >> >>>> THX
>> >> >>>>
>> >> >>>> On Tue, Jul 26, 2016 at 11:21 AM, Debasish Hazra
>> >> >>>> <debasish.hazra5 at gmail.com> wrote:
>> >> >>>>>
>> >> >>>>> Hi,
>> >> >>>>>
>> >> >>>>> I am trying to read grib2 file with ncl (v6.3) and it shows the
>> >> >>>>> following error while trying to open it:
>> >> >>>>>
>> >> >>>>> fatal:An internal error has occurred. The file format requested
>> does
>> >> >>>>> not appear to be supported, could not open
>> (ngac.t00z.aerf24.grib2)
>> >> >>>>> Variable: f
>> >> >>>>> Type: file
>> >> >>>>> (0) File Missing Value : -1
>> >> >>>>>
>> >> >>>>> I kept the data file in ftp. Any help on this is appreciated.
>> >> >>>>>
>> >> >>>>> Thanks.
>> >> >>>>> Debasish
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160727/1b612ba7/attachment.html
More information about the ncl-talk
mailing list