[ncl-talk] Problem running ncl_convert2nc (Error unpacking GRIB record section 4)
David Brown
dbrown at ucar.edu
Wed Dec 28 14:06:12 MST 2016
Hi Ligia,
I took a look at this file both with NCL and wgrib2. My version of
wgrib2 is likely newer than Dennis version and does not produce the
errors that he mentions. It produces 12 records that all have the name
BRTEMP (Brightness temperature). Unfortunately it does not output any
metadata that would distinguish these records. I believe that it does
not completely decode the PDS template 32, and therefore does not put
out any information about the simulated satellite channels.
NCL depends on the NCEP library g2clib to do the low level decoding of
GRIB2 records. This library does not yet support PDS template 32 at
all. That is the reason NCL is having a problem with this file.
Presumably CDO also does not know this template, since it tries to
make all 12 records into various timesteps of the the single BRTEMP
variable -- even though the time stamp on all the records is clearly
the same.
It would help our efforts to add support for this template if you
could send the control file that was used to generate this GRIB2 file.
Thank you.
-dave
On Wed, Dec 28, 2016 at 1:15 PM, Ligia Bernardet - NOAA Affiliate
<ligia.bernardet at noaa.gov> wrote:
> Hi Dennis,
>
> I also had the problem with the CDO not being built with GRIB support, but
> the NOAA HPC sys adm rebuilt it yesterday and it started working.
>
> The variable name is indeed nonsensical, and it is weird that CDO interprets
> the data as 1 variable at 18 times (should be 18 variables at 1 time). This
> is undesirable but tolerable, since I know what the variables are. This file
> was produced by the NCEP Unified Postprocessor (UPP) and I have the control
> file that was used to generate it, so I can see which variables are selected
> for output. They are indeed Brightness Temperatures (K) produced by a
> forecast model to simulated various channels of various satellites. Offline
> metadata is dangerous, but I guess I have to resort to it for now.
>
> Thanks,
> Ligia
>
> On Wed, Dec 28, 2016 at 1:06 PM, Dennis Shea <shea at ucar.edu> wrote:
>>
>> Initially, I did try the CDO on my local linux box but I got
>>
>> %> cdo -f nc copy gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2
>> gon_CDO.nc
>>
>> Error (gribapiScanTimestep1) : GRIB_API support not compiled in!
>>
>> ---
>> I guess my local version needs an upgrade.
>>
>> ---
>> I did try the above on another machine and got
>>
>> /glade/p/ncldev/data/grib2>ncdump -h gon_CDO.nc
>>
>> netcdf gon_CDO {
>> dimensions:
>> lon = 1440 ;
>> lat = 721 ;
>> time = UNLIMITED ; // (18 currently)
>> variables:
>> double lon(lon) ;
>> lon:standard_name = "longitude" ;
>> lon:long_name = "longitude" ;
>> lon:units = "degrees_east" ;
>> lon:axis = "X" ;
>> double lat(lat) ;
>> lat:standard_name = "latitude" ;
>> lat:long_name = "latitude" ;
>> lat:units = "degrees_north" ;
>> lat:axis = "Y" ;
>> double time(time) ;
>> time:standard_name = "time" ;
>> time:units = "hours since 2014-10-14 12:00:00" ;
>> time:calendar = "proleptic_gregorian" ;
>> float param7.5.0(time, lat, lon) ;
>> param7.5.0:_FillValue = -9.e+33f ;
>> param7.5.0:missing_value = -9.e+33f ;
>>
>> // global attributes:
>> :CDI = "Climate Data Interface version 1.6.3
>> (http://code.zmaw.de/projects/cdi)" ;
>> :Conventions = "CF-1.4" ;
>> :history = "Wed Dec 28 12:04:14 2016: cdo -f nc copy
>> gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2 gon_CDO.nc" ;
>> :institution = "National Centers for Environmental Prediction" ;
>> :CDO = "Climate Data Operators version 1.6.3
>> (http://code.zmaw.de/projects/cdo)" ;
>> }
>>
>>
>> The CDO returns the variable with a nonsensical name:
>>
>> float param7.5.0(time, lat, lon) ;
>>
>> but no units or long_name .... How do you know what it is?
>> I am sure the numbers are unpacked correctly ... but ... without using
>> some external information, how do you know to what 'param7.5.0' refers?
>>
>> ===
>> Combining the CDO file with NCL naming (via an appropriate internal lookup
>> table) you would get:
>>
>> float BRTEMP_P32_GLL0 ( lat_0, lon_0 )
>> long_name : Brightness temperature
>> units : K
>>
>> param7.5.0 ===> BRTEMP_P32_GLL0
>>
>> and the attributes should be:
>>
>> long_name : Brightness temperature
>> units : K
>>
>> D
>>
>> On Wed, Dec 28, 2016 at 11:04 AM, Ligia Bernardet - NOAA Affiliate
>> <ligia.bernardet at noaa.gov> wrote:
>>>
>>> Dennis,
>>>
>>> Thanks for your willingness to help.
>>>
>>> I was able to use CDO to convert the GRIB2 file to NetCDF. All 18 fields
>>> are in the output file (interestingly, it appears as one variable at 18 time
>>> levels, which is definitely not the case in the GRIB file ). I was able to
>>> open the resulting NetCDF file in NCL and manipulate it. So, I am on my way
>>> to get the results I need.
>>>
>>> But if you guys get this working in NCL directly, let me know.
>>>
>>> Thanks again,
>>> Ligia
>>>
>>> On Wed, Dec 28, 2016 at 8:07 AM, Dennis Shea <shea at ucar.edu> wrote:
>>>>
>>>> Hi Ligea
>>>>
>>>> FYI ... a tool unrelated to NCL also fails.
>>>>
>>>> ------
>>>> There is another tool available from NCEP: wgrib2
>>>>
>>>> http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/
>>>>
>>>> There is an option to create netCDF
>>>>
>>>> http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/netcdf.html
>>>>
>>>> ===
>>>> When I use this tool on the grib2 file you sent, it also fails:
>>>>
>>>> %> wgrib2 gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2 -netcdf
>>>> gon.nc
>>>>
>>>> ---
>>>> The wgrib2 tool prints many warning lines like:
>>>>
>>>> code_table_4.5a: product definition template #32 not supported
>>>> code_table_4.5b: product definition template #32 not supported
>>>> ....
>>>> code_table_4.5b: product definition template #32 not supported
>>>> 1:0:d=2014101412:var discipline=0 master_table=8 parmcat=5
>>>> parm=7:no_level:24 hour fcst:
>>>> code_table_4.5a: product definition template #32 not supported
>>>> ...
>>>> code_table_4.5b: product definition template #32 not supported
>>>> 2:279767:d=2014101412:var discipline=0 master_table=8 parmcat=5
>>>> parm=7:no_level:24 hour fcst:
>>>> code_table_4.5a: product definition template #32 not supported
>>>> ...
>>>>
>>>> etc, etc
>>>>
>>>> Like 'ncl_convert2nc', wgrib2 produces just one variable within the
>>>> output netCDF. Specifically,
>>>>
>>>>
>>>> %> ncddump -h gon.nc
>>>>
>>>> netcdf gon {
>>>> dimensions:
>>>> latitude = 721 ;
>>>> longitude = 1440 ;
>>>> time = UNLIMITED ; // (1 currently)
>>>> variables:
>>>> double latitude(latitude) ;
>>>> latitude:units = "degrees_north" ;
>>>> latitude:long_name = "latitude" ;
>>>> double longitude(longitude) ;
>>>> longitude:units = "degrees_east" ;
>>>> longitude:long_name = "longitude" ;
>>>> double time(time) ;
>>>> time:units = "seconds since 1970-01-01 00:00:00.0 0:00" ;
>>>> time:long_name = "verification time generated by wgrib2 function
>>>> verftime()" ;
>>>> time:reference_time = 1413288000. ;
>>>> time:reference_time_type = 3 ;
>>>> time:reference_date = "2014.10.14 12:00:00 UTC" ;
>>>> time:reference_time_description = "forecast or accumulated,
>>>> reference date is fixed" ;
>>>> time:time_step_setting = "auto" ;
>>>> time:time_step = 0. ;
>>>> float var0_5_7_no_level(time, latitude, longitude) ;
>>>> var0_5_7_no_level:_FillValue = 9.999e+20f ;
>>>> var0_5_7_no_level:short_name = "var0_5_7_no_level" ;
>>>> var0_5_7_no_level:long_name = "desc" ;
>>>> var0_5_7_no_level:level = "no_level" ;
>>>> var0_5_7_no_level:units = "unit" ;
>>>>
>>>> // global attributes:
>>>> :Conventions = "COARDS" ;
>>>> :History = "created by wgrib2" ;
>>>> :GRIB2_grid_template = 0 ;
>>>> ==========================================
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Dec 27, 2016 at 5:01 PM, Dennis Shea <shea at ucar.edu> wrote:
>>>>>
>>>>> 6.3.0 will unpack one variable also. The name of the variable will be
>>>>> different because (! wild guess !) the needed table was not complete or
>>>>> available.
>>>>>
>>>>> GRIB is a bit of a 'moving target' :-(
>>>>>
>>>>> I have opened a JIRA ticket: NCL-2533
>>>>>
>>>>> Unfortunately, this is not a great week to get things done.
>>>>>
>>>>> Maybe, an IO developer will look?
>>>>>
>>>>> D
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Dec 27, 2016 at 4:48 PM, Ligia Bernardet - NOAA Affiliate
>>>>> <ligia.bernardet at noaa.gov> wrote:
>>>>>>
>>>>>> Hi Dennis,
>>>>>>
>>>>>> There are 18 fields in the GRIB2 file. In the NetCDF file you sent
>>>>>> back to me there is 1.
>>>>>> But more importantly, I need to find a way to do this myself, so I can
>>>>>> process the multiple files I have.
>>>>>>
>>>>>> The machine I am using, a NOAA's R&D platform, I was able to find NCL
>>>>>> v6.3. Would this version be usable to read this GRIB2 file, until a newer
>>>>>> version is available?
>>>>>>
>>>>>> Thanks,
>>>>>> Ligia
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 27, 2016 at 4:11 PM, Dennis Shea <shea at ucar.edu> wrote:
>>>>>>>
>>>>>>> Attached is a gzip'd version of the netCDF file.
>>>>>>>
>>>>>>> The 'soon-to-be-released' beta 6.4.0 returns a warning message. I
>>>>>>> will open a JIRA ticket.
>>>>>>> Note: I believe the numbers are correct.
>>>>>>>
>>>>>>> =========
>>>>>>>
>>>>>>> NCAR Command Language Version 6.4.0 -27Dec2016_0148
>>>>>>>
>>>>>>> warning:NclGRIB2: BRTEMP_P32 contains records that NCL cannot
>>>>>>> currently differentiate. One or more records will be ignored.
>>>>>>>
>>>>>>> Variable: f
>>>>>>> Type: file
>>>>>>> filename: gonzalo08l.2014101412.hwrfsat.global.0p25.f024
>>>>>>> path:
>>>>>>> /Users/shea/Data/GRIB/gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2
>>>>>>> file global attributes:
>>>>>>> dimensions:
>>>>>>> lat_0 = 721
>>>>>>> lon_0 = 1440
>>>>>>> variables:
>>>>>>> float BRTEMP_P32_GLL0 ( lat_0, lon_0 )
>>>>>>> center : US National Weather Service - NCEP (WMC)
>>>>>>> production_status : Operational products
>>>>>>> long_name : Brightness temperature
>>>>>>> units : K
>>>>>>> _FillValue : 1e+20
>>>>>>> grid_type : Latitude/longitude
>>>>>>> parameter_discipline_and_category : Meteorological
>>>>>>> products, Long wave radiation
>>>>>>> parameter_template_discipline_category_number : ( 32,
>>>>>>> 0, 5, 7 )
>>>>>>> forecast_time : 0
>>>>>>> forecast_time_units : hours
>>>>>>> initial_time : 10/14/2014 (12:00)
>>>>>>>
>>>>>>> float lat_0 ( lat_0 )
>>>>>>> long_name : latitude
>>>>>>> grid_type : Latitude/Longitude
>>>>>>> units : degrees_north
>>>>>>> Dj : 0.25
>>>>>>> Di : 0.25
>>>>>>> Lo2 : 359.75
>>>>>>> La2 : -90
>>>>>>> Lo1 : 0
>>>>>>> La1 : 90
>>>>>>>
>>>>>>> float lon_0 ( lon_0 )
>>>>>>> long_name : longitude
>>>>>>> grid_type : Latitude/Longitude
>>>>>>> units : degrees_east
>>>>>>> Dj : 0.25
>>>>>>> Di : 0.25
>>>>>>> Lo2 : 359.75
>>>>>>> La2 : -90
>>>>>>> Lo1 : 0
>>>>>>> La1 : 90
>>>>>>>
>>>>>>> =========
>>>>>>> The min and max are:
>>>>>>>
>>>>>>> (0) Brightness temperature (K) : min=184.67 max=319.49
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Dec 27, 2016 at 1:13 PM, Ligia Bernardet - NOAA Affiliate
>>>>>>> <ligia.bernardet at noaa.gov> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have transferred file
>>>>>>>> gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2 to /ftp/incoming.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ligia
>>>>>>>>
>>>>>>>> On Tue, Dec 27, 2016 at 1:06 PM, Dennis Shea <shea at ucar.edu> wrote:
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> 6.1.2 (Feb. 2013) is almost 4 years old. Much has happened since.
>>>>>>>>>
>>>>>>>>> [2]
>>>>>>>>> To be able to determine the problem or if the current version of
>>>>>>>>> NCL has addressed the issue, we would need the file.
>>>>>>>>>
>>>>>>>>> ftp ftp.cgd.ucar.edu
>>>>>>>>> anonymous
>>>>>>>>> your_email
>>>>>>>>> cd incoming
>>>>>>>>> prompt
>>>>>>>>> put gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2
>>>>>>>>> quit
>>>>>>>>>
>>>>>>>>> We can not look at /ftp/incoming, so ....
>>>>>>>>>
>>>>>>>>> Let ncl-talk know when the file has been transferred.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Dec 27, 2016 at 12:56 PM, Ligia Bernardet - NOAA Affiliate
>>>>>>>>> <ligia.bernardet at noaa.gov> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I would like to use convert a GRIB2 file generated by NCEP's
>>>>>>>>>> Hurricane WRF (HWRF) model to a NetCDF file. However, I am encountering the
>>>>>>>>>> error below. Any suggestions?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ligia
>>>>>>>>>>
>>>>>>>>>> ncl_convert2nc gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2
>>>>>>>>>>
>>>>>>>>>> Processing file:
>>>>>>>>>> gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2...
>>>>>>>>>> Copyright (C) 1995-2013 - All Rights Reserved
>>>>>>>>>> University Corporation for Atmospheric Research
>>>>>>>>>> NCAR Command Language Version 6.1.2
>>>>>>>>>> The use of this software is governed by a License Agreement.
>>>>>>>>>> See http://www.ncl.ucar.edu/ for more details.
>>>>>>>>>> getpdstemplate: PDS Template 4.32 not defined.
>>>>>>>>>> warning:Error unpacking GRIB record section 4.
>>>>>>>>>> fatal:Could not open
>>>>>>>>>> (./gonzalo08l.2014101412.hwrfsat.global.0p25.f024.grb2)
>>>>>>>>>> Segmentation fault
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> ncl-talk mailing list
>>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
More information about the ncl-talk
mailing list