[ncl-talk] Problem GRIB2 file
David Brown
dbrown at ucar.edu
Wed Apr 22 14:58:44 MDT 2015
2.4 GB is the size of the only data variable in the file (Ice cover).
NCL also provides the 2D lat and lon coordinates and the vector
rotation variable. These are "value-added" variables created by NCL.
Each of these are also 2.4 GB. That explains the factor of 4
difference in size between the NCL-generated file and the CDO file.
-dave
On Wed, Apr 22, 2015 at 2:22 PM, HAVENER, KEVIN F GS-12 USAF ACC 14
WS/WXED <kevin.havener at us.af.mil> wrote:
> It probably was. My laptop doesn't have that much memory and the server I was using has 12 but is running some other pretty heavy memory intensive jobs, too. I was able to convert the file using CDO, resulting in a 2.4G file. Don't know if any compression was involved there, but I gzipped it down to 4M after converting. CDO seemed to produce a valid netcdf. I will mess around some more and see if there is anything else to be learned. Probably just use the 4km version after I find out what all parameters are actually in that grib file.
>
> Kevin Havener
>
>
> -----Original Message-----
> From: David Brown [mailto:dbrown at ucar.edu]
> Sent: Wednesday, April 22, 2015 3:16 PM
> To: HAVENER, KEVIN F GS-12 USAF ACC 14 WS/WXED
> Cc: Dennis Shea; ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problem GRIB2 file
>
> Hi Kevin,
>
> I was able to convert this file (retrieved from
> http://www.natice.noaa.gov/ims/grib2_v3_p2_1km.html) to NetCDF using ncl_convert2nc. I tracked the memory usage and found that I was using more than 10 GB of memory at the the peak. I wonder if your symptoms (the systemfunc error) are caused by a lack of sufficient memory.
>
> The output NetCDF file, created without compression using the -nc4c flag, is almost 10 GB.
> -dave
>
>
> On Wed, Apr 22, 2015 at 11:55 AM, HAVENER, KEVIN F GS-12 USAF ACC 14 WS/WXED <kevin.havener at us.af.mil> wrote:
>> Thanks for the advice, Dennis. I got rid of error number 2 on 6.2.0. Now both 6.2.0 and 6.3.0 give error 1:
>>
>> fatal:systemfunc: cannot create child process:[errno=12]
>> fatal:["Execute.c":8567]:Execute: Error occurred at or near line 270
>> in file /tmp/tmp9336-5537df08.ncl
>>
>> I will do some more thinking and cooperating with the National Ice Center. I'm not sure that 1km resolution grib file is constructed properly, anyway.
>>
>> Kevin Havener
>>
>>
>> -----Original Message-----
>> From: Dennis Shea [mailto:shea at ucar.edu]
>> Sent: Wednesday, April 22, 2015 12:27 PM
>> To: HAVENER, KEVIN F GS-12 USAF ACC 14 WS/WXED
>> Cc: ncl-talk at ucar.edu
>> Subject: Re: [ncl-talk] Problem GRIB2 file
>>
>> netCDF-3 is the default for ncl_convert2nc. I think no variable can be
>> > 2Gb
>>
>>
>> You must create nc4 (which is, under the hood, HDF5).
>>
>> %> ncl_convert2nc NIC.IMS_ice_v3_201433700_1km.GRIB2 -nc4c
>>
>>
>>
>> 9684190860 NIC.IMS_ice_v3_201433700_1km.nc4
>>
>> --
>>
>>
>> If you want a 'time' dimension
>>
>> %> ncl_convert2nc NIC.IMS_ice_v3_201433700_1km.GRIB2 -nc4c -itime
>>
>>
>> Good luck
>>
>>
>>
>>
>>
>> On Wed, Apr 22, 2015 at 8:08 AM, HAVENER, KEVIN F GS-12 USAF ACC 14 WS/WXED <kevin.havener at us.af.mil> wrote:
>>
>>
>> I am trying to ncl_convert2nc a grib2 file downloaded from the National Ice Center http://www.natice.noaa.gov/ims/grib2_v3_p1.html. I either get this error:
>>
>> Processing file: NIC.IMS_ice_v3_201504900_1km.GRIB2...
>> Copyright (C) 1995-2014 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.2.0
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> fatal:systemfunc: cannot create child process:[errno=12]
>> fatal:["Execute.c":8567]:Execute: Error occurred at or
>> near line 270 in file /tmp/tmp12209-55379965.ncl
>>
>> Or this error:
>>
>> Processing file: NIC.IMS_ice_v3_201504900_1km.GRIB2...
>> Copyright (C) 1995-2014 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.2.0
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>> ncendef: ncid 65536: NetCDF: One or more variable sizes violate format constraints
>> fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (ICEC_P31_GST0) to file (/home/havenerk/NIC.IMS_ice_v3_201504900_1km.nc)
>> fatal:["Execute.c":8567]:Execute: Error occurred at or near line 628 in file /tmp/tmp16245-5537a2e1.ncl
>> ncclose: ncid 65536: NetCDF: One or more variable sizes
>> violate format constraints
>>
>> Pay no attention to the NCL versions. I thought the first might be related to being able to write in a directory, but I got the same on 6.3.0, where I knew that wouldn't be a problem. The second error only occurred under 6.2.0 after I eliminated the potential permission problem by moving the process to my home directory. Only the first error occurs under my own personal laptop 6.3.0.
>>
>> It is a 1km resolution file on a polar stereographic grid. I suspected the file size/resolution might be a problem, so I downloaded a 4km version of the same file. Turns out it is different--has 3 variables in it instead of 1, so is actually larger. Aside from not having the right local table for two of the parameters, the ice cover (only parameter in the 1km version) converted to netcdf properly. So maybe still a resolution issue?
>>
>> Any idea what is going on here? Hopeless case, bad grib2,or user error?
>>
>> Kevin Havener
>>
>> Here is an ncl_filedump of the file:
>>
>> ncl_filedump NIC.IMS_ice_v3_201504900_1km.GRIB2
>> Copyright (C) 1995-2014 - All Rights Reserved
>> University Corporation for Atmospheric Research
>> NCAR Command Language Version 6.2.0
>> The use of this software is governed by a License Agreement.
>> See http://www.ncl.ucar.edu/ for more details.
>>
>> Variable: f
>> Type: file
>> filename: NIC.IMS_ice_v3_201504900_1km
>> path: NIC.IMS_ice_v3_201504900_1km.GRIB2
>> file global attributes:
>> dimensions:
>> ygrid_0 = 24576
>> xgrid_0 = 24576
>> variables:
>> float ICEC_P31_GST0 ( ygrid_0, xgrid_0 )
>> center : US NOAA/NESDIS
>> production_status : Operational products
>> long_name : Ice cover
>> units : Proportion
>> _FillValue : 1e+20
>> coordinates : gridlat_0 gridlon_0
>> grid_type : Polar stereographic can be south or north
>> parameter_discipline_and_category : Oceanographic products, Ice
>> parameter_template_discipline_category_number : ( 31, 10, 2, 0 )
>> forecast_time : 0
>> forecast_time_units : hours
>> initial_time : 02/17/2015 (00:00)
>>
>> float gridrot_0 ( ygrid_0, xgrid_0 )
>> long_name : vector rotation angle
>> GridType : Polar Sterographic Projection (North or South)
>> units : radians
>> formula_u : Ugrid = cos(rot)*Uearth - sin(rot)*Vearth
>> formula_v : Vgrid = sin(rot)*Uearth + cos(rot)*Vearth
>> note1 : u and v components of vector quantities are resolved relative to earth
>> note2 : apply formulas to derive u and v components relative to grid
>>
>> float gridlat_0 ( ygrid_0, xgrid_0 )
>> corners : ( -21.4961, -21.2433, -20.98764, -21.2433 )
>> long_name : latitude
>> grid_type : Polar Sterographic Projection (North or South)
>> units : degrees_north
>> Dy : 1
>> Dx : 1
>> Lov : 280
>> Lo1 : 235
>> La1 : -21.4961
>>
>> float gridlon_0 ( ygrid_0, xgrid_0 )
>> corners : ( -125, -35.27276, 55, 145.2728 )
>> long_name : longitude
>> grid_type : Polar Sterographic Projection (North or South)
>> units : degrees_east
>> Dy : 1
>> Dx : 1
>> Lov : 280
>> Lo1 : 235
>> La1 : -21.4961
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
More information about the ncl-talk
mailing list