[ncl-talk] Editing a netcdf4 file

[David Brown]

Hi Gerry,
I have a work-around fix for you, which is to change the line:

var_mod = new((/1,dimz,dimy,dimx/),"float")
to
var_mod = new((/1,dimz,dimy,dimx/),"float","No_FillValue")

Alternately you could say
delete(var_mod at _FillValue)
after the 'new' statement.

The issue is that NCL creates _FillValue attributes for variables by
default, and these get propagated when the variable is written to a
file. And while coordinates and most attributes are not passed along
when you do a value-only assignment (using (/ .../) syntax), the
_FillValue is. This was never a problem with traditional NetCDF.

But in NetCDF4 it seems the _FillValue attribute can only be written
when "the file is still in initial define mode"  (according to
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fput_005fatt_005f-type.html#nc_005fput_005fatt_005f-type).

I think this means that _FillValue attributes cannot be added to
variables that did not have them originally. We will need to account
for this in NCL, but in the meantime you can make things work by
eliminating the _FillValue attribute from the variable you are using
to replace the original QRAIN variable in the file.

Hope this helps.
 -dave




On Mon, Jun 22, 2015 at 5:41 PM, David Brown <dbrown at ucar.edu> wrote:
> Hi Gerry,
> I have been able to duplicate this seg fault and I am looking into it now.
>  -dave
>
> On Mon, Jun 22, 2015 at 4:14 PM, Gerry Creager - NOAA Affiliate
> <gerry.creager at noaa.gov> wrote:
>> David
>>
>> Looks like it may not have been fixed quite yet. Mary sent me a version and
>> I finally fooled the system into working today, when testing began. The
>> short script included, inline below, can be run against the netcdf (wrfout)
>> I uploaded recently (advise if I need to upload it again), results in a seg
>> fault.
>>
>>
>> ;=============
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>> ;=============
>> begin
>>
>> dimx = 225
>> dimy = 225
>> dimz = 55
>>
>> a = addfile("wrfinput_d02.nc", "w")
>> var_orig=a->QRAIN
>> var_mod = new((/1,dimz,dimy,dimx/),"float")
>> var_mod = 5.
>>
>> print(dimsizes(var_orig))
>> print(dimsizes(var_mod))
>>
>> a->QRAIN = (/var_mod/)
>>
>> end
>>
>>
>> On Mon, Jun 8, 2015 at 9:00 PM, David Brown <dbrown at ucar.edu> wrote:
>>>
>>> Dustan and Gerry,
>>> This issue relating to writing a variable with chunking in an existing
>>> NetCDF4 file has been fixed. Also a secondary issue involving
>>> re-writing existing variable attributes has been addressed as well.
>>> Let us know if you need a version with these fixes and we will try to
>>> accommodate you as soon as possible.
>>> Thanks for letting us know of this problem.
>>>  -dave
>>>
>>>
>>> On Wed, Jun 3, 2015 at 8:50 AM, David Brown <dbrown at ucar.edu> wrote:
>>> > Hi Dustin and Gerry,
>>> > I have confirmed there is a problem. It has been partially but not
>>> > completely fixed in the development version of the code. I am looking
>>> > into the remaining issue and will let you know when it has been fixed.
>>> >  -dave
>>> >
>>> > On Tue, Jun 2, 2015 at 10:34 AM, Gerry Creager - NOAA Affiliate
>>> > <gerry.creager at noaa.gov> wrote:
>>> >> Additional information: If we do an addfile("<filename>","r"), we can
>>> >> open
>>> >> and read the file. If we use "w", well, we can open the file but
>>> >> assigning
>>> >> an existing variable to an internal variable fails on the assign (var =
>>> >> a->QRAIN). Of course, because it fails there, we can't see if it'd also
>>> >> fail
>>> >> on write.
>>> >>
>>> >> Thanks
>>> >> Gerry
>>> >>
>>> >> On Mon, Jun 1, 2015 at 7:15 PM, Gerry Creager - NOAA Affiliate
>>> >> <gerry.creager at noaa.gov> wrote:
>>> >>>
>>> >>> I've started looking at this with Dusty. Appreciate any thought you
>>> >>> might
>>> >>> have as to why we're failing.
>>> >>>
>>> >>> I've uploaded the input file (wrfinput_d02) to the FTP incoming site.
>>> >>> I'm
>>> >>> testing with the script:
>>> >>>
>>> >>> var = "QRAIN"
>>> >>> print(var)
>>> >>> a=addfile("wrfinput_d02.nc", "w")
>>> >>> var_orig=a->QRAIN
>>> >>> print(var_orig)
>>> >>>
>>> >>>
>>> >>> The result looks like:gerry at loki:~/NEWSe/NCL> ncl test.ncl
>>> >>>  Copyright (C) 1995-2015 - All Rights Reserved
>>> >>>  University Corporation for Atmospheric Research
>>> >>>  NCAR Command Language Version 6.3.0
>>> >>>  The use of this software is governed by a License Agreement.
>>> >>>  See http://www.ncl.ucar.edu/ for more details.
>>> >>>
>>> >>>
>>> >>> Variable: var
>>> >>> Type: string
>>> >>> Total Size: 8 bytes
>>> >>>             1 values
>>> >>> Number of Dimensions: 1
>>> >>> Dimensions and sizes: [1]
>>> >>> Coordinates:
>>> >>> (0) QRAIN
>>> >>> fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in nc_def_var_chunking
>>> >>> in
>>> >>> file (wrfinput_d02.nc) for writing, at line: 3073
>>> >>>
>>> >>> line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var properties,
>>> >>> like
>>> >>> deflate, after enddef.
>>> >>>
>>> >>>
>>> >>> On Fri, May 29, 2015 at 8:54 PM, Dustan Wheatley - NOAA Affiliate
>>> >>> <dustan.wheatley at noaa.gov> wrote:
>>> >>>>
>>> >>>> I am trying to overwrite a netcdf4 file analogous to what is shown at
>>> >>>> http://www.ncl.ucar.edu/Applications/change_netCDF.shtml.  In the
>>> >>>> attached
>>> >>>> ncl script, I open the file to be modified at line 35 and attempt the
>>> >>>> modification at line 73 (with a lot of calculations in between).
>>> >>>> While I
>>> >>>> have double-checked that the modified field (var_mod) has the same
>>> >>>> dimensionality as the original variable (QRAIN), I am still get a
>>> >>>> segmentation fault/core dump.  Never had this problem when I tried to
>>> >>>> manipulate netcdf3 files.  Could it possibly be related to the
>>> >>>> netcdf4
>>> >>>> compression?  It’s possible I may be trying to do something that’s no
>>> >>>> longer
>>> >>>> possible.
>>> >>>>
>>> >>>> Thanks in advance for any suggestions.
>>> >>>>
>>> >>>> -Dusty Wheatley
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> ncl-talk mailing list
>>> >>>> ncl-talk at ucar.edu
>>> >>>> List instructions, subscriber options, unsubscribe:
>>> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Gerry Creager
>>> >>> NSSL/CIMMS
>>> >>> 405.325.6371
>>> >>> ++++++++++++++++++++++
>>> >>> “Big whorls have little whorls,
>>> >>> That feed on their velocity;
>>> >>> And little whorls have lesser whorls,
>>> >>> And so on to viscosity.”
>>> >>> Lewis Fry Richardson (1881-1953)
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Gerry Creager
>>> >> NSSL/CIMMS
>>> >> 405.325.6371
>>> >> ++++++++++++++++++++++
>>> >> “Big whorls have little whorls,
>>> >> That feed on their velocity;
>>> >> And little whorls have lesser whorls,
>>> >> And so on to viscosity.”
>>> >> Lewis Fry Richardson (1881-1953)
>>> >>
>>> >> _______________________________________________
>>> >> ncl-talk mailing list
>>> >> ncl-talk at ucar.edu
>>> >> List instructions, subscriber options, unsubscribe:
>>> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >>
>>
>>
>>
>>
>> --
>> Gerry Creager
>> NSSL/CIMMS
>> 405.325.6371
>> ++++++++++++++++++++++
>> “Big whorls have little whorls,
>> That feed on their velocity;
>> And little whorls have lesser whorls,
>> And so on to viscosity.”
>> Lewis Fry Richardson (1881-1953)