[ncl-talk] Editing a netcdf4 file

Gerry Creager - NOAA Affiliate gerry.creager at noaa.gov
Tue Jun 23 09:37:15 MDT 2015 

Dave

I modified the script and...

[gerry at wof-vis NCL]$ ncl test2.ncl
 Copyright (C) 1995-2015 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.3.0
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in nc_def_var_chunking in
file (wrfinput_d02.nc) for writing, at line: 3073

line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var properties, like
deflate, after enddef.

Where to, now?

On Tue, Jun 23, 2015 at 3:35 PM, Gerry Creager - NOAA Affiliate <
gerry.creager at noaa.gov> wrote:

> Dave
>
> Thanks. And I'll bring this up to Russ and the crew at Unidata!
>
> We'll test.
>
> Gerry
>
> On Tue, Jun 23, 2015 at 12:49 AM, David Brown <dbrown at ucar.edu> wrote:
>
>> Hi Gerry,
>> I have a work-around fix for you, which is to change the line:
>>
>> var_mod = new((/1,dimz,dimy,dimx/),"float")
>> to
>> var_mod = new((/1,dimz,dimy,dimx/),"float","No_FillValue")
>>
>> Alternately you could say
>> delete(var_mod at _FillValue)
>> after the 'new' statement.
>>
>> The issue is that NCL creates _FillValue attributes for variables by
>> default, and these get propagated when the variable is written to a
>> file. And while coordinates and most attributes are not passed along
>> when you do a value-only assignment (using (/ .../) syntax), the
>> _FillValue is. This was never a problem with traditional NetCDF.
>>
>> But in NetCDF4 it seems the _FillValue attribute can only be written
>> when "the file is still in initial define mode"  (according to
>>
>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fput_005fatt_005f-type.html#nc_005fput_005fatt_005f-type
>> ).
>>
>> I think this means that _FillValue attributes cannot be added to
>> variables that did not have them originally. We will need to account
>> for this in NCL, but in the meantime you can make things work by
>> eliminating the _FillValue attribute from the variable you are using
>> to replace the original QRAIN variable in the file.
>>
>> Hope this helps.
>>  -dave
>>
>>
>>
>>
>> On Mon, Jun 22, 2015 at 5:41 PM, David Brown <dbrown at ucar.edu> wrote:
>> > Hi Gerry,
>> > I have been able to duplicate this seg fault and I am looking into it
>> now.
>> >  -dave
>> >
>> > On Mon, Jun 22, 2015 at 4:14 PM, Gerry Creager - NOAA Affiliate
>> > <gerry.creager at noaa.gov> wrote:
>> >> David
>> >>
>> >> Looks like it may not have been fixed quite yet. Mary sent me a
>> version and
>> >> I finally fooled the system into working today, when testing began. The
>> >> short script included, inline below, can be run against the netcdf
>> (wrfout)
>> >> I uploaded recently (advise if I need to upload it again), results in
>> a seg
>> >> fault.
>> >>
>> >>
>> >> ;=============
>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>> >> ;=============
>> >> begin
>> >>
>> >> dimx = 225
>> >> dimy = 225
>> >> dimz = 55
>> >>
>> >> a = addfile("wrfinput_d02.nc", "w")
>> >> var_orig=a->QRAIN
>> >> var_mod = new((/1,dimz,dimy,dimx/),"float")
>> >> var_mod = 5.
>> >>
>> >> print(dimsizes(var_orig))
>> >> print(dimsizes(var_mod))
>> >>
>> >> a->QRAIN = (/var_mod/)
>> >>
>> >> end
>> >>
>> >>
>> >> On Mon, Jun 8, 2015 at 9:00 PM, David Brown <dbrown at ucar.edu> wrote:
>> >>>
>> >>> Dustan and Gerry,
>> >>> This issue relating to writing a variable with chunking in an existing
>> >>> NetCDF4 file has been fixed. Also a secondary issue involving
>> >>> re-writing existing variable attributes has been addressed as well.
>> >>> Let us know if you need a version with these fixes and we will try to
>> >>> accommodate you as soon as possible.
>> >>> Thanks for letting us know of this problem.
>> >>>  -dave
>> >>>
>> >>>
>> >>> On Wed, Jun 3, 2015 at 8:50 AM, David Brown <dbrown at ucar.edu> wrote:
>> >>> > Hi Dustin and Gerry,
>> >>> > I have confirmed there is a problem. It has been partially but not
>> >>> > completely fixed in the development version of the code. I am
>> looking
>> >>> > into the remaining issue and will let you know when it has been
>> fixed.
>> >>> >  -dave
>> >>> >
>> >>> > On Tue, Jun 2, 2015 at 10:34 AM, Gerry Creager - NOAA Affiliate
>> >>> > <gerry.creager at noaa.gov> wrote:
>> >>> >> Additional information: If we do an addfile("<filename>","r"), we
>> can
>> >>> >> open
>> >>> >> and read the file. If we use "w", well, we can open the file but
>> >>> >> assigning
>> >>> >> an existing variable to an internal variable fails on the assign
>> (var =
>> >>> >> a->QRAIN). Of course, because it fails there, we can't see if it'd
>> also
>> >>> >> fail
>> >>> >> on write.
>> >>> >>
>> >>> >> Thanks
>> >>> >> Gerry
>> >>> >>
>> >>> >> On Mon, Jun 1, 2015 at 7:15 PM, Gerry Creager - NOAA Affiliate
>> >>> >> <gerry.creager at noaa.gov> wrote:
>> >>> >>>
>> >>> >>> I've started looking at this with Dusty. Appreciate any thought
>> you
>> >>> >>> might
>> >>> >>> have as to why we're failing.
>> >>> >>>
>> >>> >>> I've uploaded the input file (wrfinput_d02) to the FTP incoming
>> site.
>> >>> >>> I'm
>> >>> >>> testing with the script:
>> >>> >>>
>> >>> >>> var = "QRAIN"
>> >>> >>> print(var)
>> >>> >>> a=addfile("wrfinput_d02.nc", "w")
>> >>> >>> var_orig=a->QRAIN
>> >>> >>> print(var_orig)
>> >>> >>>
>> >>> >>>
>> >>> >>> The result looks like:gerry at loki:~/NEWSe/NCL> ncl test.ncl
>> >>> >>>  Copyright (C) 1995-2015 - All Rights Reserved
>> >>> >>>  University Corporation for Atmospheric Research
>> >>> >>>  NCAR Command Language Version 6.3.0
>> >>> >>>  The use of this software is governed by a License Agreement.
>> >>> >>>  See http://www.ncl.ucar.edu/ for more details.
>> >>> >>>
>> >>> >>>
>> >>> >>> Variable: var
>> >>> >>> Type: string
>> >>> >>> Total Size: 8 bytes
>> >>> >>>             1 values
>> >>> >>> Number of Dimensions: 1
>> >>> >>> Dimensions and sizes: [1]
>> >>> >>> Coordinates:
>> >>> >>> (0) QRAIN
>> >>> >>> fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in
>> nc_def_var_chunking
>> >>> >>> in
>> >>> >>> file (wrfinput_d02.nc) for writing, at line: 3073
>> >>> >>>
>> >>> >>> line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var
>> properties,
>> >>> >>> like
>> >>> >>> deflate, after enddef.
>> >>> >>>
>> >>> >>>
>> >>> >>> On Fri, May 29, 2015 at 8:54 PM, Dustan Wheatley - NOAA Affiliate
>> >>> >>> <dustan.wheatley at noaa.gov> wrote:
>> >>> >>>>
>> >>> >>>> I am trying to overwrite a netcdf4 file analogous to what is
>> shown at
>> >>> >>>> http://www.ncl.ucar.edu/Applications/change_netCDF.shtml.  In
>> the
>> >>> >>>> attached
>> >>> >>>> ncl script, I open the file to be modified at line 35 and
>> attempt the
>> >>> >>>> modification at line 73 (with a lot of calculations in between).
>> >>> >>>> While I
>> >>> >>>> have double-checked that the modified field (var_mod) has the
>> same
>> >>> >>>> dimensionality as the original variable (QRAIN), I am still get a
>> >>> >>>> segmentation fault/core dump.  Never had this problem when I
>> tried to
>> >>> >>>> manipulate netcdf3 files.  Could it possibly be related to the
>> >>> >>>> netcdf4
>> >>> >>>> compression?  It’s possible I may be trying to do something
>> that’s no
>> >>> >>>> longer
>> >>> >>>> possible.
>> >>> >>>>
>> >>> >>>> Thanks in advance for any suggestions.
>> >>> >>>>
>> >>> >>>> -Dusty Wheatley
>> >>> >>>>
>> >>> >>>> _______________________________________________
>> >>> >>>> ncl-talk mailing list
>> >>> >>>> ncl-talk at ucar.edu
>> >>> >>>> List instructions, subscriber options, unsubscribe:
>> >>> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >>> >>>>
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> --
>> >>> >>> Gerry Creager
>> >>> >>> NSSL/CIMMS
>> >>> >>> 405.325.6371
>> >>> >>> ++++++++++++++++++++++
>> >>> >>> “Big whorls have little whorls,
>> >>> >>> That feed on their velocity;
>> >>> >>> And little whorls have lesser whorls,
>> >>> >>> And so on to viscosity.”
>> >>> >>> Lewis Fry Richardson (1881-1953)
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Gerry Creager
>> >>> >> NSSL/CIMMS
>> >>> >> 405.325.6371
>> >>> >> ++++++++++++++++++++++
>> >>> >> “Big whorls have little whorls,
>> >>> >> That feed on their velocity;
>> >>> >> And little whorls have lesser whorls,
>> >>> >> And so on to viscosity.”
>> >>> >> Lewis Fry Richardson (1881-1953)
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> ncl-talk mailing list
>> >>> >> ncl-talk at ucar.edu
>> >>> >> List instructions, subscriber options, unsubscribe:
>> >>> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Gerry Creager
>> >> NSSL/CIMMS
>> >> 405.325.6371
>> >> ++++++++++++++++++++++
>> >> “Big whorls have little whorls,
>> >> That feed on their velocity;
>> >> And little whorls have lesser whorls,
>> >> And so on to viscosity.”
>> >> Lewis Fry Richardson (1881-1953)
>>
>
>
>
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
>



-- 
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150623/f46ad66e/attachment.html

More information about the ncl-talk mailing list