[ncl-talk] Editing a netcdf4 file

Tue Jun 23 09:44:59 MDT 2015

Never mind. Silly user error. It works fine with the right code package.

gerry

On Tue, Jun 23, 2015 at 3:37 PM, Gerry Creager - NOAA Affiliate <
gerry.creager at noaa.gov> wrote:

> Dave
>
> I modified the script and...
>
> [gerry at wof-vis NCL]$ ncl test2.ncl
>  Copyright (C) 1995-2015 - All Rights Reserved
>  University Corporation for Atmospheric Research
>  NCAR Command Language Version 6.3.0
>  The use of this software is governed by a License Agreement.
>  See http://www.ncl.ucar.edu/ for more details.
> fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in nc_def_var_chunking in
> file (wrfinput_d02.nc) for writing, at line: 3073
>
> line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var properties, like
> deflate, after enddef.
>
> Where to, now?
>
> On Tue, Jun 23, 2015 at 3:35 PM, Gerry Creager - NOAA Affiliate <
> gerry.creager at noaa.gov> wrote:
>
>> Dave
>>
>> Thanks. And I'll bring this up to Russ and the crew at Unidata!
>>
>> We'll test.
>>
>> Gerry
>>
>> On Tue, Jun 23, 2015 at 12:49 AM, David Brown <dbrown at ucar.edu> wrote:
>>
>>> Hi Gerry,
>>> I have a work-around fix for you, which is to change the line:
>>>
>>> var_mod = new((/1,dimz,dimy,dimx/),"float")
>>> to
>>> var_mod = new((/1,dimz,dimy,dimx/),"float","No_FillValue")
>>>
>>> Alternately you could say
>>> delete(var_mod at _FillValue)
>>> after the 'new' statement.
>>>
>>> The issue is that NCL creates _FillValue attributes for variables by
>>> default, and these get propagated when the variable is written to a
>>> file. And while coordinates and most attributes are not passed along
>>> when you do a value-only assignment (using (/ .../) syntax), the
>>> _FillValue is. This was never a problem with traditional NetCDF.
>>>
>>> But in NetCDF4 it seems the _FillValue attribute can only be written
>>> when "the file is still in initial define mode"  (according to
>>>
>>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fput_005fatt_005f-type.html#nc_005fput_005fatt_005f-type
>>> ).
>>>
>>> I think this means that _FillValue attributes cannot be added to
>>> variables that did not have them originally. We will need to account
>>> for this in NCL, but in the meantime you can make things work by
>>> eliminating the _FillValue attribute from the variable you are using
>>> to replace the original QRAIN variable in the file.
>>>
>>> Hope this helps.
>>>  -dave
>>>
>>>
>>>
>>>
>>> On Mon, Jun 22, 2015 at 5:41 PM, David Brown <dbrown at ucar.edu> wrote:
>>> > Hi Gerry,
>>> > I have been able to duplicate this seg fault and I am looking into it
>>> now.
>>> >  -dave
>>> >
>>> > On Mon, Jun 22, 2015 at 4:14 PM, Gerry Creager - NOAA Affiliate
>>> > <gerry.creager at noaa.gov> wrote:
>>> >> David
>>> >>
>>> >> Looks like it may not have been fixed quite yet. Mary sent me a
>>> version and
>>> >> I finally fooled the system into working today, when testing began.
>>> The
>>> >> short script included, inline below, can be run against the netcdf
>>> (wrfout)
>>> >> I uploaded recently (advise if I need to upload it again), results in
>>> a seg
>>> >> fault.
>>> >>
>>> >>
>>> >> ;=============
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>>> >> ;=============
>>> >> begin
>>> >>
>>> >> dimx = 225
>>> >> dimy = 225
>>> >> dimz = 55
>>> >>
>>> >> a = addfile("wrfinput_d02.nc", "w")
>>> >> var_orig=a->QRAIN
>>> >> var_mod = new((/1,dimz,dimy,dimx/),"float")
>>> >> var_mod = 5.
>>> >>
>>> >> print(dimsizes(var_orig))
>>> >> print(dimsizes(var_mod))
>>> >>
>>> >> a->QRAIN = (/var_mod/)
>>> >>
>>> >> end
>>> >>
>>> >>
>>> >> On Mon, Jun 8, 2015 at 9:00 PM, David Brown <dbrown at ucar.edu> wrote:
>>> >>>
>>> >>> Dustan and Gerry,
>>> >>> This issue relating to writing a variable with chunking in an
>>> existing
>>> >>> NetCDF4 file has been fixed. Also a secondary issue involving
>>> >>> re-writing existing variable attributes has been addressed as well.
>>> >>> Let us know if you need a version with these fixes and we will try to
>>> >>> accommodate you as soon as possible.
>>> >>> Thanks for letting us know of this problem.
>>> >>>  -dave
>>> >>>
>>> >>>
>>> >>> On Wed, Jun 3, 2015 at 8:50 AM, David Brown <dbrown at ucar.edu> wrote:
>>> >>> > Hi Dustin and Gerry,
>>> >>> > I have confirmed there is a problem. It has been partially but not
>>> >>> > completely fixed in the development version of the code. I am
>>> looking
>>> >>> > into the remaining issue and will let you know when it has been
>>> fixed.
>>> >>> >  -dave
>>> >>> >
>>> >>> > On Tue, Jun 2, 2015 at 10:34 AM, Gerry Creager - NOAA Affiliate
>>> >>> > <gerry.creager at noaa.gov> wrote:
>>> >>> >> Additional information: If we do an addfile("<filename>","r"), we
>>> can
>>> >>> >> open
>>> >>> >> and read the file. If we use "w", well, we can open the file but
>>> >>> >> assigning
>>> >>> >> an existing variable to an internal variable fails on the assign
>>> (var =
>>> >>> >> a->QRAIN). Of course, because it fails there, we can't see if
>>> it'd also
>>> >>> >> fail
>>> >>> >> on write.
>>> >>> >>
>>> >>> >> Thanks
>>> >>> >> Gerry
>>> >>> >>
>>> >>> >> On Mon, Jun 1, 2015 at 7:15 PM, Gerry Creager - NOAA Affiliate
>>> >>> >> <gerry.creager at noaa.gov> wrote:
>>> >>> >>>
>>> >>> >>> I've started looking at this with Dusty. Appreciate any thought
>>> you
>>> >>> >>> might
>>> >>> >>> have as to why we're failing.
>>> >>> >>>
>>> >>> >>> I've uploaded the input file (wrfinput_d02) to the FTP incoming
>>> site.
>>> >>> >>> I'm
>>> >>> >>> testing with the script:
>>> >>> >>>
>>> >>> >>> var = "QRAIN"
>>> >>> >>> print(var)
>>> >>> >>> a=addfile("wrfinput_d02.nc", "w")
>>> >>> >>> var_orig=a->QRAIN
>>> >>> >>> print(var_orig)
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> The result looks like:gerry at loki:~/NEWSe/NCL> ncl test.ncl
>>> >>> >>>  Copyright (C) 1995-2015 - All Rights Reserved
>>> >>> >>>  University Corporation for Atmospheric Research
>>> >>> >>>  NCAR Command Language Version 6.3.0
>>> >>> >>>  The use of this software is governed by a License Agreement.
>>> >>> >>>  See http://www.ncl.ucar.edu/ for more details.
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Variable: var
>>> >>> >>> Type: string
>>> >>> >>> Total Size: 8 bytes
>>> >>> >>>             1 values
>>> >>> >>> Number of Dimensions: 1
>>> >>> >>> Dimensions and sizes: [1]
>>> >>> >>> Coordinates:
>>> >>> >>> (0) QRAIN
>>> >>> >>> fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in
>>> nc_def_var_chunking
>>> >>> >>> in
>>> >>> >>> file (wrfinput_d02.nc) for writing, at line: 3073
>>> >>> >>>
>>> >>> >>> line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var
>>> properties,
>>> >>> >>> like
>>> >>> >>> deflate, after enddef.
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> On Fri, May 29, 2015 at 8:54 PM, Dustan Wheatley - NOAA Affiliate
>>> >>> >>> <dustan.wheatley at noaa.gov> wrote:
>>> >>> >>>>
>>> >>> >>>> I am trying to overwrite a netcdf4 file analogous to what is
>>> shown at
>>> >>> >>>> http://www.ncl.ucar.edu/Applications/change_netCDF.shtml.  In
>>> the
>>> >>> >>>> attached
>>> >>> >>>> ncl script, I open the file to be modified at line 35 and
>>> attempt the
>>> >>> >>>> modification at line 73 (with a lot of calculations in between).
>>> >>> >>>> While I
>>> >>> >>>> have double-checked that the modified field (var_mod) has the
>>> same
>>> >>> >>>> dimensionality as the original variable (QRAIN), I am still get
>>> a
>>> >>> >>>> segmentation fault/core dump.  Never had this problem when I
>>> tried to
>>> >>> >>>> manipulate netcdf3 files.  Could it possibly be related to the
>>> >>> >>>> netcdf4
>>> >>> >>>> compression?  It’s possible I may be trying to do something
>>> that’s no
>>> >>> >>>> longer
>>> >>> >>>> possible.
>>> >>> >>>>
>>> >>> >>>> Thanks in advance for any suggestions.
>>> >>> >>>>
>>> >>> >>>> -Dusty Wheatley
>>> >>> >>>>
>>> >>> >>>> _______________________________________________
>>> >>> >>>> ncl-talk mailing list
>>> >>> >>>> ncl-talk at ucar.edu
>>> >>> >>>> List instructions, subscriber options, unsubscribe:
>>> >>> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >>> >>>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> --
>>> >>> >>> Gerry Creager
>>> >>> >>> NSSL/CIMMS
>>> >>> >>> 405.325.6371
>>> >>> >>> ++++++++++++++++++++++
>>> >>> >>> “Big whorls have little whorls,
>>> >>> >>> That feed on their velocity;
>>> >>> >>> And little whorls have lesser whorls,
>>> >>> >>> And so on to viscosity.”
>>> >>> >>> Lewis Fry Richardson (1881-1953)
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> Gerry Creager
>>> >>> >> NSSL/CIMMS
>>> >>> >> 405.325.6371
>>> >>> >> ++++++++++++++++++++++
>>> >>> >> “Big whorls have little whorls,
>>> >>> >> That feed on their velocity;
>>> >>> >> And little whorls have lesser whorls,
>>> >>> >> And so on to viscosity.”
>>> >>> >> Lewis Fry Richardson (1881-1953)
>>> >>> >>
>>> >>> >> _______________________________________________
>>> >>> >> ncl-talk mailing list
>>> >>> >> ncl-talk at ucar.edu
>>> >>> >> List instructions, subscriber options, unsubscribe:
>>> >>> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Gerry Creager
>>> >> NSSL/CIMMS
>>> >> 405.325.6371
>>> >> ++++++++++++++++++++++
>>> >> “Big whorls have little whorls,
>>> >> That feed on their velocity;
>>> >> And little whorls have lesser whorls,
>>> >> And so on to viscosity.”
>>> >> Lewis Fry Richardson (1881-1953)
>>>
>>
>>
>>
>> --
>> Gerry Creager
>> NSSL/CIMMS
>> 405.325.6371
>> ++++++++++++++++++++++
>> “Big whorls have little whorls,
>> That feed on their velocity;
>> And little whorls have lesser whorls,
>> And so on to viscosity.”
>> Lewis Fry Richardson (1881-1953)
>>
>
>
>
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
>

-- 
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150623/36dab9de/attachment.html 