[ncl-talk] Editing a netcdf4 file

Tue Jun 23 09:35:01 MDT 2015

Dave

Thanks. And I'll bring this up to Russ and the crew at Unidata!

We'll test.

Gerry

On Tue, Jun 23, 2015 at 12:49 AM, David Brown <dbrown at ucar.edu> wrote:

> Hi Gerry,
> I have a work-around fix for you, which is to change the line:
>
> var_mod = new((/1,dimz,dimy,dimx/),"float")
> to
> var_mod = new((/1,dimz,dimy,dimx/),"float","No_FillValue")
>
> Alternately you could say
> delete(var_mod at _FillValue)
> after the 'new' statement.
>
> The issue is that NCL creates _FillValue attributes for variables by
> default, and these get propagated when the variable is written to a
> file. And while coordinates and most attributes are not passed along
> when you do a value-only assignment (using (/ .../) syntax), the
> _FillValue is. This was never a problem with traditional NetCDF.
>
> But in NetCDF4 it seems the _FillValue attribute can only be written
> when "the file is still in initial define mode"  (according to
>
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fput_005fatt_005f-type.html#nc_005fput_005fatt_005f-type
> ).
>
> I think this means that _FillValue attributes cannot be added to
> variables that did not have them originally. We will need to account
> for this in NCL, but in the meantime you can make things work by
> eliminating the _FillValue attribute from the variable you are using
> to replace the original QRAIN variable in the file.
>
> Hope this helps.
>  -dave
>
>
>
>
> On Mon, Jun 22, 2015 at 5:41 PM, David Brown <dbrown at ucar.edu> wrote:
> > Hi Gerry,
> > I have been able to duplicate this seg fault and I am looking into it
> now.
> >  -dave
> >
> > On Mon, Jun 22, 2015 at 4:14 PM, Gerry Creager - NOAA Affiliate
> > <gerry.creager at noaa.gov> wrote:
> >> David
> >>
> >> Looks like it may not have been fixed quite yet. Mary sent me a version
> and
> >> I finally fooled the system into working today, when testing began. The
> >> short script included, inline below, can be run against the netcdf
> (wrfout)
> >> I uploaded recently (advise if I need to upload it again), results in a
> seg
> >> fault.
> >>
> >>
> >> ;=============
> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> >> ;=============
> >> begin
> >>
> >> dimx = 225
> >> dimy = 225
> >> dimz = 55
> >>
> >> a = addfile("wrfinput_d02.nc", "w")
> >> var_orig=a->QRAIN
> >> var_mod = new((/1,dimz,dimy,dimx/),"float")
> >> var_mod = 5.
> >>
> >> print(dimsizes(var_orig))
> >> print(dimsizes(var_mod))
> >>
> >> a->QRAIN = (/var_mod/)
> >>
> >> end
> >>
> >>
> >> On Mon, Jun 8, 2015 at 9:00 PM, David Brown <dbrown at ucar.edu> wrote:
> >>>
> >>> Dustan and Gerry,
> >>> This issue relating to writing a variable with chunking in an existing
> >>> NetCDF4 file has been fixed. Also a secondary issue involving
> >>> re-writing existing variable attributes has been addressed as well.
> >>> Let us know if you need a version with these fixes and we will try to
> >>> accommodate you as soon as possible.
> >>> Thanks for letting us know of this problem.
> >>>  -dave
> >>>
> >>>
> >>> On Wed, Jun 3, 2015 at 8:50 AM, David Brown <dbrown at ucar.edu> wrote:
> >>> > Hi Dustin and Gerry,
> >>> > I have confirmed there is a problem. It has been partially but not
> >>> > completely fixed in the development version of the code. I am looking
> >>> > into the remaining issue and will let you know when it has been
> fixed.
> >>> >  -dave
> >>> >
> >>> > On Tue, Jun 2, 2015 at 10:34 AM, Gerry Creager - NOAA Affiliate
> >>> > <gerry.creager at noaa.gov> wrote:
> >>> >> Additional information: If we do an addfile("<filename>","r"), we
> can
> >>> >> open
> >>> >> and read the file. If we use "w", well, we can open the file but
> >>> >> assigning
> >>> >> an existing variable to an internal variable fails on the assign
> (var =
> >>> >> a->QRAIN). Of course, because it fails there, we can't see if it'd
> also
> >>> >> fail
> >>> >> on write.
> >>> >>
> >>> >> Thanks
> >>> >> Gerry
> >>> >>
> >>> >> On Mon, Jun 1, 2015 at 7:15 PM, Gerry Creager - NOAA Affiliate
> >>> >> <gerry.creager at noaa.gov> wrote:
> >>> >>>
> >>> >>> I've started looking at this with Dusty. Appreciate any thought you
> >>> >>> might
> >>> >>> have as to why we're failing.
> >>> >>>
> >>> >>> I've uploaded the input file (wrfinput_d02) to the FTP incoming
> site.
> >>> >>> I'm
> >>> >>> testing with the script:
> >>> >>>
> >>> >>> var = "QRAIN"
> >>> >>> print(var)
> >>> >>> a=addfile("wrfinput_d02.nc", "w")
> >>> >>> var_orig=a->QRAIN
> >>> >>> print(var_orig)
> >>> >>>
> >>> >>>
> >>> >>> The result looks like:gerry at loki:~/NEWSe/NCL> ncl test.ncl
> >>> >>>  Copyright (C) 1995-2015 - All Rights Reserved
> >>> >>>  University Corporation for Atmospheric Research
> >>> >>>  NCAR Command Language Version 6.3.0
> >>> >>>  The use of this software is governed by a License Agreement.
> >>> >>>  See http://www.ncl.ucar.edu/ for more details.
> >>> >>>
> >>> >>>
> >>> >>> Variable: var
> >>> >>> Type: string
> >>> >>> Total Size: 8 bytes
> >>> >>>             1 values
> >>> >>> Number of Dimensions: 1
> >>> >>> Dimensions and sizes: [1]
> >>> >>> Coordinates:
> >>> >>> (0) QRAIN
> >>> >>> fatal:["NclNetCDF4.c":3073]:NclNetCDF4.c: Error in
> nc_def_var_chunking
> >>> >>> in
> >>> >>> file (wrfinput_d02.nc) for writing, at line: 3073
> >>> >>>
> >>> >>> line 3074 of NclNetCDF4.c: NetCDF: Attempt to define var
> properties,
> >>> >>> like
> >>> >>> deflate, after enddef.
> >>> >>>
> >>> >>>
> >>> >>> On Fri, May 29, 2015 at 8:54 PM, Dustan Wheatley - NOAA Affiliate
> >>> >>> <dustan.wheatley at noaa.gov> wrote:
> >>> >>>>
> >>> >>>> I am trying to overwrite a netcdf4 file analogous to what is
> shown at
> >>> >>>> http://www.ncl.ucar.edu/Applications/change_netCDF.shtml.  In the
> >>> >>>> attached
> >>> >>>> ncl script, I open the file to be modified at line 35 and attempt
> the
> >>> >>>> modification at line 73 (with a lot of calculations in between).
> >>> >>>> While I
> >>> >>>> have double-checked that the modified field (var_mod) has the same
> >>> >>>> dimensionality as the original variable (QRAIN), I am still get a
> >>> >>>> segmentation fault/core dump.  Never had this problem when I
> tried to
> >>> >>>> manipulate netcdf3 files.  Could it possibly be related to the
> >>> >>>> netcdf4
> >>> >>>> compression?  It’s possible I may be trying to do something
> that’s no
> >>> >>>> longer
> >>> >>>> possible.
> >>> >>>>
> >>> >>>> Thanks in advance for any suggestions.
> >>> >>>>
> >>> >>>> -Dusty Wheatley
> >>> >>>>
> >>> >>>> _______________________________________________
> >>> >>>> ncl-talk mailing list
> >>> >>>> ncl-talk at ucar.edu
> >>> >>>> List instructions, subscriber options, unsubscribe:
> >>> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>> >>>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Gerry Creager
> >>> >>> NSSL/CIMMS
> >>> >>> 405.325.6371
> >>> >>> ++++++++++++++++++++++
> >>> >>> “Big whorls have little whorls,
> >>> >>> That feed on their velocity;
> >>> >>> And little whorls have lesser whorls,
> >>> >>> And so on to viscosity.”
> >>> >>> Lewis Fry Richardson (1881-1953)
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Gerry Creager
> >>> >> NSSL/CIMMS
> >>> >> 405.325.6371
> >>> >> ++++++++++++++++++++++
> >>> >> “Big whorls have little whorls,
> >>> >> That feed on their velocity;
> >>> >> And little whorls have lesser whorls,
> >>> >> And so on to viscosity.”
> >>> >> Lewis Fry Richardson (1881-1953)
> >>> >>
> >>> >> _______________________________________________
> >>> >> ncl-talk mailing list
> >>> >> ncl-talk at ucar.edu
> >>> >> List instructions, subscriber options, unsubscribe:
> >>> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>> >>
> >>
> >>
> >>
> >>
> >> --
> >> Gerry Creager
> >> NSSL/CIMMS
> >> 405.325.6371
> >> ++++++++++++++++++++++
> >> “Big whorls have little whorls,
> >> That feed on their velocity;
> >> And little whorls have lesser whorls,
> >> And so on to viscosity.”
> >> Lewis Fry Richardson (1881-1953)
>

-- 
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150623/5f78ef82/attachment.html 