[ncl-talk] Bug: _FillValue attribute erases previous data

[Dave Allured - NOAA Affiliate]

There is a netcdf-C Github ticket for this issue:
https://github.com/Unidata/netcdf-c/issues/390

I would like to suggest that NCL should wait and follow whatever fix is
made in the netcdf-C API.  I could see that going either of two valid ways:
either correct support for changing _FillValue in netcdf-4 format, or
reverting to firm prohibition.  The core HDF5 library has a firm
prohibition, and there are some good reasons for that.

--Dave A.


On Fri, Apr 7, 2017 at 11:31 AM, David Brown <dbrown at ucar.edu> wrote:

> A JIRA ticket NCL-2589 has been created for this issue with major
> priority to be fixed for 6.4.1.
>
> On Tue, Mar 28, 2017 at 10:29 PM, David Brown <dbrown at ucar.edu> wrote:
> > As Rick recounts above, we thought that the NetCDF4 library went
> > through a lot of hoops in 4.4.1 to do just as you suggest Dave,
> > totally rewriting the variable, just to allow changing the _FillValue,
> > as was possible in NetCDF3.
> >
> > However, it seems that, at least as far back as 4.1.3 (~2011) the
> > documentation (nc_def_var_fill) has explicitly stated that the
> > _FillValue cannot be changed once the data variable has been read or
> > written.
> >
> > I knew this about HDF5, but until now I thought that NetCDF4 had
> > overcome this limitation. Given the situation, it appears you are
> > correct that the only correct response is to refuse to try to change
> > the _FillValue and return an error, although actually it seems that it
> > should really be the responsibility of the NetCDF library to give us
> > the ELATEFILL error message and not allow the data to be modified.
> >  -dave
> >
> >
> > On Tue, Mar 28, 2017 at 6:09 PM, Dave Allured - NOAA Affiliate
> > <dave.allured at noaa.gov> wrote:
> >> Rick,
> >>
> >> I discovered that my current complaint is all due to the underlying
> Netcdf
> >> library, currently version 4.4.1.  I would not have sent an NCL bug
> report
> >> if I had realized this earlier.  I do intend to report this back to
> Netcdf.
> >>
> >> In Netcdf-4 file format, there is apparently a restriction that
> _FillValue
> >> may not be changed after data is written.  This is a notable difference
> from
> >> Netcdf-3 format.  Functions in the nc_put_att family list this possible
> >> error, which I think applies to Netcdf-4 file access only:
> >>
> >>     NC_ELATEFILL -- Fill values must be written while the file is still
> in
> >> initial define mode.
> >>
> >> This is probably due to the underlying HDF5 layer:
> >>
> >>     "the fill value for a dataset cannot be modified once the dataset
> has
> >> been created"
> >>     https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue
> >>
> >> This is why I think the correct response for both NCL and C programs
> should
> >> be to fail with NC_ELATEFILL or equivalent, rather than "just work".  To
> >> "just work" might involve a large performance penalty along the lines of
> >> scanning or re-writing the entire file variable.  Either way, previous
> data
> >> should not be erased, i.e. set to _FillValue.
> >>
> >> --Dave
> >>
> >>
> >> On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <brownrig at ucar.edu>
> wrote:
> >>>
> >>> Hi Dave,
> >>>
> >>> I will look into this and likely file a bug report. At one point, there
> >>> was an issue whereby if _FillValue (or any attribute) was added to a
> >>> variable, it caused all the other attributes to be dropped (but the
> data was
> >>> preserved). This was traced to a bug in the underlying NetCDF library,
> which
> >>> was fixed in version 4.4.1 around summer 2016.  The NetCDF library was
> going
> >>> to a lot of effort to make it possible to rewrite a preexisting
> variable, so
> >>> I would think the correct response is for it to "just work", rather
> than
> >>> error out and do nothing?
> >>>
> >>> Rick
> >>>
> >>> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate
> >>> <dave.allured at noaa.gov> wrote:
> >>>>
> >>>> NCL team,
> >>>>
> >>>> Under some conditions, adding a _FillValue attribute to an existing
> >>>> Netcdf-4 file erases previous data.  I find this behavior with both
> NCL
> >>>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan).  I get the same
> behavior with
> >>>> both netcdf4-classic and full netcdf4 formats.
> >>>>
> >>>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s test3.nc
> >>>> netcdf test3 {
> >>>> dimensions:
> >>>> dim = 10 ;
> >>>> variables:
> >>>> int var(dim) ;
> >>>> var:_Storage = "contiguous" ;
> >>>> var:_Endianness = "little" ;
> >>>>
> >>>> // global attributes:
> >>>> :_NCProperties = "version=1|netcdflibversion=4.
> 4.1|hdf5libversion=1.8.17"
> >>>> ;
> >>>> :_SuperblockVersion = 0 ;
> >>>> :_IsNetcdf4 = 1 ;
> >>>> :_Format = "netCDF-4 classic model" ;
> >>>> data:
> >>>>
> >>>>  var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
> >>>> }
> >>>>
> >>>> mac56:~/ncl/bugs/fillvalue1 3> ncl
> >>>>  Copyright (C) 1995-2017 - All Rights Reserved
> >>>>  University Corporation for Atmospheric Research
> >>>>  NCAR Command Language Version 6.4.0
> >>>>  The use of this software is governed by a License Agreement.
> >>>>  See http://www.ncl.ucar.edu/ for more details.
> >>>> ncl 0> f=addfile("test3.nc","w")
> >>>> ncl 1> f->var at _FillValue = 99
> >>>> ncl 2> exit
> >>>>
> >>>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s test3.nc
> >>>> netcdf test3 {
> >>>> dimensions:
> >>>> dim = 10 ;
> >>>> variables:
> >>>> int var(dim) ;
> >>>> var:_FillValue = 99 ;
> >>>> var:_Storage = "contiguous" ;
> >>>> var:_Endianness = "little" ;
> >>>>
> >>>> // global attributes:
> >>>> :_NCProperties = "version=1|netcdflibversion=4.
> 4.1|hdf5libversion=1.8.17"
> >>>> ;
> >>>> :_SuperblockVersion = 0 ;
> >>>> :_IsNetcdf4 = 1 ;
> >>>> :_Format = "netCDF-4 classic model" ;
> >>>> data:
> >>>>
> >>>>  var = _, _, _, _, _, _, _, _, _, _ ;
> >>>> }
> >>>>
> >>>> In my opinion, the correct response for netcdf-4 formats should be an
> >>>> error, something like "Can not change _FillValue after data is
> written in
> >>>> netcdf-4", and the file is unchanged.  In any case, previous data
> should not
> >>>> be silently erased.  If you agree, please fix this for the next
> release.
> >>>> Thanks.
> >>>>
> >>>> --Dave
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170407/181a947d/attachment.html