[ncl-talk] Bug: _FillValue attribute erases previous data

[David Brown]

Thanks for the info Dave. Waiting is probably the most prudent option
at this point.
 -dave

On Fri, Apr 7, 2017 at 2:02 PM, Dave Allured - NOAA Affiliate
<dave.allured at noaa.gov> wrote:
> There is a netcdf-C Github ticket for this issue:
> https://github.com/Unidata/netcdf-c/issues/390
>
> I would like to suggest that NCL should wait and follow whatever fix is made
> in the netcdf-C API.  I could see that going either of two valid ways:
> either correct support for changing _FillValue in netcdf-4 format, or
> reverting to firm prohibition.  The core HDF5 library has a firm
> prohibition, and there are some good reasons for that.
>
> --Dave A.
>
>
> On Fri, Apr 7, 2017 at 11:31 AM, David Brown <dbrown at ucar.edu> wrote:
>>
>> A JIRA ticket NCL-2589 has been created for this issue with major
>> priority to be fixed for 6.4.1.
>>
>> On Tue, Mar 28, 2017 at 10:29 PM, David Brown <dbrown at ucar.edu> wrote:
>> > As Rick recounts above, we thought that the NetCDF4 library went
>> > through a lot of hoops in 4.4.1 to do just as you suggest Dave,
>> > totally rewriting the variable, just to allow changing the _FillValue,
>> > as was possible in NetCDF3.
>> >
>> > However, it seems that, at least as far back as 4.1.3 (~2011) the
>> > documentation (nc_def_var_fill) has explicitly stated that the
>> > _FillValue cannot be changed once the data variable has been read or
>> > written.
>> >
>> > I knew this about HDF5, but until now I thought that NetCDF4 had
>> > overcome this limitation. Given the situation, it appears you are
>> > correct that the only correct response is to refuse to try to change
>> > the _FillValue and return an error, although actually it seems that it
>> > should really be the responsibility of the NetCDF library to give us
>> > the ELATEFILL error message and not allow the data to be modified.
>> >  -dave
>> >
>> >
>> > On Tue, Mar 28, 2017 at 6:09 PM, Dave Allured - NOAA Affiliate
>> > <dave.allured at noaa.gov> wrote:
>> >> Rick,
>> >>
>> >> I discovered that my current complaint is all due to the underlying
>> >> Netcdf
>> >> library, currently version 4.4.1.  I would not have sent an NCL bug
>> >> report
>> >> if I had realized this earlier.  I do intend to report this back to
>> >> Netcdf.
>> >>
>> >> In Netcdf-4 file format, there is apparently a restriction that
>> >> _FillValue
>> >> may not be changed after data is written.  This is a notable difference
>> >> from
>> >> Netcdf-3 format.  Functions in the nc_put_att family list this possible
>> >> error, which I think applies to Netcdf-4 file access only:
>> >>
>> >>     NC_ELATEFILL -- Fill values must be written while the file is still
>> >> in
>> >> initial define mode.
>> >>
>> >> This is probably due to the underlying HDF5 layer:
>> >>
>> >>     "the fill value for a dataset cannot be modified once the dataset
>> >> has
>> >> been created"
>> >>     https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue
>> >>
>> >> This is why I think the correct response for both NCL and C programs
>> >> should
>> >> be to fail with NC_ELATEFILL or equivalent, rather than "just work".
>> >> To
>> >> "just work" might involve a large performance penalty along the lines
>> >> of
>> >> scanning or re-writing the entire file variable.  Either way, previous
>> >> data
>> >> should not be erased, i.e. set to _FillValue.
>> >>
>> >> --Dave
>> >>
>> >>
>> >> On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <brownrig at ucar.edu>
>> >> wrote:
>> >>>
>> >>> Hi Dave,
>> >>>
>> >>> I will look into this and likely file a bug report. At one point,
>> >>> there
>> >>> was an issue whereby if _FillValue (or any attribute) was added to a
>> >>> variable, it caused all the other attributes to be dropped (but the
>> >>> data was
>> >>> preserved). This was traced to a bug in the underlying NetCDF library,
>> >>> which
>> >>> was fixed in version 4.4.1 around summer 2016.  The NetCDF library was
>> >>> going
>> >>> to a lot of effort to make it possible to rewrite a preexisting
>> >>> variable, so
>> >>> I would think the correct response is for it to "just work", rather
>> >>> than
>> >>> error out and do nothing?
>> >>>
>> >>> Rick
>> >>>
>> >>> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate
>> >>> <dave.allured at noaa.gov> wrote:
>> >>>>
>> >>>> NCL team,
>> >>>>
>> >>>> Under some conditions, adding a _FillValue attribute to an existing
>> >>>> Netcdf-4 file erases previous data.  I find this behavior with both
>> >>>> NCL
>> >>>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan).  I get the same
>> >>>> behavior with
>> >>>> both netcdf4-classic and full netcdf4 formats.
>> >>>>
>> >>>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s test3.nc
>> >>>> netcdf test3 {
>> >>>> dimensions:
>> >>>> dim = 10 ;
>> >>>> variables:
>> >>>> int var(dim) ;
>> >>>> var:_Storage = "contiguous" ;
>> >>>> var:_Endianness = "little" ;
>> >>>>
>> >>>> // global attributes:
>> >>>> :_NCProperties =
>> >>>> "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>> >>>> ;
>> >>>> :_SuperblockVersion = 0 ;
>> >>>> :_IsNetcdf4 = 1 ;
>> >>>> :_Format = "netCDF-4 classic model" ;
>> >>>> data:
>> >>>>
>> >>>>  var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
>> >>>> }
>> >>>>
>> >>>> mac56:~/ncl/bugs/fillvalue1 3> ncl
>> >>>>  Copyright (C) 1995-2017 - All Rights Reserved
>> >>>>  University Corporation for Atmospheric Research
>> >>>>  NCAR Command Language Version 6.4.0
>> >>>>  The use of this software is governed by a License Agreement.
>> >>>>  See http://www.ncl.ucar.edu/ for more details.
>> >>>> ncl 0> f=addfile("test3.nc","w")
>> >>>> ncl 1> f->var at _FillValue = 99
>> >>>> ncl 2> exit
>> >>>>
>> >>>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s test3.nc
>> >>>> netcdf test3 {
>> >>>> dimensions:
>> >>>> dim = 10 ;
>> >>>> variables:
>> >>>> int var(dim) ;
>> >>>> var:_FillValue = 99 ;
>> >>>> var:_Storage = "contiguous" ;
>> >>>> var:_Endianness = "little" ;
>> >>>>
>> >>>> // global attributes:
>> >>>> :_NCProperties =
>> >>>> "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>> >>>> ;
>> >>>> :_SuperblockVersion = 0 ;
>> >>>> :_IsNetcdf4 = 1 ;
>> >>>> :_Format = "netCDF-4 classic model" ;
>> >>>> data:
>> >>>>
>> >>>>  var = _, _, _, _, _, _, _, _, _, _ ;
>> >>>> }
>> >>>>
>> >>>> In my opinion, the correct response for netcdf-4 formats should be an
>> >>>> error, something like "Can not change _FillValue after data is
>> >>>> written in
>> >>>> netcdf-4", and the file is unchanged.  In any case, previous data
>> >>>> should not
>> >>>> be silently erased.  If you agree, please fix this for the next
>> >>>> release.
>> >>>> Thanks.
>> >>>>
>> >>>> --Dave
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>