[ncl-talk] Bug: _FillValue attribute erases previous data

[David Brown]

As Rick recounts above, we thought that the NetCDF4 library went
through a lot of hoops in 4.4.1 to do just as you suggest Dave,
totally rewriting the variable, just to allow changing the _FillValue,
as was possible in NetCDF3.

However, it seems that, at least as far back as 4.1.3 (~2011) the
documentation (nc_def_var_fill) has explicitly stated that the
_FillValue cannot be changed once the data variable has been read or
written.

I knew this about HDF5, but until now I thought that NetCDF4 had
overcome this limitation. Given the situation, it appears you are
correct that the only correct response is to refuse to try to change
the _FillValue and return an error, although actually it seems that it
should really be the responsibility of the NetCDF library to give us
the ELATEFILL error message and not allow the data to be modified.
 -dave






On Tue, Mar 28, 2017 at 6:09 PM, Dave Allured - NOAA Affiliate
<dave.allured at noaa.gov> wrote:
> Rick,
>
> I discovered that my current complaint is all due to the underlying Netcdf
> library, currently version 4.4.1.  I would not have sent an NCL bug report
> if I had realized this earlier.  I do intend to report this back to Netcdf.
>
> In Netcdf-4 file format, there is apparently a restriction that _FillValue
> may not be changed after data is written.  This is a notable difference from
> Netcdf-3 format.  Functions in the nc_put_att family list this possible
> error, which I think applies to Netcdf-4 file access only:
>
>     NC_ELATEFILL -- Fill values must be written while the file is still in
> initial define mode.
>
> This is probably due to the underlying HDF5 layer:
>
>     "the fill value for a dataset cannot be modified once the dataset has
> been created"
>     https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue
>
> This is why I think the correct response for both NCL and C programs should
> be to fail with NC_ELATEFILL or equivalent, rather than "just work".  To
> "just work" might involve a large performance penalty along the lines of
> scanning or re-writing the entire file variable.  Either way, previous data
> should not be erased, i.e. set to _FillValue.
>
> --Dave
>
>
> On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>
>> Hi Dave,
>>
>> I will look into this and likely file a bug report. At one point, there
>> was an issue whereby if _FillValue (or any attribute) was added to a
>> variable, it caused all the other attributes to be dropped (but the data was
>> preserved). This was traced to a bug in the underlying NetCDF library, which
>> was fixed in version 4.4.1 around summer 2016.  The NetCDF library was going
>> to a lot of effort to make it possible to rewrite a preexisting variable, so
>> I would think the correct response is for it to "just work", rather than
>> error out and do nothing?
>>
>> Rick
>>
>> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate
>> <dave.allured at noaa.gov> wrote:
>>>
>>> NCL team,
>>>
>>> Under some conditions, adding a _FillValue attribute to an existing
>>> Netcdf-4 file erases previous data.  I find this behavior with both NCL
>>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan).  I get the same behavior with
>>> both netcdf4-classic and full netcdf4 formats.
>>>
>>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s test3.nc
>>> netcdf test3 {
>>> dimensions:
>>> dim = 10 ;
>>> variables:
>>> int var(dim) ;
>>> var:_Storage = "contiguous" ;
>>> var:_Endianness = "little" ;
>>>
>>> // global attributes:
>>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>>> ;
>>> :_SuperblockVersion = 0 ;
>>> :_IsNetcdf4 = 1 ;
>>> :_Format = "netCDF-4 classic model" ;
>>> data:
>>>
>>>  var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
>>> }
>>>
>>> mac56:~/ncl/bugs/fillvalue1 3> ncl
>>>  Copyright (C) 1995-2017 - All Rights Reserved
>>>  University Corporation for Atmospheric Research
>>>  NCAR Command Language Version 6.4.0
>>>  The use of this software is governed by a License Agreement.
>>>  See http://www.ncl.ucar.edu/ for more details.
>>> ncl 0> f=addfile("test3.nc","w")
>>> ncl 1> f->var at _FillValue = 99
>>> ncl 2> exit
>>>
>>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s test3.nc
>>> netcdf test3 {
>>> dimensions:
>>> dim = 10 ;
>>> variables:
>>> int var(dim) ;
>>> var:_FillValue = 99 ;
>>> var:_Storage = "contiguous" ;
>>> var:_Endianness = "little" ;
>>>
>>> // global attributes:
>>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>>> ;
>>> :_SuperblockVersion = 0 ;
>>> :_IsNetcdf4 = 1 ;
>>> :_Format = "netCDF-4 classic model" ;
>>> data:
>>>
>>>  var = _, _, _, _, _, _, _, _, _, _ ;
>>> }
>>>
>>> In my opinion, the correct response for netcdf-4 formats should be an
>>> error, something like "Can not change _FillValue after data is written in
>>> netcdf-4", and the file is unchanged.  In any case, previous data should not
>>> be silently erased.  If you agree, please fix this for the next release.
>>> Thanks.
>>>
>>> --Dave
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>