[ncl-talk] Bug: _FillValue attribute erases previous data

[David Brown]

A JIRA ticket NCL-2589 has been created for this issue with major
priority to be fixed for 6.4.1.

On Tue, Mar 28, 2017 at 10:29 PM, David Brown <dbrown at ucar.edu> wrote:
> As Rick recounts above, we thought that the NetCDF4 library went
> through a lot of hoops in 4.4.1 to do just as you suggest Dave,
> totally rewriting the variable, just to allow changing the _FillValue,
> as was possible in NetCDF3.
>
> However, it seems that, at least as far back as 4.1.3 (~2011) the
> documentation (nc_def_var_fill) has explicitly stated that the
> _FillValue cannot be changed once the data variable has been read or
> written.
>
> I knew this about HDF5, but until now I thought that NetCDF4 had
> overcome this limitation. Given the situation, it appears you are
> correct that the only correct response is to refuse to try to change
> the _FillValue and return an error, although actually it seems that it
> should really be the responsibility of the NetCDF library to give us
> the ELATEFILL error message and not allow the data to be modified.
>  -dave
>
>
>
>
>
>
> On Tue, Mar 28, 2017 at 6:09 PM, Dave Allured - NOAA Affiliate
> <dave.allured at noaa.gov> wrote:
>> Rick,
>>
>> I discovered that my current complaint is all due to the underlying Netcdf
>> library, currently version 4.4.1.  I would not have sent an NCL bug report
>> if I had realized this earlier.  I do intend to report this back to Netcdf.
>>
>> In Netcdf-4 file format, there is apparently a restriction that _FillValue
>> may not be changed after data is written.  This is a notable difference from
>> Netcdf-3 format.  Functions in the nc_put_att family list this possible
>> error, which I think applies to Netcdf-4 file access only:
>>
>>     NC_ELATEFILL -- Fill values must be written while the file is still in
>> initial define mode.
>>
>> This is probably due to the underlying HDF5 layer:
>>
>>     "the fill value for a dataset cannot be modified once the dataset has
>> been created"
>>     https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue
>>
>> This is why I think the correct response for both NCL and C programs should
>> be to fail with NC_ELATEFILL or equivalent, rather than "just work".  To
>> "just work" might involve a large performance penalty along the lines of
>> scanning or re-writing the entire file variable.  Either way, previous data
>> should not be erased, i.e. set to _FillValue.
>>
>> --Dave
>>
>>
>> On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>>
>>> Hi Dave,
>>>
>>> I will look into this and likely file a bug report. At one point, there
>>> was an issue whereby if _FillValue (or any attribute) was added to a
>>> variable, it caused all the other attributes to be dropped (but the data was
>>> preserved). This was traced to a bug in the underlying NetCDF library, which
>>> was fixed in version 4.4.1 around summer 2016.  The NetCDF library was going
>>> to a lot of effort to make it possible to rewrite a preexisting variable, so
>>> I would think the correct response is for it to "just work", rather than
>>> error out and do nothing?
>>>
>>> Rick
>>>
>>> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate
>>> <dave.allured at noaa.gov> wrote:
>>>>
>>>> NCL team,
>>>>
>>>> Under some conditions, adding a _FillValue attribute to an existing
>>>> Netcdf-4 file erases previous data.  I find this behavior with both NCL
>>>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan).  I get the same behavior with
>>>> both netcdf4-classic and full netcdf4 formats.
>>>>
>>>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s test3.nc
>>>> netcdf test3 {
>>>> dimensions:
>>>> dim = 10 ;
>>>> variables:
>>>> int var(dim) ;
>>>> var:_Storage = "contiguous" ;
>>>> var:_Endianness = "little" ;
>>>>
>>>> // global attributes:
>>>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>>>> ;
>>>> :_SuperblockVersion = 0 ;
>>>> :_IsNetcdf4 = 1 ;
>>>> :_Format = "netCDF-4 classic model" ;
>>>> data:
>>>>
>>>>  var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
>>>> }
>>>>
>>>> mac56:~/ncl/bugs/fillvalue1 3> ncl
>>>>  Copyright (C) 1995-2017 - All Rights Reserved
>>>>  University Corporation for Atmospheric Research
>>>>  NCAR Command Language Version 6.4.0
>>>>  The use of this software is governed by a License Agreement.
>>>>  See http://www.ncl.ucar.edu/ for more details.
>>>> ncl 0> f=addfile("test3.nc","w")
>>>> ncl 1> f->var at _FillValue = 99
>>>> ncl 2> exit
>>>>
>>>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s test3.nc
>>>> netcdf test3 {
>>>> dimensions:
>>>> dim = 10 ;
>>>> variables:
>>>> int var(dim) ;
>>>> var:_FillValue = 99 ;
>>>> var:_Storage = "contiguous" ;
>>>> var:_Endianness = "little" ;
>>>>
>>>> // global attributes:
>>>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>>>> ;
>>>> :_SuperblockVersion = 0 ;
>>>> :_IsNetcdf4 = 1 ;
>>>> :_Format = "netCDF-4 classic model" ;
>>>> data:
>>>>
>>>>  var = _, _, _, _, _, _, _, _, _, _ ;
>>>> }
>>>>
>>>> In my opinion, the correct response for netcdf-4 formats should be an
>>>> error, something like "Can not change _FillValue after data is written in
>>>> netcdf-4", and the file is unchanged.  In any case, previous data should not
>>>> be silently erased.  If you agree, please fix this for the next release.
>>>> Thanks.
>>>>
>>>> --Dave
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>