[ncl-talk] Bug: _FillValue attribute erases previous data

[Dave Allured - NOAA Affiliate]

Rick,

I discovered that my current complaint is all due to the underlying Netcdf
library, currently version 4.4.1.  I would not have sent an NCL bug report
if I had realized this earlier.  I do intend to report this back to Netcdf.

In Netcdf-4 file format, there is apparently a restriction that _FillValue
may not be changed after data is written.  This is a notable difference
from Netcdf-3 format.  Functions in the nc_put_att family list this
possible error, which I think applies to Netcdf-4 file access only:

    NC_ELATEFILL -- Fill values must be written while the file is still in
initial define mode.

This is probably due to the underlying HDF5 layer:

    "the fill value for a dataset cannot be modified once the dataset has
been created"
    https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue

This is why I think the correct response for both NCL and C programs should
be to fail with NC_ELATEFILL or equivalent, rather than "just work".  To
"just work" might involve a large performance penalty along the lines of
scanning or re-writing the entire file variable.  Either way, previous data
should not be erased, i.e. set to _FillValue.

--Dave


On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:

> Hi Dave,
>
> I will look into this and likely file a bug report. At one point, there
> was an issue whereby if _FillValue (or any attribute) was added to a
> variable, it caused all the other attributes to be dropped (but the data
> was preserved). This was traced to a bug in the underlying NetCDF library,
> which was fixed in version 4.4.1 around summer 2016.  The NetCDF library
> was going to a lot of effort to make it possible to rewrite a preexisting
> variable, so I would think the correct response is for it to "just work",
> rather than error out and do nothing?
>
> Rick
>
> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> NCL team,
>>
>> Under some conditions, adding a _FillValue attribute to an existing
>> Netcdf-4 file erases previous data.  I find this behavior with both NCL
>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan).  I get the same behavior with
>> both netcdf4-classic and full netcdf4 formats.
>>
>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s test3.nc
>> netcdf test3 {
>> dimensions:
>> dim = 10 ;
>> variables:
>> int var(dim) ;
>> var:_Storage = "contiguous" ;
>> var:_Endianness = "little" ;
>>
>> // global attributes:
>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>> ;
>> :_SuperblockVersion = 0 ;
>> :_IsNetcdf4 = 1 ;
>> :_Format = "netCDF-4 classic model" ;
>> data:
>>
>>  var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
>> }
>>
>> mac56:~/ncl/bugs/fillvalue1 3> ncl
>>  Copyright (C) 1995-2017 - All Rights Reserved
>>  University Corporation for Atmospheric Research
>>  NCAR Command Language Version 6.4.0
>>  The use of this software is governed by a License Agreement.
>>  See http://www.ncl.ucar.edu/ for more details.
>> ncl 0> f=addfile("test3.nc","w")
>> ncl 1> f->var at _FillValue = 99
>> ncl 2> exit
>>
>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s test3.nc
>> netcdf test3 {
>> dimensions:
>> dim = 10 ;
>> variables:
>> int var(dim) ;
>> var:_FillValue = 99 ;
>> var:_Storage = "contiguous" ;
>> var:_Endianness = "little" ;
>>
>> // global attributes:
>> :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17"
>> ;
>> :_SuperblockVersion = 0 ;
>> :_IsNetcdf4 = 1 ;
>> :_Format = "netCDF-4 classic model" ;
>> data:
>>
>>  var = _, _, _, _, _, _, _, _, _, _ ;
>> }
>>
>> In my opinion, the correct response for netcdf-4 formats should be an
>> error, something like "Can not change _FillValue after data is written in
>> netcdf-4", and the file is unchanged.  In any case, previous data should
>> not be silently erased.  If you agree, please fix this for the next
>> release.  Thanks.
>>
> --Dave
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170328/89332ea4/attachment.html