<div dir="ltr">There is a netcdf-C Github ticket for this issue:<br><div><a href="https://github.com/Unidata/netcdf-c/issues/390">https://github.com/Unidata/netcdf-c/issues/390</a><br></div><div><br></div><div>I would like to suggest that NCL should wait and follow whatever fix is made in the netcdf-C API. I could see that going either of two valid ways: either correct support for changing _FillValue in netcdf-4 format, or reverting to firm prohibition. The core HDF5 library has a firm prohibition, and there are some good reasons for that.</div><div><br></div><div>--Dave A.</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 7, 2017 at 11:31 AM, David Brown <span dir="ltr"><<a href="mailto:dbrown@ucar.edu" target="_blank">dbrown@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">A JIRA ticket NCL-2589 has been created for this issue with major<br>
priority to be fixed for 6.4.1.<br>
<br>
On Tue, Mar 28, 2017 at 10:29 PM, David Brown <<a href="mailto:dbrown@ucar.edu">dbrown@ucar.edu</a>> wrote:<br>
> As Rick recounts above, we thought that the NetCDF4 library went<br>
> through a lot of hoops in 4.4.1 to do just as you suggest Dave,<br>
> totally rewriting the variable, just to allow changing the _FillValue,<br>
> as was possible in NetCDF3.<br>
><br>
> However, it seems that, at least as far back as 4.1.3 (~2011) the<br>
> documentation (nc_def_var_fill) has explicitly stated that the<br>
> _FillValue cannot be changed once the data variable has been read or<br>
> written.<br>
><br>
> I knew this about HDF5, but until now I thought that NetCDF4 had<br>
> overcome this limitation. Given the situation, it appears you are<br>
> correct that the only correct response is to refuse to try to change<br>
> the _FillValue and return an error, although actually it seems that it<br>
> should really be the responsibility of the NetCDF library to give us<br>
> the ELATEFILL error message and not allow the data to be modified.<br>
> -dave<br>><br>
><br>
> On Tue, Mar 28, 2017 at 6:09 PM, Dave Allured - NOAA Affiliate<br>
> <<a href="mailto:dave.allured@noaa.gov">dave.allured@noaa.gov</a>> wrote:<br>
>> Rick,<br>
>><br>
>> I discovered that my current complaint is all due to the underlying Netcdf<br>
>> library, currently version 4.4.1. I would not have sent an NCL bug report<br>
>> if I had realized this earlier. I do intend to report this back to Netcdf.<br>
>><br>
>> In Netcdf-4 file format, there is apparently a restriction that _FillValue<br>
>> may not be changed after data is written. This is a notable difference from<br>
>> Netcdf-3 format. Functions in the nc_put_att family list this possible<br>
>> error, which I think applies to Netcdf-4 file access only:<br>
>><br>
>> NC_ELATEFILL -- Fill values must be written while the file is still in<br>
>> initial define mode.<br>
>><br>
>> This is probably due to the underlying HDF5 layer:<br>
>><br>
>> "the fill value for a dataset cannot be modified once the dataset has<br>
>> been created"<br>
>> <a href="https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue" rel="noreferrer" target="_blank">https://support.hdfgroup.org/<wbr>HDF5/hdf5-quest.html#<wbr>pfillvalue</a><br>
>><br>
>> This is why I think the correct response for both NCL and C programs should<br>
>> be to fail with NC_ELATEFILL or equivalent, rather than "just work". To<br>
>> "just work" might involve a large performance penalty along the lines of<br>
>> scanning or re-writing the entire file variable. Either way, previous data<br>
>> should not be erased, i.e. set to _FillValue.<br>
>><br>
>> --Dave<br>
>><br>
>><br>
>> On Tue, Mar 28, 2017 at 4:44 PM, Rick Brownrigg <<a href="mailto:brownrig@ucar.edu">brownrig@ucar.edu</a>> wrote:<br>
>>><br>
>>> Hi Dave,<br>
>>><br>
>>> I will look into this and likely file a bug report. At one point, there<br>
>>> was an issue whereby if _FillValue (or any attribute) was added to a<br>
>>> variable, it caused all the other attributes to be dropped (but the data was<br>
>>> preserved). This was traced to a bug in the underlying NetCDF library, which<br>
>>> was fixed in version 4.4.1 around summer 2016. The NetCDF library was going<br>
>>> to a lot of effort to make it possible to rewrite a preexisting variable, so<br>
>>> I would think the correct response is for it to "just work", rather than<br>
>>> error out and do nothing?<br>
>>><br>
>>> Rick<br>
>>><br>
>>> On Tue, Mar 28, 2017 at 2:59 PM, Dave Allured - NOAA Affiliate<br>
>>> <<a href="mailto:dave.allured@noaa.gov">dave.allured@noaa.gov</a>> wrote:<br>
>>>><br>
>>>> NCL team,<br>
>>>><br>
>>>> Under some conditions, adding a _FillValue attribute to an existing<br>
>>>> Netcdf-4 file erases previous data. I find this behavior with both NCL<br>
>>>> 6.3.0 and 6.4.0 on Mac OS 10.11 (El Capitan). I get the same behavior with<br>
>>>> both netcdf4-classic and full netcdf4 formats.<br>
>>>><br>
>>>> mac56:~/ncl/bugs/fillvalue1 2> ncdump -s <a href="http://test3.nc" rel="noreferrer" target="_blank">test3.nc</a><br>
>>>> netcdf test3 {<br>
>>>> dimensions:<br>
>>>> dim = 10 ;<br>
>>>> variables:<br>
>>>> int var(dim) ;<br>
>>>> var:_Storage = "contiguous" ;<br>
>>>> var:_Endianness = "little" ;<br>
>>>><br>
>>>> // global attributes:<br>
>>>> :_NCProperties = "version=1|netcdflibversion=4.<wbr>4.1|hdf5libversion=1.8.17"<br>
>>>> ;<br>
>>>> :_SuperblockVersion = 0 ;<br>
>>>> :_IsNetcdf4 = 1 ;<br>
>>>> :_Format = "netCDF-4 classic model" ;<br>
>>>> data:<br>
>>>><br>
>>>> var = 11, 22, 33, 44, 55, _, _, _, _, _ ;<br>
>>>> }<br>
>>>><br>
>>>> mac56:~/ncl/bugs/fillvalue1 3> ncl<br>
>>>> Copyright (C) 1995-2017 - All Rights Reserved<br>
>>>> University Corporation for Atmospheric Research<br>
>>>> NCAR Command Language Version 6.4.0<br>
>>>> The use of this software is governed by a License Agreement.<br>
>>>> See <a href="http://www.ncl.ucar.edu/" rel="noreferrer" target="_blank">http://www.ncl.ucar.edu/</a> for more details.<br>
>>>> ncl 0> f=addfile("<a href="http://test3.nc" rel="noreferrer" target="_blank">test3.nc</a>","w")<br>
>>>> ncl 1> f->var@_FillValue = 99<br>
>>>> ncl 2> exit<br>
>>>><br>
>>>> mac56:~/ncl/bugs/fillvalue1 4> ncdump -s <a href="http://test3.nc" rel="noreferrer" target="_blank">test3.nc</a><br>
>>>> netcdf test3 {<br>
>>>> dimensions:<br>
>>>> dim = 10 ;<br>
>>>> variables:<br>
>>>> int var(dim) ;<br>
>>>> var:_FillValue = 99 ;<br>
>>>> var:_Storage = "contiguous" ;<br>
>>>> var:_Endianness = "little" ;<br>
>>>><br>
>>>> // global attributes:<br>
>>>> :_NCProperties = "version=1|netcdflibversion=4.<wbr>4.1|hdf5libversion=1.8.17"<br>
>>>> ;<br>
>>>> :_SuperblockVersion = 0 ;<br>
>>>> :_IsNetcdf4 = 1 ;<br>
>>>> :_Format = "netCDF-4 classic model" ;<br>
>>>> data:<br>
>>>><br>
>>>> var = _, _, _, _, _, _, _, _, _, _ ;<br>
>>>> }<br>
>>>><br>
>>>> In my opinion, the correct response for netcdf-4 formats should be an<br>
>>>> error, something like "Can not change _FillValue after data is written in<br>
>>>> netcdf-4", and the file is unchanged. In any case, previous data should not<br>
>>>> be silently erased. If you agree, please fix this for the next release.<br>
>>>> Thanks.<br>
>>>><br>
>>>> --Dave<br></blockquote></div></div></div></div>