[ncl-talk] variable metadata getting overwritten when variable is written to a NetCDF4 file

[David Brown]

I note that your script sets the output format to "NetCDF4Classic". I
also tried it with pure "NetCDF4" and it still worked with the caveat
that you can't ncdump a NetCDF4 file before you write it out fully for
the first time, so I had to slightly modify the script to account for
that issue.
 -dave

On Tue, Nov 3, 2015 at 12:14 AM, David Brown <dbrown at ucar.edu> wrote:
> Well yes that would definitely be a bug, But that is not what I was
> seeing with your script (I ran it only on my Mac with my 6.3.1
> development version).
>
> Here is the output of your test script on my system:
>
>  ncl file_var_metadata_clobber.ncl
>  Copyright (C) 1995-2015 - All Rights Reserved
>  University Corporation for Atmospheric Research
>  NCAR Command Language Version 6.3.1
>  The use of this software is governed by a License Agreement.
>  See http://www.ncl.ucar.edu/ for more details.
>
> netcdf file_var_metadata_clobber {
> variables:
> float testvar ;
> testvar:long_name = "long_name" ;
> testvar:units = "units" ;
> }
>
> netcdf file_var_metadata_clobber {
> variables:
> float testvar ;
> testvar:long_name = "long_name" ;
> testvar:units = "units" ;
> testvar:_FillValue = -99.f ;
> }
>
> The copy & paste seems to have removed the indentation, but otherwise
> this seems like the expected output to me.
>
> Is the script giving different results somewhere else?
>  -dave
>
>
>
> On Mon, Nov 2, 2015 at 7:28 PM, Keith Lindsay <klindsay at ucar.edu> wrote:
>> Dave,
>>
>> Thanks for your description of the subtleties of _FillValue and the
>> value-only assignment. Please note though that not only is the _FillValue
>> attribute getting added, the variable's pre-existing attributes are getting
>> lost. Do you consider this latter aspect a bug or feature? It is this that I
>> am more concerned about. Also, this removal of the pre-existing attributes
>> happens with NetCDF4 (for ncl version 6.3.0, but not 6.2.1) and not with
>> NetCDF3.
>>
>> Keith
>>
>> On Mon, Nov 2, 2015 at 5:13 PM, David Brown <dbrown at ucar.edu> wrote:
>>>
>>> Hi Keith,
>>> I think that what you are seeing is actually documented (although
>>> poorly apparently) behavior. At least, if, as I am assuming from your
>>> test script, the only attribute that gets passed through the
>>> value-only assignment is the _FillValue. The NCL Reference Manual says
>>> this about value-only assignment:
>>>
>>> "No dimension names, coordinate variables or attributes other than
>>> _FillValue are assigned. "
>>>
>>> In fact the _FillValue is a special attribute that even values (as
>>> opposed to variables) in NCL carry around. And it is propagated even
>>> through value-only assignment. That is because it is considered an
>>> essential component of the value.
>>>
>>> If in fact you are seeing other attributes get propagated let us know.
>>> Your test script only showed the _FillValue propagation as far as I
>>> could tell.
>>>  -dave
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 2, 2015 at 9:03 AM, Keith Lindsay <klindsay at ucar.edu> wrote:
>>> > Hi,
>>> > Thanks Dennis. For other users that run into this behavior, a workaround
>>> > to
>>> > avoid it is to ensure that the variable on the rhs of the problematic
>>> > assignment does not have the _FillValue attribute. I.e.
>>> >       if (isatt(tmpvar, "_FillValue")) then
>>> >         delete(tmpvar at _FillValue)
>>> >       end if
>>> > Keith
>>> >
>>> >
>>> > On Mon, Nov 2, 2015 at 8:44 AM, Dennis Shea <shea at ucar.edu> wrote:
>>> >>
>>> >> A JIRA ticket [  NCL-2302 ] has been opened.  THX
>>> >>
>>> >> On Mon, Nov 2, 2015 at 8:06 AM, Keith Lindsay <klindsay at ucar.edu>
>>> >> wrote:
>>> >> > Hi,
>>> >> >
>>> >> > I have come across an instance where I'm writing a variable to a file
>>> >> > and
>>> >> > the variable metadata in the file is getting overwritten, even though
>>> >> > I'm
>>> >> > using (/ ... /) on the right-hand side of the write to file
>>> >> > statement.
>>> >> > That
>>> >> > is, the statement
>>> >> > f->$varname$ = (/ tmpvar /)
>>> >> > is overwriting the metadata for the file variable $varname$ in the
>>> >> > file
>>> >> > pointed to by f. The attached script demonstrates the problem. This
>>> >> > seems
>>> >> > like an ncl bug to me.
>>> >> >
>>> >> > Details of my script that seem to matter are:
>>> >> > 1) the file is NetCDF4
>>> >> > 2) the file variable whose metadata is getting overwritten has been
>>> >> > defined
>>> >> > in the file but not written to yet
>>> >> > 3) the file variable whose metadata is getting overwritten does not
>>> >> > have
>>> >> > the
>>> >> > _FillValue attribute
>>> >> > 4) the variable on rhs of write to file statement has _FillValue
>>> >> > attribute
>>> >> >
>>> >> > The problem happens with ncl 6.3.0 on yellowstone. It does not happen
>>> >> > with
>>> >> > ncl 6.2.1, though with 6.2.1 the _FillValue gets added to the file
>>> >> > variable's metadata, which is unexpected to me.
>>> >> >
>>> >> > Keith
>>> >> >
>>> >> > _______________________________________________
>>> >> > ncl-talk mailing list
>>> >> > ncl-talk at ucar.edu
>>> >> > List instructions, subscriber options, unsubscribe:
>>> >> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > ncl-talk mailing list
>>> > ncl-talk at ucar.edu
>>> > List instructions, subscriber options, unsubscribe:
>>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >
>>
>>