[ncl-talk] problem removing missing values from obs dataset

[Adam Herrington]

hi rich. I probably should have mentioned that I tried that and it didn't
work.

dave, very helpful explainer of the array resulting from using .eq. in a
where statement. Thank you for helping me avoid a great deal of potentially
wasted time.

On Thu, Feb 27, 2020 at 5:38 PM Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> That was not quite right.  My apologies.  Please see documentation for the
> *where* function.  It is actually legal for the conditional mask to
> contain missing values.  But when the mask contains missing values, then
> missing values are returned in the output array, in the corresponding
> positions.  This is the behavior that you saw originally.
>
> The remedy is the same as before, use *where (ismissing ())* as shown
> below.
>
>
> On Thu, Feb 27, 2020 at 5:11 PM Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> This is not valid:
>>
>>    *VAR = where (VAR.eq.VAR at _FillValue, 0., VAR)*
>>
>> ... because the array result of the *".eq."* operator, with fill values
>> present, can contain three values:  True, False, and fill value.  The
>> *"where"* function must receive a conditional mask (first argument) that
>> contains only *True* and *False*.
>>
>> This construction will form the desired missing value mask, and do what
>> you intended:
>>
>>    *VAR = where (ismissing (VAR), 0., VAR)*
>>
>> Next, if *VAR at missing_value* is different than the original
>> *VAR at _FillValue*, then you should keep the second statement the way you
>> had it, to change *"missing_values"* to zero.
>>
>>
>> On Thu, Feb 27, 2020 at 4:50 PM Adam Herrington via ncl-talk <
>> ncl-talk at ucar.edu> wrote:
>>
>>> Hi All,
>>>
>>> I have a variable that I am trying to remove the missing values from.
>>> Here are the attributes from printVarSummary call:
>>>
>>> Variable: VAR
>>> Type: float
>>> Total Size: 8220672 bytes
>>>             2055168 values
>>> Number of Dimensions: 3
>>> Dimensions and sizes: [time | 223] x [lat | 32] x [lon | 288]
>>> Coordinates:
>>>             time: [9504..10170]
>>>             lat: [-14.60732984293194..14.60732984293194]
>>>             lon: [   0..358.75]
>>> Number Of Attributes: 9
>>>   remap : remapped via ESMF_regrid_with_weights: Bilinear
>>>   _FillValue : -9999.9
>>>   missing_value : -9999.9
>>>   time_statistic : instantaneous
>>>   units :
>>>   long_name : precipitation
>>>   comments : Unknown1 variable comment
>>>   delta_t : 0000-00-00 03:00:00
>>>   avg_period : 0000-00-00 01:00:00
>>>
>>> So there are two missing value attributes. To set all missing values to
>>> zero, I try:
>>>
>>>   VAR = where(VAR.eq.VAR at _FillValue,0.,VAR)
>>>   VAR = where(VAR.eq.VAR at missing_value,0.,VAR)
>>>
>>> But then subsequently looking for missing values: nMsg =
>>> num(ismissing(VAR)), it says there are still all the missing values:
>>>
>>> (0) missing values present: not allowed: nMsg=17913
>>>
>>> Does anyone see why this is occurring, why my where statements are not
>>> resulting in num(ismissing(VAR))=0?
>>>
>>> Thanks,
>>>
>>> Adam
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20200227/34fe78f9/attachment.html>