[ncl-talk] Fw: Re: Re: Abnormal data points

Dennis Shea shea at ucar.edu
Sat Apr 17 14:10:39 MDT 2021 

A colleague looked att he problem.

*The source file is not well formed.*  It has:

[1] BOGUS 0s
     ...and...
[2] repetitive grid point locations

%> *ncl chathu.PrintVal.ncl*

The edited output:

(0)       JJ=75  *74.58460022702872   183.930768858386   0*
(1)       JJ=75  74.26130005285533   186.4945623256203  -0.0645499
...
(254)   JJ=75  7*4.58460022702872   183.930768858386   -0.0572294*
(255)   JJ=75  74.26130005285533   186.4945623256203   0
===========================================================
(0)       JJ=125  *13.32306688654348  161.7684132838731   0*
(1)       JJ=125  13.1366684627729    163.4541039710322  -0.0766504
...
(254)   JJ=125  *13.32306688654348   161.7684132838731   -0.0784275*
(255)   JJ=125  13.1366684627729     163.4541039710322   0
====
Fortunately, reading a select subset of the gridded values addresses the
problem. Specifically:

  sfile= "chathu.nc"
  f = addfile(sfile,"r")

  NII = 256

  tauvo_ssp = f->tauvo(:,:,1:NII-2)   ; skip the first and last "I"
[longitude] values.
  printVarSummary(tauvo_ssp)
  printMinMax(tauvo_ssp,0)
  print("---------------")

  LAT2D = f->lat(:,1:NII-2)
  LON2D = f->lon(:,1:NII-2)

The colleague provided an  ESMF regrid script.

%> N2Charhu.ncl

See attached plot

On Thu, Apr 15, 2021 at 3:07 PM Gus Correa via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> Hi Chathurika
>
> ESMF_regrid accepts a mask:
> https://www.ncl.ucar.edu/Document/Functions/ESMF/ESMF_regrid.shtml
> See the SrcGridMask/DstGridMask option.
>
> You may be able to craft a mask that covers exactly the abnormal data
> points.
> In normal conditions the "where" function is a great tool to create masks,
> using data values in different ranges.
> However, since the data and abnormal values are in the same range in your
> case,
> you may need to inspect the lat and lon (or lat2d and lon2d) coordinates
> of your abnormal values to locate the points that are in/out of the mask.
> Some artwork for sure.
>
> In any case, double check your original data file to make sure these
> abnormal values are
> really in the valid range, because if they are out of the valid range you
> can easily use "where"
> to create a mask, so that they don't enter the regrid calculation.
>
> Gus
>
> On Thu, Apr 15, 2021 at 3:45 PM Chathurika <chatu at scsio.ac.cn> wrote:
>
>> Dear Gus,
>>
>>
>> Thank you very much for the reply. The problem is that the data, along
>> the line, is in the same range as other values. if I use "where", all other
>> important values also will be removed, not only from the line. That is why
>> I am also confused here.
>>
>>
>> Many thanks and best regards,
>>
>> Chathu
>>
>>
>> -----Original Messages-----
>> *From:*"Gus Correa via ncl-talk" <ncl-talk at mailman.ucar.edu>
>> *Sent Time:*2021-04-16 03:17:48 (Friday)
>> *To:* "NCL Talk" <ncl-talk at ucar.edu>
>> *Cc:*
>> *Subject:* Re: [ncl-talk] Fw: Re: Re: Abnormal data points
>>
>> Hi Chathurika
>>
>> Maybe to use the NCL "where" function, before you use the ESMF regrid
>> function?
>> https://www.ncl.ucar.edu/Document/Functions/Built-in/where.shtml
>> Something like this:
>>
>> myFillvalue=myvar at _FillValue
>> myvar = where(myvar .ge. abnormal_threshold, myvar=myFillValue, myvar)
>> ...
>> [now do the ESMF regrid]
>> ...
>>
>> I hope this helps,
>> Gus Correa
>>
>> On Thu, Apr 15, 2021 at 2:57 PM Chathurika via ncl-talk <
>> ncl-talk at mailman.ucar.edu> wrote:
>>
>>>
>>>
>>>
>>> -----Original Messages-----
>>> *From:*Chathurika <chatu at scsio.ac.cn>
>>> *Sent Time:*2021-04-16 02:55:41 (Friday)
>>> *To:* "Dennis Shea" <shea at ucar.edu>
>>> *Cc:*
>>> *Subject:* Re: Re: [ncl-talk] Abnormal data points
>>>
>>> Dear Dennis,
>>>
>>>
>>> Thank you so much for the reply. Even before, I use the regridding (ESMF_regrid),
>>> I plotted the data just to check. Still there is this line with abnormal
>>> data values (in the original file). I want to know whether there is any
>>> method which I can use to avoid this abnormal data points. How can I remove
>>> them? Because they are in the same range as normal data.
>>>
>>>
>>> Thank you once again
>>>
>>>
>>> and best regards,
>>>
>>> Chathu
>>>
>>> -----Original Messages-----
>>> *From:*"Dennis Shea" <shea at ucar.edu>
>>> *Sent Time:*2021-04-16 02:37:21 (Friday)
>>> *To:* Chathurika <chatu at scsio.ac.cn>
>>> *Cc:* "Soma Roy" <somaroy892 at gmail.com>, ncl-talk <ncl-talk at ucar.edu>
>>> *Subject:* Re: [ncl-talk] Abnormal data points
>>>
>>> If you have used the ESMF regrid functions for other variables and there
>>> is "no problem", I do not understand why there would be an issue with this
>>> variable. Something MUST be different.  It is up to you to determine just
>>> what the difference might be.
>>> ---------------
>>> The 'line' looks like a single satellite trajectory.
>>> ---------------
>>> Are the grid points missing? or, are they 'bad' values?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Apr 15, 2021 at 11:18 AM Chathurika via ncl-talk <
>>> ncl-talk at mailman.ucar.edu> wrote:
>>>
>>>> Dear Soma,
>>>>
>>>>
>>>> it is NetCDF4, curvilinear data two dimensional coordinates (lat(i,j)
>>>> lon(i,j) ). I regrided using ESMF regridding. (Note: I used ESMF regridding
>>>> for all other variables also. I had no problem).
>>>>
>>>>
>>>> Thank you so much.
>>>>
>>>>
>>>> Best
>>>>
>>>> Chathu
>>>>
>>>>
>>>> -----Original Messages-----
>>>> *From:*"Soma Roy" <somaroy892 at gmail.com>
>>>> *Sent Time:*2021-04-16 00:51:10 (Friday)
>>>> *To:* Chathurika <chatu at scsio.ac.cn>
>>>> *Cc:*
>>>> *Subject:* Re: [ncl-talk] Abnormal data points
>>>>
>>>> What is the data file format?
>>>>
>>>> Soma
>>>>
>>>> On Thu, Apr 15, 2021, 22:10 Chathurika via ncl-talk <
>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>
>>>>> Dear All,
>>>>>
>>>>>
>>>>> Does anyone know which method I can use to remove (or interpolate) the
>>>>> data assigned with this abnormal line (which is in the black box of below
>>>>> figure)?
>>>>>
>>>>>
>>>>> If anyone can help me, this will be a big help. I can send the data
>>>>> file if you want to look at it.
>>>>>
>>>>>
>>>>> Thank you in advance and best regards,
>>>>>
>>>>> Chathu
>>>>>
>>>>> ------------------------------
>>>>>
>>>>>
>>>>>
>>>>> Wickramage Chathurika Hemamali
>>>>> Msc in Physical Oceanography
>>>>> State Key Laboratory of Tropical Oceanography
>>>>> South China Sea Institute of Oceanology
>>>>> University of Chinese Academy of Science
>>>>> China
>>>>>
>>>>> *Specialized in Oceanography and Marine Geology (Bachelor)*
>>>>> *University of Ruhuna*
>>>>> *Matara*
>>>>> *Sri Lanka*
>>>>>
>>>>> Email : wickramagechathurika at rocketmail.com
>>>>> chatu at scsio.ac.cn
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at mailman.ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> Wickramage Chathurika Hemamali
>>>> Msc in Physical Oceanography
>>>> State Key Laboratory of Tropical Oceanography
>>>> South China Sea Institute of Oceanology
>>>> University of Chinese Academy of Science
>>>> China
>>>>
>>>> *Specialized in Oceanography and Marine Geology (Bachelor)*
>>>> *University of Ruhuna*
>>>> *Matara*
>>>> *Sri Lanka*
>>>>
>>>> Email : wickramagechathurika at rocketmail.com
>>>> chatu at scsio.ac.cn
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at mailman.ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>> Wickramage Chathurika Hemamali
>>> Msc in Physical Oceanography
>>> State Key Laboratory of Tropical Oceanography
>>> South China Sea Institute of Oceanology
>>> University of Chinese Academy of Science
>>> China
>>>
>>> *Specialized in Oceanography and Marine Geology (Bachelor)*
>>> *University of Ruhuna*
>>> *Matara*
>>> *Sri Lanka*
>>>
>>> Email : wickramagechathurika at rocketmail.com
>>> chatu at scsio.ac.cn
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>>
>>>
>>> Wickramage Chathurika Hemamali
>>> Msc in Physical Oceanography
>>> State Key Laboratory of Tropical Oceanography
>>> South China Sea Institute of Oceanology
>>> University of Chinese Academy of Science
>>> China
>>>
>>> *Specialized in Oceanography and Marine Geology (Bachelor)*
>>> *University of Ruhuna*
>>> *Matara*
>>> *Sri Lanka*
>>>
>>> Email : wickramagechathurika at rocketmail.com
>>> chatu at scsio.ac.cn
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at mailman.ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>>
>> ------------------------------
>>
>>
>>
>> Wickramage Chathurika Hemamali
>> Msc in Physical Oceanography
>> State Key Laboratory of Tropical Oceanography
>> South China Sea Institute of Oceanology
>> University of Chinese Academy of Science
>> China
>>
>> *Specialized in Oceanography and Marine Geology (Bachelor)*
>> *University of Ruhuna*
>> *Matara*
>> *Sri Lanka*
>>
>> Email : wickramagechathurika at rocketmail.com
>> chatu at scsio.ac.cn
>>
>> _______________________________________________
> ncl-talk mailing list
> ncl-talk at mailman.ucar.edu
> List instructions, subscriber options, unsubscribe:
> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210417/8e101322/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: chathu.PrintVal.ncl
Type: application/octet-stream
Size: 865 bytes
Desc: not available
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210417/8e101322/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: N2Chathu.ncl
Type: application/octet-stream
Size: 2348 bytes
Desc: not available
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210417/8e101322/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_Chathu_Data.png
Type: image/png
Size: 248230 bytes
Desc: not available
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210417/8e101322/attachment.png>

More information about the ncl-talk mailing list