[ncl-talk] toint function

[David Brown]

The problem is floating point precision, which is only between 6 and 7
digits for decimal 32 bit floating point numbers. If you make your input
value double precision you will obtain the correct values.
So you simply need to add 'd' to your input values:

ncl 1> print(toint(2003201.942))
(0)     2003202
ncl 2> print(toint(2003201.942d))
(0)     2003201

 -dave

On Thu, Mar 1, 2018 at 8:29 AM, Sheng-Hung Wang <wang.446 at osu.edu> wrote:

> Hi
>
> How does function toint work?
>
> Here is my problem
>
> print(toint(2003201.932))
>
> print(toint(2003201.942))
>
> Accoring to NCL web page, toint should truncate any fractional part of
> vales, but I got 2003201 for 1st one, and 2003202 for 2nd one.
>
> I know I can replace toint to floor in this case. I just like to know
> what's really going on for the future.
>
> Thanks
>
> Sheng-Hung
>
> --
> ========================================
>  Sheng-Hung Wang, PhD
>  Polar Meteorology Group
>  Byrd Polar and Climate Research Center
>  The Ohio State University
>  1090 Carmack Road, Columbus, Ohio 43210
>  Phone: (614) 292-1060
>  Fax:   (614) 292-4697
>  email: wang.446 at osu.edu
> ========================================
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180301/23695dba/attachment.html>