# [ncl-talk] unexpected results from very simple code

danghy at gmail.com danghy at gmail.com
Mon Jul 20 23:43:18 MDT 2015

```Thanks, Rick!  I'll read the document.

danghy at gmail.com

From: Rick Brownrigg
Date: 2015-07-21 11:19
To: danghy at gmail.com
CC: Arne Melsom; ncl-talk
Subject: Re: [ncl-talk] unexpected results from very simple code
Hi Hongyan,

As Arne indicated, with 32-bit floating point, you can't expect more than 7 significant digits (14 for double), and in your case, you are asking for precision 2 orders of magnitude beyond that.

For the truly curious, all the gory reality about computing and precision/accuracy is detailed in this doc:

http://www.cse.msu.edu/~cse320/Documents/FloatingPoint.pdf

HTH...
Rick

On Mon, Jul 20, 2015 at 7:26 PM, danghy at gmail.com <danghy at gmail.com> wrote:
Hi Arne,

Thanks a lot for the information!  I will probably use "integer" to solve the problem.  But I was really curious why the floating point number returns "210200048", not "210200049"... the perfect case would be that it returns "210200051".

Thanks again.

Hongyan

danghy at gmail.com

From: Arne Melsom
Date: 2015-07-20 17:27
To: danghy
CC: ncl-talk
Subject: Re: [ncl-talk] unexpected results from very simple code
Hi Danghy,

I think that you can't expect (4 byte) floating point numbers to deliver the accuracy (no. digits) that you request. In your case, I suggest using double precision or integer representation. Here's a part of your example using the original representation and the two alternatives:

begin

a=210200051.
print("output / a :")
print(tostring(a))
print(sprintf("%9.0f",a))

ai=210200051
print("output / ai:")
print(tostring(ai))
print(sprinti("%9i",ai))

end

...which gives the following output:

(0) output / a :
(0) 210200048.000000
(0) 210200048
(0) output / ad:
(0) 210200051.000000
(0) 210200051
(0) output / ai:
(0) 210200051
(0) 210200051

Hope this helps!
Arne M.

----- Original Message -----
> Dear all,
>
> I met a problem when reading "station ID" - I've simplified the code as
> follows:
>
> begin
>
> degrees = (/210200051, 121, 36, 43,38,58,26/)
>
> print(sprintf("%9.0f",degrees(0)))
> print(tostring(degrees(0)))
>
> a=210200051.
> print(tostring(a))
> print(sprintf("%9.0f",a))
>
> end
>
> The NCL output on screen is:
>
> ncl test.ncl
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.3.0
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> (0) 210200048
> (0) 210200051
> (0) 210200048.000000
> (0) 210200048
>
> Only the second method "tostring" gives the correct station ID. Could you
> please give me a clue why is this happening?
>
> Appreciate!
>
>
> danghy at gmail.com
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150721/f9414416/attachment.html
```