[ncl-talk] unexpected results from very simple code

[Rick Brownrigg]

Hi Hongyan,

As Arne indicated, with 32-bit floating point, you can't expect more than 7
significant digits (14 for double), and in your case, you are asking for
precision 2 orders of magnitude beyond that.

For the truly curious, all the gory reality about computing and
precision/accuracy is detailed in this doc:

http://www.cse.msu.edu/~cse320/Documents/FloatingPoint.pdf

HTH...
Rick

On Mon, Jul 20, 2015 at 7:26 PM, danghy at gmail.com <danghy at gmail.com> wrote:

> Hi Arne,
>
> Thanks a lot for the information!  I will probably use "integer" to solve
> the problem.  But I was really curious why the floating point number
> returns "210200048", not "210200049"... the perfect case would be that it
> returns "210200051".
>
> Thanks again.
>
> Hongyan
> ------------------------------
> danghy at gmail.com
>
>
> *From:* Arne Melsom <arne.melsom at met.no>
> *Date:* 2015-07-20 17:27
> *To:* danghy <danghy at gmail.com>
> *CC:* ncl-talk <ncl-talk at ucar.edu>
> *Subject:* Re: [ncl-talk] unexpected results from very simple code
> Hi Danghy,
>
> I think that you can't expect (4 byte) floating point numbers to deliver
> the accuracy (no. digits) that you request. In your case, I suggest using
> double precision or integer representation. Here's a part of your example
> using the original representation and the two alternatives:
>
> begin
>
> a=210200051.
> print("output / a :")
> print(tostring(a))
> print(sprintf("%9.0f",a))
>
> ad=210200051.d
> print("output / ad:")
> print(tostring(ad))
> print(sprintf("%9.0f",ad))
>
> ai=210200051
> print("output / ai:")
> print(tostring(ai))
> print(sprinti("%9i",ai))
>
> end
>
>
> ...which gives the following output:
>
> (0) output / a :
> (0) 210200048.000000
> (0) 210200048
> (0) output / ad:
> (0) 210200051.000000
> (0) 210200051
> (0) output / ai:
> (0) 210200051
> (0) 210200051
>
> Hope this helps!
> Arne M.
>
>
>
> ----- Original Message -----
> > Dear all,
> >
> > I met a problem when reading "station ID" - I've simplified the code as
> > follows:
> >
> > load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> > load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> > load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> > begin
> >
> > degrees = (/210200051, 121, 36, 43,38,58,26/)
> >
> > print(sprintf("%9.0f",degrees(0)))
> > print(tostring(degrees(0)))
> >
> > a=210200051.
> > print(tostring(a))
> > print(sprintf("%9.0f",a))
> >
> > end
> >
> > The NCL output on screen is:
> >
> > ncl test.ncl
> > Copyright (C) 1995-2015 - All Rights Reserved
> > University Corporation for Atmospheric Research
> > NCAR Command Language Version 6.3.0
> > The use of this software is governed by a License Agreement.
> > See http://www.ncl.ucar.edu/ for more details.
> > (0) 210200048
> > (0) 210200051
> > (0) 210200048.000000
> > (0) 210200048
> >
> > Only the second method "tostring" gives the correct station ID. Could you
> > please give me a clue why is this happening?
> >
> > Appreciate!
> >
> >
> > danghy at gmail.com
> >
> > _______________________________________________
> > ncl-talk mailing list
> > ncl-talk at ucar.edu
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150720/ac906fce/attachment.html