[ncl-talk] string to float conversion

[Dave Allured - NOAA Affiliate]

Rashed,

This is not a question of rounding.  Data type "float" in NCL is precise to
only 7 decimal digits.  For this application you need to preserve 9 decimal
digits, so you can not use "tofloat".  "todouble" would work fine.

abc  = (/"130511001","130511002","130511003","130511004","
130511005","130511006"/)
cde1  = sprintf("%10f",todouble(abc))
cde2  = toint(cde1)
cde3  = toint(abc)

You could also use any of the NCL 32-bit or 64-bit integer types to
preserve 9 digits: toint, toint64, etc.  Doubles will preserve 15 integer
digits.  32-bit integers "integer" will preserve 9 digits.  Signed 64-bit
integers "int64" will preserve 18 digits.  Unsigned 64-bit integers
"uint64" will preserve 19 digits.

--Dave


On Fri, Apr 27, 2018 at 2:29 PM, Rashed Mahmood <rashidcomsis at gmail.com>
wrote:

> Hi all,
>
> I am wondering if there is an option to stop the function "tofloat" from
> rounding. Here is a simple example that explains the problem:
>
> abc  = (/"130511001","130511002","130511003","130511004","
> 130511005","130511006"/)
> cde1  = sprintf("%10f",tofloat(abc))
> cde2  = toint(cde1)
> cde3  = toint(abc)
>
> print(abc+" = "+cde1+" = "+cde2+" = "+cde3)
>
> This results:
>
> (0)     130511001 = 130511000.000000 = 130511000 = 130511001
> (1)     130511002 = 130511000.000000 = 130511000 = 130511002
> (2)     130511003 = 130511000.000000 = 130511000 = 130511003
> (3)     130511004 = 130511008.000000 = 130511008 = 130511004
> (4)     130511005 = 130511008.000000 = 130511008 = 130511005
> (5)     130511006 = 130511008.000000 = 130511008 = 130511006
>
> This loss of precision could be acceptable if the values were some data
> numbers, however, if these are unique station IDs then this is not correct.
>
> I know the solution is simply use "toint" or "todouble" to avoid this
> situation. Just curious if a rounding option is possible in "tofloat".
>
> Cheers,
> Rashed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180427/d8e40fa3/attachment.html>