[ncl-talk] string to float conversion
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Fri Apr 27 16:15:23 MDT 2018
Rashed,
This is not a question of rounding. Data type "float" in NCL is precise to
only 7 decimal digits. For this application you need to preserve 9 decimal
digits, so you can not use "tofloat". "todouble" would work fine.
abc = (/"130511001","130511002","130511003","130511004","
130511005","130511006"/)
cde1 = sprintf("%10f",todouble(abc))
cde2 = toint(cde1)
cde3 = toint(abc)
You could also use any of the NCL 32-bit or 64-bit integer types to
preserve 9 digits: toint, toint64, etc. Doubles will preserve 15 integer
digits. 32-bit integers "integer" will preserve 9 digits. Signed 64-bit
integers "int64" will preserve 18 digits. Unsigned 64-bit integers
"uint64" will preserve 19 digits.
--Dave
On Fri, Apr 27, 2018 at 2:29 PM, Rashed Mahmood <rashidcomsis at gmail.com>
wrote:
> Hi all,
>
> I am wondering if there is an option to stop the function "tofloat" from
> rounding. Here is a simple example that explains the problem:
>
> abc = (/"130511001","130511002","130511003","130511004","
> 130511005","130511006"/)
> cde1 = sprintf("%10f",tofloat(abc))
> cde2 = toint(cde1)
> cde3 = toint(abc)
>
> print(abc+" = "+cde1+" = "+cde2+" = "+cde3)
>
> This results:
>
> (0) 130511001 = 130511000.000000 = 130511000 = 130511001
> (1) 130511002 = 130511000.000000 = 130511000 = 130511002
> (2) 130511003 = 130511000.000000 = 130511000 = 130511003
> (3) 130511004 = 130511008.000000 = 130511008 = 130511004
> (4) 130511005 = 130511008.000000 = 130511008 = 130511005
> (5) 130511006 = 130511008.000000 = 130511008 = 130511006
>
> This loss of precision could be acceptable if the values were some data
> numbers, however, if these are unique station IDs then this is not correct.
>
> I know the solution is simply use "toint" or "todouble" to avoid this
> situation. Just curious if a rounding option is possible in "tofloat".
>
> Cheers,
> Rashed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180427/d8e40fa3/attachment.html>
More information about the ncl-talk
mailing list