[ncl-talk] issue about multiply an integer with 10 digits or more
Sylvia Tang
stang471 at gmail.com
Tue Mar 27 19:55:20 MDT 2018
Dear NCL support,
My name is Sylvia Tang, a high school student from Syosset High School, New
York, USA. I’m doing cyclone research with Dr. Colle from Stony Brook
University in New York.
I have recently used the NCL program to calculate the hurricane PDI (power
dissipation index). I realized that when I multiply an integer with 10
digits or more, OR two integers whose products contain 11 digits or more,
the results for both are incorrect.
ncl 0> PDI=0.002*1.2*40000000000*(20.52839^3)
ncl 1> print(PDI)
(0) 2.793144e+10
The actual result should be:
8.3049E+11
Or
830492859254
==
My environment is:
root at LAPTOP-EF1RBS4V:~# ncl -V
6.4.0
root at LAPTOP-EF1RBS4V:~# uname -a
Linux LAPTOP-EF1RBS4V 4.4.0-43-Microsoft
root at LAPTOP-EF1RBS4V:~# gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
==
I have ftped 2 files integer_10_digit_issue.txt
and integer_with_10_digit_issue.docx (same file with different format)
==
Below is my analysis.
1. 9 digits, is correct
ncl 1> print(400000000);
(0) 400000000
2. 10 digits, becomes a negative number
ncl 0> print(4000000000)
(0) -294967296
ncl 1> x=4000000000
ncl 2> print(x)
(0) -294967296
Note, by using calculator, 4000000000/294967296=13.56
3. 11 digit, becomes a positive number, but near 30 times smaller
ncl 3> print(40000000000)
(0) 1345294336
Note, by using calculator, 40000000000/1345294336=29.73
4. 11 digit number divided by another number, becomes 1
ncl 1> print(40000000000/1345294336)
(0) 1
ncl 2> y=40000000000/1345294336
ncl 3> print(y)
(0) 1
5. 9 digit number times an integer, correct
ncl 5> print(5*400000000)
(0) 2000000000
ncl 6> print(5.9999999*400000000)
(0) 2.4e+09
6. 9 digit number times an integer, if product greater than 2.4e+09,
become a negative number
ncl 7> print(6*400000000)
-1894967296
ncl 8> print(-6*400000000)
(0) 1894967296
7. 9 digit number, product greater than 2.4e+09, result is correct if
one is integer and one is float
ncl 9> print(6.0*400000000)
2.4e+09
8. Multiply 2 integers with a predicted 10 digit product, correct
ncl 10> print(100000*10000)
(0) 1000000000
But if the product is greater than 2147450000, result will be wrong.
ncl 55> print(42949*50000);
(0) 2147450000
ncl 51> print(42950*50000);
(0) -2147467296
9. Multiply 2 integers with a predicted 11 digit product, wrong
ncl 11> print(100000*100000)
(0) 1410065408
Note, by using calculator, (100000*100000)/1410065408=7.09
10. 11 digit integer multiplied by another integer, becomes a negative
number
ncl 12> print(12*40000000000)
(0) -1036337152
11. 11 digit integer multiplied by a float, becomes a positive number,
but result is wrong
ncl 13> print(12.0*40000000000)
(0) 1.614353e+10
Note, by using calculator, (12*40000000000)/ 16143530000=29.73
12. 11 digit float multiplied by another float, result is correct
ncl 14> print(12.0*40000000000.0)
(0) 4.8e+11
13. 11 digit correct result divided by incorrect result, results in
nearly a 30 times difference
ncl 15> print((12.0*40000000000.0)/(12.0*40000000000))
(0) 29.73327
14. Product is 11 digits, correct result divided by incorrect result, 7
times difference
ncl 16> print((100000*100000.0)/(100000*100000))
(0) 7.09187
15. 12 digits, correct result divided by incorrect result, 704 times
difference
ncl 17> print((12.0*400,000,000,000.0)/(12.0*400000000000))
(0) 704.1739
16. 13 digits, correct result divided by incorrect result, 2887 times
difference
ncl 18> print((12.0*4000000000000.0)/(12.0*4000000000000))
(0) 2887.154
17. Square is OK
ncl 20> print(50000^2); square
(0) 2.5e+09 --correct
ncl 21> print(50000*50000);
(0) -1794967296 –wrong
ncl 75> print(50000*50000.0); float
(0) 2.5e+09 --correct
18. Square root more than 15 digital number integer
print(sqrt(100000000000000))
(0) -nan
print(sqrt(1000000000000000.0)); float
(0) 3.162278e+07 --correct
Therefore, I use float 40000000000.0 instead of integer 40000000000 to
produce a correct PDI result.
ncl 0> PDI=0.002*1.2**40000000000.0**(20.52839^3)
ncl 1> print(PDI)
(0) 8.30493e+11
The correct PD is near 30 times larger than incorrect PD
ncl 2>
print((0.002*1.2*40000000000.0*(20.52839^3))/(0.002*1.2*40000000000*(20.52839^3)))
(0) 29.73327
Thank you
Sylvia Tang
stang471 at gmail.com
3/27/2018
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180327/83615e64/attachment.html>
More information about the ncl-talk
mailing list