[ncl-talk] issue about multiply an integer with 10 digits or more

Tue Mar 27 19:55:20 MDT 2018

Dear NCL support,

My name is Sylvia Tang, a high school student from Syosset High School, New
York, USA. I’m doing cyclone research with Dr. Colle from Stony Brook
University in New York.

I have recently used the NCL program to calculate the hurricane PDI (power
dissipation index).  I realized that when I multiply an integer with 10
digits or more, OR two integers whose products contain 11 digits or more,
the results for both are incorrect.

ncl 0> PDI=0.002*1.2*40000000000*(20.52839^3)

ncl 1> print(PDI)

(0)     2.793144e+10

The actual result should be:

8.3049E+11

Or

830492859254

==

My environment is:

root at LAPTOP-EF1RBS4V:~# ncl -V

6.4.0

root at LAPTOP-EF1RBS4V:~# uname -a

Linux LAPTOP-EF1RBS4V 4.4.0-43-Microsoft

root at LAPTOP-EF1RBS4V:~# gcc --version

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

==

I have ftped 2 files integer_10_digit_issue.txt
and integer_with_10_digit_issue.docx (same file with different format)

==

Below is my analysis.

1.       9 digits, is correct

ncl 1> print(400000000);

(0)     400000000

2.       10 digits, becomes a negative number

ncl 0> print(4000000000)

(0)         -294967296

ncl 1> x=4000000000

ncl 2> print(x)

(0)         -294967296

Note, by using calculator, 4000000000/294967296=13.56

3.       11 digit, becomes a positive number, but near 30 times smaller

ncl 3> print(40000000000)

(0)         1345294336

Note, by using calculator, 40000000000/1345294336=29.73

4.       11 digit number divided by another number, becomes 1

ncl 1> print(40000000000/1345294336)

(0)         1

ncl 2> y=40000000000/1345294336

              ncl 3> print(y)

(0)         1

5.       9 digit number times an integer, correct

ncl 5> print(5*400000000)

(0)     2000000000

ncl 6> print(5.9999999*400000000)

(0)         2.4e+09

6.       9 digit number times an integer, if product greater than 2.4e+09,
become a negative number

ncl 7>  print(6*400000000)

-1894967296

ncl 8> print(-6*400000000)

(0)     1894967296

7.       9 digit number, product greater than 2.4e+09, result is correct if
one is integer and one is float

ncl 9> print(6.0*400000000)

2.4e+09

8.       Multiply 2 integers with a predicted 10 digit product, correct

ncl 10> print(100000*10000)

(0)         1000000000

But if the product is greater than 2147450000, result will be wrong.

ncl 55> print(42949*50000);

(0)         2147450000

ncl 51> print(42950*50000);

(0)     -2147467296

9.       Multiply 2 integers with a predicted 11 digit product, wrong

ncl 11> print(100000*100000)

(0)         1410065408

Note, by using calculator, (100000*100000)/1410065408=7.09

10.   11 digit integer multiplied by another integer, becomes a negative
number

ncl 12>  print(12*40000000000)

(0)         -1036337152

11.   11 digit integer multiplied by a float, becomes a positive number,
but result is wrong

ncl 13> print(12.0*40000000000)

(0)         1.614353e+10

Note, by using calculator, (12*40000000000)/ 16143530000=29.73

12.   11 digit float multiplied by another float, result is correct

ncl 14> print(12.0*40000000000.0)

(0)         4.8e+11

13.   11 digit correct result divided by incorrect result, results in
nearly a 30 times difference

ncl 15> print((12.0*40000000000.0)/(12.0*40000000000))

(0)         29.73327

14.   Product is 11 digits, correct result divided by incorrect result, 7
times difference

ncl 16> print((100000*100000.0)/(100000*100000))

(0)         7.09187

15.   12 digits, correct result divided by incorrect result, 704 times
difference

ncl 17> print((12.0*400,000,000,000.0)/(12.0*400000000000))

(0)         704.1739

16.   13 digits, correct result divided by incorrect result, 2887 times
difference

ncl 18> print((12.0*4000000000000.0)/(12.0*4000000000000))

(0)         2887.154

17.   Square is OK

ncl 20> print(50000^2);  square

(0)     2.5e+09  --correct

ncl 21> print(50000*50000);

(0)         -1794967296 –wrong

ncl 75> print(50000*50000.0); float

(0)     2.5e+09  --correct

18.   Square root more than 15 digital number integer

print(sqrt(100000000000000))

(0)     -nan

print(sqrt(1000000000000000.0)); float

(0)     3.162278e+07 --correct

Therefore, I use float 40000000000.0 instead of integer 40000000000 to
produce a correct PDI result.

ncl 0> PDI=0.002*1.2**40000000000.0**(20.52839^3)

ncl 1> print(PDI)

 (0)     8.30493e+11

The correct PD is near 30 times larger than incorrect PD

ncl 2>
print((0.002*1.2*40000000000.0*(20.52839^3))/(0.002*1.2*40000000000*(20.52839^3)))

(0)     29.73327

Thank you

Sylvia Tang

stang471 at gmail.com

3/27/2018
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180327/83615e64/attachment.html>