[ncl-talk] issue about multiply an integer with 10 digits or more
Dennis Shea
shea at ucar.edu
Tue Mar 27 21:01:49 MDT 2018
Hi Sylvia,
Well written question! I hope my response matches the thoroughness of your
question. :-)
Computers have different numbers of 'bits' and 'bytes' that are used
internally to represent various number types.
Some languages (eg, NCL, C) will, by default, use 4-bytes for integers and
'real' numbers. Each byte is 8-bits. Hence, a default integer would be
32-bits. Commonly, one bit is reserved as a sign (- or +). The (signed)
integer number spans -2147483647 to +2147483647. Usually the preceding +
sign is not shown. It is 'understood' to be positive.
Some numbers need more than 4-bytes/32-bits for proper representation. NCL
supports "long" integers and "double" reals. A long integer is represented
by 8-bytes or 64-bits. Again, one bit is reserved for the sign (+ or -). A
'long' 64-bit integer can span -9223372036854775808 to +
9223372036854775808.
If there is no sign-bit then numbers can span 0 to 18446744073709551615.
Yikes!!!
This representation is called an unsigned long integer (ulong).
OK ... How to print 400000000 correctly in NCL? Append an 'l' (long) at the
end of the number.
print(400000000l)
print(12.0*40000000000l)
print(42949l*50000l)
print(42949*50000l
print(50000*50000l) ; Note that you need just one appended 'l'.
; NCL will 'promote' the result to the
numeric type
; that is 'bigger'
print((12.0*40000000000.0)/(12.0*40000000000l)) ; note the 'l' <= 1
print((12*40000000000l)/(12*40000000000l)) ; note the 'l' <= 1
Why does the following fail?
print((12.0*40000000000.0)/(12.0*40000000000 )) ; no 'l'
Because when NCL or C sees 40000000000 without an 'l', it mangles (not sure
how) the 40000000000 number.
print(0.002*1.2*40000000000l*(20.52839^3))
print(0.002*1.2*40000000000.0*(20.52839^3))
===
Here is what NCL currently supports:
https://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclDataTypes.shtml
There is a lot of detail bt, I hink, the top table gives you the idea.
A geneic WWW link:
https://docs.mql4.com/basis/types/integer/integertypes
Great work
Dennis
On Tue, Mar 27, 2018 at 7:55 PM, Sylvia Tang <stang471 at gmail.com> wrote:
> Dear NCL support,
>
>
>
> My name is Sylvia Tang, a high school student from Syosset High School,
> New York, USA. I’m doing cyclone research with Dr. Colle from Stony Brook
> University in New York.
>
> I have recently used the NCL program to calculate the hurricane PDI (power
> dissipation index). I realized that when I multiply an integer with 10
> digits or more, OR two integers whose products contain 11 digits or more,
> the results for both are incorrect.
>
> ncl 0> PDI=0.002*1.2*40000000000*(20.52839^3)
>
> ncl 1> print(PDI)
>
> (0) 2.793144e+10
>
>
>
> The actual result should be:
>
> 8.3049E+11
>
> Or
>
> 830492859254
>
>
> ==
>
> My environment is:
>
> root at LAPTOP-EF1RBS4V:~# ncl -V
>
> 6.4.0
>
>
>
> root at LAPTOP-EF1RBS4V:~# uname -a
>
> Linux LAPTOP-EF1RBS4V 4.4.0-43-Microsoft
>
>
>
> root at LAPTOP-EF1RBS4V:~# gcc --version
>
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
>
> Copyright (C) 2015 Free Software Foundation, Inc.
>
>
>
> ==
>
> I have ftped 2 files integer_10_digit_issue.txt and integer_with_10_digit_issue.docx
> (same file with different format)
>
>
> ==
>
> Below is my analysis.
>
> 1. 9 digits, is correct
>
> ncl 1> print(400000000);
>
> (0) 400000000
>
>
>
> 2. 10 digits, becomes a negative number
>
> ncl 0> print(4000000000)
>
> (0) -294967296
>
> ncl 1> x=4000000000
>
> ncl 2> print(x)
>
> (0) -294967296
>
>
>
> Note, by using calculator, 4000000000/294967296=13.56
>
> 3. 11 digit, becomes a positive number, but near 30 times smaller
>
> ncl 3> print(40000000000)
>
> (0) 1345294336
>
> Note, by using calculator, 40000000000/1345294336=29.73
>
> 4. 11 digit number divided by another number, becomes 1
>
> ncl 1> print(40000000000/1345294336)
>
> (0) 1
>
> ncl 2> y=40000000000/1345294336
>
> ncl 3> print(y)
>
> (0) 1
>
>
>
> 5. 9 digit number times an integer, correct
>
> ncl 5> print(5*400000000)
>
> (0) 2000000000
>
> ncl 6> print(5.9999999*400000000)
>
> (0) 2.4e+09
>
>
>
> 6. 9 digit number times an integer, if product greater than
> 2.4e+09, become a negative number
>
> ncl 7> print(6*400000000)
>
> -1894967296
>
> ncl 8> print(-6*400000000)
>
> (0) 1894967296
>
>
>
> 7. 9 digit number, product greater than 2.4e+09, result is correct
> if one is integer and one is float
>
> ncl 9> print(6.0*400000000)
>
> 2.4e+09
>
>
>
> 8. Multiply 2 integers with a predicted 10 digit product, correct
>
> ncl 10> print(100000*10000)
>
> (0) 1000000000
>
> But if the product is greater than 2147450000 <(214)%20745-0000>, result
> will be wrong.
>
> ncl 55> print(42949*50000);
>
> (0) 2147450000 <(214)%20745-0000>
>
>
>
> ncl 51> print(42950*50000);
>
> (0) -2147467296 <(214)%20746-7296>
>
>
>
> 9. Multiply 2 integers with a predicted 11 digit product, wrong
>
> ncl 11> print(100000*100000)
>
> (0) 1410065408
>
> Note, by using calculator, (100000*100000)/1410065408=7.09
>
> 10. 11 digit integer multiplied by another integer, becomes a negative
> number
>
> ncl 12> print(12*40000000000)
>
> (0) -1036337152
>
>
>
> 11. 11 digit integer multiplied by a float, becomes a positive number,
> but result is wrong
>
> ncl 13> print(12.0*40000000000)
>
> (0) 1.614353e+10
>
> Note, by using calculator, (12*40000000000)/ 16143530000
> <(614)%20353-0000>=29.73
>
> 12. 11 digit float multiplied by another float, result is correct
>
> ncl 14> print(12.0*40000000000.0)
>
> (0) 4.8e+11
>
>
>
> 13. 11 digit correct result divided by incorrect result, results in
> nearly a 30 times difference
>
> ncl 15> print((12.0*40000000000.0)/(12.0*40000000000))
>
> (0) 29.73327
>
>
>
> 14. Product is 11 digits, correct result divided by incorrect result, 7
> times difference
>
> ncl 16> print((100000*100000.0)/(100000*100000))
>
> (0) 7.09187
>
>
>
> 15. 12 digits, correct result divided by incorrect result, 704 times
> difference
>
> ncl 17> print((12.0*400,000,000,000.0)/(12.0*400000000000))
>
> (0) 704.1739
>
>
>
> 16. 13 digits, correct result divided by incorrect result, 2887 times
> difference
>
> ncl 18> print((12.0*4000000000000.0)/(12.0*4000000000000))
>
> (0) 2887.154
>
>
>
> 17. Square is OK
>
> ncl 20> print(50000^2); square
>
> (0) 2.5e+09 --correct
>
> ncl 21> print(50000*50000);
>
> (0) -1794967296 –wrong
>
> ncl 75> print(50000*50000.0); float
>
> (0) 2.5e+09 --correct
>
>
>
> 18. Square root more than 15 digital number integer
>
> print(sqrt(100000000000000))
>
> (0) -nan
>
> print(sqrt(1000000000000000.0)); float
>
> (0) 3.162278e+07 --correct
>
>
>
> Therefore, I use float 40000000000.0 instead of integer 40000000000 to
> produce a correct PDI result.
>
> ncl 0> PDI=0.002*1.2**40000000000.0**(20.52839^3)
>
> ncl 1> print(PDI)
>
> (0) 8.30493e+11
>
>
>
> The correct PD is near 30 times larger than incorrect PD
>
> ncl 2> print((0.002*1.2*40000000000.0*(20.52839^3))/(0.002*1.2*
> 40000000000*(20.52839^3)))
>
> (0) 29.73327
>
>
> Thank you
>
>
>
>
>
> Sylvia Tang
>
> stang471 at gmail.com
>
> 3/27/2018
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180327/14ec76db/attachment.html>
More information about the ncl-talk
mailing list