12.7 - 20 + 7.3 = -8.8817841970013e-016
I got this problem when I read Programming in Lua book, and I cannot figure it out how to get zero as the result of the calculation.
When I formatted the result to a float:
io.write(string.format("%f", 12.7 - 20 + 7.3)) -- -0.000000
But I got nil if I tried to convert it to an integer:
io.write(math.tointeger(12.7 - 20 + 7.3) -- nil
Is there any explaination why the result turned like this? And how to achieve the result as zero?
I've tried seperate the calculation as two different variables.
local x = 12.7 - 20
local y = 7.3
io.write(x + y)
And turned out, it doesn't happen in other float calculations.
io.write(3.7 - 13 + 9.3) -- 0.0
io.write(14.4 - 31 + 16.6) -- 0.0
And also it doesn't happen with -7.3 + 7.3. The result is 0.0.
When I think the problem just occured with 12.7 - 20 + 7.3, I got this result:
local x = 14.7 - 20
local y = 5.3
io.write(x + y) -- -8.8817841970013e-016
Any calculation with 7.3 and below doesn't return 0.0:
io.write(12.7 - 20 + 7.3)
io.write(13.7 - 20 + 6.3)
io.write(14.7 - 20 + 5.3)
io.write(15.7 - 20 + 4.3)
But, when it is 8.3, it returns 0.0: io.write(11.7 - 20 + 8.3) -- 0.0
I believe this is due to floating point arithmetic error. Floating point numbers are meant to represent any decimal point number. But since computers use binary bits and can only use so many bits per variable, they can represent only a finite set of decimal point numbers.
This fact of computing means that certain decimal point numbers are approximated when converted to the binary format. Any computational result that came from the approximation will be off by a little amount, hence the unexpected answer you received.
The error due to this approximation is normally very, on the order of e-16 in your case. But when asserting to floating point numbers as equal, you will always need to a tolerance to account for the error.