Computer Science – 13.3 Floating-point numbers, representation and manipulation | e-Consult

Catastrophic cancellation occurs when subtracting two nearly equal numbers in floating-point arithmetic. Due to the finite precision of binary representation, the difference between the two numbers might be lost, leading to a significant loss of significant digits. This can result in a result that is drastically inaccurate.

Consider the following example:

Let's say we want to calculate 1.0 - 0.9999999999999999. In binary, 0.9999999999999999 is very close to 1.0, but not exactly equal. When we subtract them, the result is not exactly zero. Instead, the result is a number with a significant loss of precision. The loss of precision happens because the leading digits of both numbers are cancelled out, leaving only a few significant digits.

Mathematically, let's represent the numbers as:

When we subtract 0.9999999999999999 from 1.0, we are essentially subtracting a number that is very close to 1.0. The result is a number that is significantly less accurate than it should be, demonstrating the catastrophic cancellation effect. This is a serious issue in numerical analysis and can lead to incorrect results.