Truncation Error vs. Roundoff Error in Finite Differences

Summary

When approximating derivatives with finite differences, truncation error shrinks as the step size $h$ decreases — but roundoff error grows. Below a certain $h$, floating-point noise wins. Central differences reach $O(h^2)$ truncation error and are usually preferable to forward differences.

Forward difference

A first-order forward approximation:

From Taylor expansion about $x$:

Rearranging:

So the truncation error of the forward-difference formula is $O(h)$ — halving $h$ roughly halves the discretization error.

Roundoff error and catastrophic cancellation

In floating-point arithmetic, subtracting two nearly equal numbers loses significant digits. Example:

The true difference is dominated by floating-point noise rather than the mathematical signal. This is catastrophic cancellation.

Trade-off:

• Smaller $h$ → smaller truncation error
• Too small $h$ → larger relative roundoff error

There is an optimal step size that balances both effects.

Central difference — why it is better

Expand both directions:

Subtracting:

Solve for $f'(x)$:

The central-difference formula has truncation error $O(h^2)$ — one order better than forward difference — while even-order terms cancel by symmetry.

Takeaways

1. Truncation error comes from the Taylor-series approximation; it decreases as $h \to 0$.
2. Roundoff error grows when $h$ is too small because of catastrophic cancellation.
3. Central differences give $O(h^2)$ accuracy and are the standard choice when both $f(x+h)$ and $f(x-h)$ are available.
4. In practice, choose $h$ using a balance rule (often related to machine epsilon and the scale of $f$).

References & code

Add notes, plots, or experiments here when ready.