Flow Lab

Syllabus / Module 2: Multivariable & Vector Calculus

Lesson 2.4

Taylor Series in One and Several Variables

Taylor polynomials, remainder estimates and big-O notation; the multivariable Taylor expansion. The single tool behind every finite-difference formula and every error estimate in numerical methods.

Written byClaude Opus

🌊
Why this matters for simulation: Every discretization of the Navier–Stokes equations is justified by a Taylor expansion.

By the end of this lesson you can…

  • Build the degree-n Taylor polynomial of a function about a point and bound its error with the Lagrange remainder.
  • Read and manipulate big-O notation, and predict how an O(h^p) error changes when h is halved.
  • Derive forward, backward, central and second-derivative finite-difference formulas from Taylor expansions and state their order of accuracy.
  • Measure the order of a method from a log–log convergence study and explain the round-off floor.
  • Use Richardson extrapolation to cancel the leading error term and gain accuracy for free.
  • Write the second-order Taylor expansion of f(x, y) using the gradient and the Hessian.
  • Explain why Euler's method is first order and how Runge–Kutta methods are designed by matching Taylor terms.
On this page
·
  1. Polynomials that imitate a function
  2. How wrong is a Taylor polynomial?
  3. Worked example: how good is sin⁡x≈x−x3/6\sin x \approx x - x^3/6sinx≈x−x3/6?
  4. Worked example: how many terms for e1e^1e1?
  5. Big-O notation: the language of error
  6. Finite differences from Taylor
  7. Forward and backward differences
  8. The central difference
  9. The second derivative
  10. Worked example: checking the theory on exe^xex
  11. Truncation versus round-off: the V-shaped error curve
  12. Richardson extrapolation: cancel the error you know
  13. Why Euler is first order and RK4 is fourth
  14. Taylor in several variables
  15. Worked example
  16. Finite differences in 2D: the five-point Laplacian
  17. Exercise: higher-order stencils and a convergence study
  18. Key Takeaways

In your Simulations course you met a whole zoo of integrators (explicit Euler, implicit Euler, Heun, midpoint, RK4, RK45) and a set of facts about them: Euler is “first order”, RK4 is “fourth order”, halving the step of RK4 cuts the error by about 16, and RK45 compares two estimates to choose its step. Where do those facts come from? Every one of them comes from a single tool: the Taylor expansion. The same tool produces every finite-difference stencil a fluid solver uses, from the gradient in the pressure-gradient force (2.3 Gradient & Directional Derivatives) to the Laplacian in viscous diffusion. In 2.2 Linearization, the Chain Rule & the Jacobian you approximated a function by its tangent plane. This lesson keeps going: add the quadratic term, the cubic term, and so on, and, most importantly, learn to say exactly how wrong each truncation is.

▶ Video Taylor polynomials, finite differences, and why methods have an ordercaptions only
Transcript
  1. Every numerical method you have ever used, Euler, R K four, finite differences, is secretly a Taylor series with the tail chopped off. Today we learn to read that tail.
  2. Here is a function, and a point a where we know everything about it: its value, its slope, its curvature, every derivative. How well can a polynomial imitate it nearby?
  3. Degree zero just copies the value. A flat line. Degree one also copies the slope: that is the tangent line, the linearization from earlier in this module.
  4. Degree two copies the curvature, and degree three the rate of change of curvature. Each new term is the next derivative, divided by k factorial, times x minus a to the k.
  5. Keep going and the polynomial hugs the curve over a wider and wider window. The factorial is exactly what makes the k-th derivative of the polynomial match the k-th derivative of f.
  6. Now the question that matters for numerics: how wrong is it? Stay with the tangent line and measure the gap, in red, at a distance h from the point.
  7. Halve h, and the gap drops by about four. Halve it again, four again. The error of the tangent line behaves like h squared.
  8. That is Lagrange's remainder theorem. The error of the degree n polynomial equals the next derivative, at some point xi in between, times h to the n plus one, over n plus one factorial.
  9. We usually compress this into big O notation: the error is order h to the n plus one. The exact constant matters less than the power, because the power tells you how fast the error dies when you shrink h.
  10. Taylor's real superpower is in reverse: building approximations to derivatives. Suppose all we have are samples on a grid, at x minus h, x, and x plus h.
  11. Write the Taylor series for f at x plus h. Move f of x to the left, divide by h, and you get the forward difference: the derivative, plus an error that starts at h over two times f double prime. That is first order.
  12. Now write the series at x minus h. The odd terms flip sign. Subtract the two series, and the h squared terms cancel exactly. The central difference is second order.
  13. You can see it in the picture. The violet central secant is nearly parallel to the green tangent, while the red forward secant is visibly tilted. Symmetry buys you an order.
  14. Let's test this on a computer. Differentiate sine at x equals one for step sizes from one down to ten to the minus fifteen, and plot the error on log log axes.
  15. On log log axes, error equals C times h to the p is a straight line with slope p. The forward difference has slope one. The central difference has slope two: every factor of ten in h buys a factor of one hundred in accuracy.
  16. But below a certain h the curves turn around and climb. That is round-off: we subtract two nearly equal numbers, lose digits, and divide by a tiny h. Truncation error falls, round-off error rises, and the best h sits at the bottom of the V.
  17. Now the payoff for your simulation course. Expand the true solution of y prime equals f over one time step.
  18. Euler's method keeps exactly the first two terms. What it throws away starts at h squared. So each step makes an error of order h squared.
  19. But to reach a fixed time T you need T over h steps, and the errors pile up. T over h times h squared is order h. That is precisely why Euler is called a first order method.
  20. Runge Kutta methods are designed by one rule: sample f cleverly so the step agrees with this Taylor series for as many terms as possible. Midpoint and Heun match through h squared. R K four matches through h to the fourth, so it is fourth order.
  21. The same idea works in several variables. Here are the level curves, in cyan, of a function of x and y.
  22. The Taylor expansion uses the gradient for the linear part, and the Hessian matrix of second partial derivatives for the quadratic part: one half h transpose H h.
  23. The quadratic model, in amber, matches the true contours beautifully near the center, and drifts away farther out, exactly as the cubic error term predicts.
  24. And here is why this matters for fluids. Add the second difference in x to the second difference in y and you get the five point Laplacian, the stencil at the heart of diffusion, viscosity and the pressure solve.
  25. Taylor tells you it is second order, and it even tells you the exact error term. Every discretization of the Navier Stokes equations is justified this way.
  26. To recap. A Taylor polynomial copies derivatives, and its error is order h to the n plus one. Finite differences come from combining expansions so unwanted terms cancel. And the order of a method is the power of h in the first term it gets wrong.
  27. Next, we switch from derivatives to integrals, and learn to add up quantities like mass over a whole region of space.

Polynomials that imitate a function

Suppose you know everything about a function ff at one point aa: its value, slope, curvature, and every higher derivative. Which polynomial of degree nn best imitates ff near aa? The natural answer is the one whose first nn derivatives at aa agree with those of ff.

Write the polynomial in powers of (x−a)(x - a):

Pn(x)=c0+c1(x−a)+c2(x−a)2+⋯+cn(x−a)n.P_n(x) = c_0 + c_1 (x-a) + c_2 (x-a)^2 + \cdots + c_n (x-a)^n .

Setting x=ax = a kills every term but the first, so Pn(a)=c0P_n(a) = c_0. Differentiate once and set x=ax = a: Pn′(a)=c1P_n'(a) = c_1. Differentiate twice: the (x−a)2(x-a)^2 term becomes 2c22c_2, so Pn′′(a)=2c2P_n''(a) = 2c_2. In general the kk-th derivative of ck(x−a)kc_k(x-a)^k is k! ckk!\,c_k, and every other term either vanishes at x=ax = a or was already differentiated away. Matching Pn(k)(a)=f(k)(a)P_n^{(k)}(a) = f^{(k)}(a) gives

ck=f(k)(a)k!,Pn(x)=∑k=0nf(k)(a)k! (x−a)k.c_k = \frac{f^{(k)}(a)}{k!}, \qquad P_n(x) = \sum_{k=0}^{n} \frac{f^{(k)}(a)}{k!}\,(x-a)^k .

This is the Taylor polynomial of degree nn about aa (called a Maclaurin polynomial when a=0a = 0). The factorial is no accident: it exactly undoes the k!k! that differentiation produces.

Degree 0 is the constant f(a)f(a). Degree 1 is the tangent line f(a)+f′(a)(x−a)f(a) + f'(a)(x-a), the linearization from 2.2 Linearization, the Chain Rule & the Jacobian. Degree 2 adds curvature. The workhorse expansions, all about a=0a = 0, are worth memorizing:

ex=1+x+x22!+x33!+⋯sin⁡x=x−x33!+x55!−⋯cos⁡x=1−x22!+x44!−⋯ln⁡(1+x)=x−x22+x33−⋯(−1<x≤1)11−x=1+x+x2+x3+⋯(∣x∣<1)\begin{aligned} e^x &= 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots \\ \sin x &= x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots \\ \cos x &= 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \cdots \\ \ln(1+x) &= x - \frac{x^2}{2} + \frac{x^3}{3} - \cdots \qquad (-1 \lt x \le 1)\\ \frac{1}{1-x} &= 1 + x + x^2 + x^3 + \cdots \qquad (|x| \lt 1) \end{aligned}

Letting n→∞n \to \infty gives the Taylor series. For exe^x, sin⁡x\sin x and cos⁡x\cos x the series converges to the function for every xx. For ln⁡(1+x)\ln(1+x) and 1/(1−x)1/(1-x) it only converges within a distance 1 of the center, the radius of convergence. A striking example is 1/(1+x2)1/(1+x^2): it is perfectly smooth on the whole real line, yet its series about 0 diverges for ∣x∣>1|x| \gt 1. The reason lives in the complex plane, where the function blows up at x=±ix = \pm i, a distance 1 from the origin. You’ll use series like these to solve ODEs in 5.8 Series Solutions.

◆ Interactive Taylor polynomial explorer: raise the degree, drag the center

Loading simulation…

Try this:

  1. With sin⁡x\sin x, raise nn from 1 to 15. The window where the amber polynomial hugs the curve widens steadily. In the lower strip, the error near aa sits at machine precision (about 10−1610^{-16}) and rises steeply away from it.
  2. Switch to 1/(1+x2)1/(1+x^2) and raise nn. Inside the dashed lines the fit improves. Outside, it gets worse with every term. Drag the center to a=2a = 2: the radius grows to 5\sqrt{5}, the distance from 22 to ±i\pm i.
  3. Click a point at distance hh from aa and read the error. Halve the distance. With degree nn the error should fall by roughly 2n+12^{n+1}.

How wrong is a Taylor polynomial?

For numerics, the series matters less than the remainder, Rn(x)=f(x)−Pn(x)R_n(x) = f(x) - P_n(x). The key fact is:

The remainder looks exactly like the next term of the series, except that the derivative is evaluated at an unknown point ξ\xi rather than at aa. You can’t usually find ξ\xi, but you don’t need to. If ∣f(n+1)∣≤M|f^{(n+1)}| \le M on the interval, then

∣Rn(x)∣≤M(n+1)! ∣x−a∣n+1.|R_n(x)| \le \frac{M}{(n+1)!}\,|x-a|^{n+1}.

Why it’s true (sketch). For n=0n = 0 this is the Mean Value Theorem: f(x)−f(a)=f′(ξ)(x−a)f(x) - f(a) = f'(\xi)(x-a). For higher nn, note that the error Rn(t)=f(t)−Pn(t)R_n(t) = f(t) - P_n(t) vanishes at t=at = a together with its first nn derivatives, while its (n+1)(n+1)-st derivative is exactly f(n+1)f^{(n+1)} (because Pn(n+1)=0P_n^{(n+1)} = 0). Comparing RnR_n with (t−a)n+1(t-a)^{n+1} and applying the Mean Value Theorem (in Cauchy’s form) n+1n+1 times walks the point of evaluation from xx in to some ξ\xi between aa and xx. The intuition is the important part: the first term you drop controls the error.

Worked example: how good is sin⁡x≈x−x3/6\sin x \approx x - x^3/6?

Take a=0a = 0 and ∣x∣≤0.5|x| \le 0.5. Since the x4x^4 coefficient of sine is zero, x−x3/6x - x^3/6 is both P3P_3 and P4P_4, so we can use the degree-4 remainder:

∣R4(x)∣≤max⁡∣sin⁡(5)∣5!∣x∣5=∣cos⁡ξ∣120∣x∣5≤(0.5)5120≈2.60×10−4.|R_4(x)| \le \frac{\max|\sin^{(5)}|}{5!}|x|^5 = \frac{|\cos\xi|}{120}|x|^5 \le \frac{(0.5)^5}{120} \approx 2.60\times10^{-4}.

Check at x=0.5x = 0.5: sin⁡0.5=0.4794255\sin 0.5 = 0.4794255 and 0.5−0.125/6=0.47916670.5 - 0.125/6 = 0.4791667. The actual error is 2.59×10−42.59 \times 10^{-4}, just under the bound.

Worked example: how many terms for e1e^1?

To compute e=e1e = e^1 from its series with error below 10−610^{-6}: on [0,1][0, 1], ∣f(n+1)(ξ)∣=eξ≤e<3|f^{(n+1)}(\xi)| = e^\xi \le e \lt 3, so we need 3/(n+1)!<10−63/(n+1)! \lt 10^{-6}, i.e. (n+1)!>3×106(n+1)! \gt 3\times 10^6. Since 9!=362,8809! = 362{,}880 is too small and 10!=3,628,80010! = 3{,}628{,}800 is enough, n=9n = 9 suffices.

✏️ Pen & paperBounding a logarithm●○○
  1. Write the degree-3 Taylor polynomial of f(x)=ln⁡(1+x)f(x) = \ln(1+x) about a=0a = 0.
  2. Use the Lagrange remainder to bound the error of P3(0.2)P_3(0.2) as an approximation of ln⁡1.2\ln 1.2.
  3. Compare with the actual error.
Show worked solution
  1. f(0)=0f(0) = 0, f′=(1+x)−1f' = (1+x)^{-1}, f′′=−(1+x)−2f'' = -(1+x)^{-2}, f′′′=2(1+x)−3f''' = 2(1+x)^{-3}, so f′(0)=1f'(0) = 1, f′′(0)=−1f''(0) = -1, f′′′(0)=2f'''(0) = 2 and P3(x)=x−x22+23!x3=x−x22+x33P_3(x) = x - \dfrac{x^2}{2} + \dfrac{2}{3!}x^3 = x - \dfrac{x^2}{2} + \dfrac{x^3}{3}.
  2. f(4)(x)=−6(1+x)−4f^{(4)}(x) = -6(1+x)^{-4}. For ξ∈(0,0.2)\xi \in (0, 0.2), ∣f(4)(ξ)∣=6/(1+ξ)4≤6|f^{(4)}(\xi)| = 6/(1+\xi)^4 \le 6. So ∣R3(0.2)∣≤64!(0.2)4=624(0.0016)=4.0×10−4|R_3(0.2)| \le \dfrac{6}{4!}(0.2)^4 = \dfrac{6}{24}(0.0016) = 4.0\times10^{-4}.
  3. P3(0.2)=0.2−0.02+0.0026‾=0.182667P_3(0.2) = 0.2 - 0.02 + 0.002\overline{6} = 0.182667, while ln⁡1.2=0.182322\ln 1.2 = 0.182322. The actual error is 3.45×10−43.45\times10^{-4}, within the bound. The bound is slightly pessimistic because it used the worst case ξ=0\xi = 0.

Big-O notation: the language of error

Numerical analysts rarely care about the exact constant in an error. They care about how the error scales with the step size hh. Big-O notation captures exactly that.

So Taylor’s theorem says

f(a+h)=f(a)+hf′(a)+h22f′′(a)+O(h3)f(a+h) = f(a) + h f'(a) + \frac{h^2}{2}f''(a) + O(h^3)

whenever f′′′f''' is bounded near aa. Big-O expressions follow a few simple rules (all as h→0h \to 0):

  • O(hp)+O(hq)=O(hmin⁡(p,q))O(h^p) + O(h^q) = O(h^{\min(p,q)}): the lowest power dominates.
  • hm⋅O(hp)=O(hm+p)h^m \cdot O(h^p) = O(h^{m+p}), and O(hp)/h=O(hp−1)O(h^p)/h = O(h^{p-1}).
  • c⋅O(hp)=O(hp)c \cdot O(h^p) = O(h^p) for any constant cc.

The practical meaning: if an error is O(hp)O(h^p) with a nonzero leading term, halving hh divides the error by about 2p2^p. A first-order error halves; a second-order error drops by 4; a fourth-order error drops by 16. Take logs of E≈ChpE \approx C h^p:

log⁡E≈log⁡C+plog⁡h,\log E \approx \log C + p \log h,

so on log–log axes the error is a straight line with slope pp. That is how you measure the order of any method experimentally, and how you catch bugs. An RK4 implementation that only shows slope 1 has a typo in it.

✅ Concept check

A method has error $O(h^2)$. You reduce $h$ from $0.01$ to $0.001$. Roughly how does the error change?

Finite differences from Taylor

Now turn Taylor around. Instead of using derivatives to approximate function values, use function values on a grid to approximate derivatives. This is the core of every grid-based PDE solver (10.1 Finite Differences: Discretizing Derivatives).

Forward and backward differences

Expand ff one step ahead:

f(x+h)=f(x)+hf′(x)+h22f′′(x)+h36f′′′(x)+O(h4).f(x+h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x) + \frac{h^3}{6} f'''(x) + O(h^4).

Subtract f(x)f(x) and divide by hh:

f(x+h)−f(x)h=f′(x)+h2f′′(x)+O(h2)⏟truncation error.\frac{f(x+h) - f(x)}{h} = f'(x) + \underbrace{\frac{h}{2} f''(x) + O(h^2)}_{\text{truncation error}} .

The forward difference is first order: its error is O(h)O(h), led by h2f′′\tfrac h2 f''. The same steps with f(x−h)f(x - h) give the backward difference (f(x)−f(x−h))/h=f′(x)−h2f′′(x)+O(h2)\big(f(x) - f(x-h)\big)/h = f'(x) - \tfrac h2 f''(x) + O(h^2), also first order, with the opposite sign of error.

The central difference

Write both expansions:

f(x+h)=f+hf′+h22f′′+h36f′′′+h424f′′′′+O(h5),f(x−h)=f−hf′+h22f′′−h36f′′′+h424f′′′′+O(h5).\begin{aligned} f(x+h) &= f + h f' + \tfrac{h^2}{2} f'' + \tfrac{h^3}{6} f''' + \tfrac{h^4}{24} f'''' + O(h^5), \\ f(x-h) &= f - h f' + \tfrac{h^2}{2} f'' - \tfrac{h^3}{6} f''' + \tfrac{h^4}{24} f'''' + O(h^5). \end{aligned}

The even-power terms are identical; the odd-power terms flip sign. Subtract, and ff, f′′f'' and f′′′′f'''' all cancel:

f(x+h)−f(x−h)=2hf′+h33f′′′+O(h5)⟹f(x+h)−f(x−h)2h=f′(x)+h26f′′′(x)+O(h4).f(x+h) - f(x-h) = 2h f' + \frac{h^3}{3} f''' + O(h^5) \quad\Longrightarrow\quad \frac{f(x+h) - f(x-h)}{2h} = f'(x) + \frac{h^2}{6} f'''(x) + O(h^4).

The central difference is second order. By sampling symmetrically we got an extra order of accuracy for free, with the same two function evaluations.

The second derivative

Add the two expansions instead, and now the odd terms cancel:

f(x+h)+f(x−h)=2f+h2f′′+h412f′′′′+O(h6),f(x+h) + f(x-h) = 2f + h^2 f'' + \frac{h^4}{12} f'''' + O(h^6), f(x+h)−2f(x)+f(x−h)h2=f′′(x)+h212f′′′′(x)+O(h4).\frac{f(x+h) - 2f(x) + f(x-h)}{h^2} = f''(x) + \frac{h^2}{12} f''''(x) + O(h^4).

This three-point stencil [1,−2,1]/h2[1, -2, 1]/h^2 is the most important formula in numerical PDEs. It is the 1D Laplacian, the core of the heat equation (10.2 Explicit Methods for the Heat Equation), and its 2D version appears below.

Worked example: checking the theory on exe^x

Approximate f′(0)f'(0) and f′′(0)f''(0) for f=exf = e^x (exact values: both 1) with h=0.1h = 0.1:

FormulaValueActual errorPredicted leading error
forward (e0.1−1)/0.1(e^{0.1} - 1)/0.11.0517091.0517095.17×10−25.17\times10^{-2}h2f′′=5.0×10−2\tfrac h2 f'' = 5.0\times10^{-2}
central (e0.1−e−0.1)/0.2(e^{0.1} - e^{-0.1})/0.21.0016681.0016681.67×10−31.67\times10^{-3}h26f′′′=1.67×10−3\tfrac{h^2}{6} f''' = 1.67\times10^{-3}
second deriv. (e0.1−2+e−0.1)/0.01(e^{0.1} - 2 + e^{-0.1})/0.011.0008341.0008348.34×10−48.34\times10^{-4}h212f′′′′=8.33×10−4\tfrac{h^2}{12} f'''' = 8.33\times10^{-4}

The Taylor error terms aren’t just bounds. For small hh they predict the error, digits and all. Halve hh to 0.050.05 and the forward error becomes 0.02540.0254, about half, while the central error becomes 4.17×10−44.17\times10^{-4}, about a quarter.

✏️ Pen & paperA one-sided second-order formula●●○

At a wall boundary you can’t sample f(x−h)f(x - h). Find weights a,b,ca, b, c so that

af(x)+bf(x+h)+cf(x+2h)h=f′(x)+O(h2),\frac{a f(x) + b f(x+h) + c f(x+2h)}{h} = f'(x) + O(h^2),

and find the leading error term. Check it numerically for f=exf = e^x, x=0x = 0, h=0.1h = 0.1.

Show worked solution

Expand: f(x+h)=f+hf′+h22f′′+h36f′′′+⋯f(x+h) = f + hf' + \tfrac{h^2}{2}f'' + \tfrac{h^3}{6}f''' + \cdots and f(x+2h)=f+2hf′+2h2f′′+4h33f′′′+⋯f(x+2h) = f + 2hf' + 2h^2 f'' + \tfrac{4h^3}{3}f''' + \cdots.

Then af(x)+bf(x+h)+cf(x+2h)=(a+b+c)f+(b+2c) hf′+(b2+2c)h2f′′+(b6+4c3)h3f′′′+⋯a f(x) + b f(x+h) + c f(x+2h) = (a+b+c) f + (b + 2c)\,h f' + \left(\tfrac b2 + 2c\right) h^2 f'' + \left(\tfrac b6 + \tfrac{4c}{3}\right) h^3 f''' + \cdots.

After dividing by hh we need:

  • a+b+c=0a + b + c = 0 (no f/hf/h term),
  • b+2c=1b + 2c = 1 (the coefficient of f′f' is 1),
  • b2+2c=0\tfrac b2 + 2c = 0 (the hf′′h f'' term cancels).

The third equation gives b=−4cb = -4c. Substituting into the second gives −2c=1-2c = 1, so c=−12c = -\tfrac12, b=2b = 2 and a=−32a = -\tfrac32. Multiplying top and bottom by 2:

f′(x)≈−3f(x)+4f(x+h)−f(x+2h)2h.f'(x) \approx \frac{-3 f(x) + 4 f(x+h) - f(x+2h)}{2h}.

Leading error: (b6+4c3)h2f′′′=(13−23)h2f′′′=−h23f′′′(x)\left(\tfrac b6 + \tfrac{4c}{3}\right) h^2 f''' = \left(\tfrac13 - \tfrac23\right) h^2 f''' = -\tfrac{h^2}{3} f'''(x). It is second order, but with a constant twice as large as the central difference’s h26\tfrac{h^2}{6}, the price of being one-sided.

Check: (−3+4e0.1−e0.2)/0.2=(−3+4.420684−1.221403)/0.2=0.996405(-3 + 4e^{0.1} - e^{0.2})/0.2 = (-3 + 4.420684 - 1.221403)/0.2 = 0.996405. The error is −3.6×10−3-3.6\times 10^{-3}, close to the predicted −0.01/3=−3.3×10−3-0.01/3 = -3.3\times10^{-3}. Formulas like this impose Neumann (zero-gradient) boundary conditions in fluid solvers.

Truncation versus round-off: the V-shaped error curve

The theory says: shrink hh and the error shrinks. A computer disagrees once hh gets small enough. Doubles carry about 16 significant digits (machine epsilon ε≈2.2×10−16\varepsilon \approx 2.2\times10^{-16}). When hh is tiny, f(x+h)f(x+h) and f(x)f(x) agree in most of their digits, and subtracting them leaves mostly noise of size about ε∣f∣\varepsilon |f|. Dividing by hh amplifies that noise. For the forward difference the total error is roughly

E(h)≈h2∣f′′∣⏟truncation+ε∣f∣h⏟round-off.E(h) \approx \underbrace{\frac{h}{2}|f''|}_{\text{truncation}} + \underbrace{\frac{\varepsilon |f|}{h}}_{\text{round-off}} .

The first term falls as hh shrinks and the second rises. Minimizing over hh (set dE/dh=0dE/dh = 0) gives h∗∼ε≈10−8h^\ast \sim \sqrt{\varepsilon} \approx 10^{-8}, with a best possible error around 10−810^{-8}: half your digits are gone. For the central difference, E≈h26∣f′′′∣+ε∣f∣/hE \approx \tfrac{h^2}{6}|f'''| + \varepsilon|f|/h gives h∗∼ε1/3≈6×10−6h^\ast \sim \varepsilon^{1/3} \approx 6\times 10^{-6} and a best error near 10−1110^{-11}. For the second derivative the round-off grows like ε/h2\varepsilon/h^2, so h∗∼ε1/4≈10−4h^\ast \sim \varepsilon^{1/4} \approx 10^{-4}. These are typical-case figures. The round-off arm is jagged noise, so a particular grid of hh values can get lucky and land on a near-cancellation: the convergence cell below bottoms out one or two orders of magnitude lower than 10−810^{-8}, at an hh that shifts if you change the grid. Trust the shape of the V and the order of magnitude, not the single best point.

◆ Interactive Finite-difference error lab: log–log error vs step size

Loading simulation…

Things to try:

  1. With forward and central ticked, slide hh from 11 down toward 10−510^{-5}. Compare each straight stretch with the grey reference slopes, and watch the readout’s “observed order” settle at 1 and 2.
  2. Keep going. Below 10−810^{-8} (forward) and 10−510^{-5} (central) the curves turn around and climb with slope −1-1. That is round-off, and its jaggedness is the fingerprint of floating-point noise.
  3. Tick the 5-point formula: slope 4, but its V bottoms out at a larger hh. Tick the second derivative: its round-off arm climbs with slope −2-2, and its best error is only about 10−810^{-8}.
✅ Concept check

In a convergence study, a method gives error $8.0\times10^{-5}$ at $h = 0.1$ and $5.0\times10^{-6}$ at $h = 0.05$. What is its observed order $p$?

✅ Concept check

You shrink $h$ in a central-difference derivative from $10^{-4}$ to $10^{-10}$ and the error gets much worse. Why?

Run a convergence study yourself:

🐍 PythonConvergence study: forward vs central vs second derivative
⌘/Ctrl + Enter

Richardson extrapolation: cancel the error you know

Taylor tells you not just the order but the form of the error. For the central difference,

D(h)=f′(x)+c2h2+c4h4+⋯ ,D(h) = f'(x) + c_2 h^2 + c_4 h^4 + \cdots,

where c2=f′′′(x)/6c_2 = f'''(x)/6 is unknown but doesn’t depend on hh. Compute with two step sizes:

D(h)=f′+c2h2+O(h4),D(h/2)=f′+14c2h2+O(h4).D(h) = f' + c_2 h^2 + O(h^4), \qquad D(h/2) = f' + \tfrac14 c_2 h^2 + O(h^4).

Multiply the second by 4 and subtract the first. The c2c_2 terms cancel:

4 D(h/2)−D(h)3=f′(x)+O(h4).\frac{4\,D(h/2) - D(h)}{3} = f'(x) + O(h^4).

Two second-order estimates combine into a fourth-order one. In general, if D(h)=L+c hp+⋯D(h) = L + c\,h^p + \cdots, then 2pD(h/2)−D(h)2p−1=L+O(hp+1)\dfrac{2^p D(h/2) - D(h)}{2^p - 1} = L + O(h^{p+1}) or better. And as a bonus, D(h/2)−D(h)≈−34c2h2D(h/2) - D(h) \approx -\tfrac34 c_2 h^2 estimates the error of D(h/2)D(h/2) itself.

Worked example. For f=exf = e^x at 00: D(0.1)=1.0016675D(0.1) = 1.0016675 and D(0.05)=1.0004167D(0.05) = 1.0004167. Then (4(1.0004167)−1.0016675)/3=0.99999979\big(4(1.0004167) - 1.0016675\big)/3 = 0.99999979. The error fell from 4×10−44\times10^{-4} to 2×10−72\times10^{-7} with no new function evaluations, just arithmetic on numbers we already had. (Expanding 4D(h/2)−D(h)4D(h/2) - D(h) in full shows that it is the 5-point stencil from the error lab, with spacing h/2h/2.)

✅ Concept check

A forward difference satisfies $D(h) = f' + c\,h + O(h^2)$. Which combination is second-order accurate?

Why Euler is first order and RK4 is fourth

Here is the connection your Simulations course probably stated without proof. Take the ODE y′=f(t,y)y' = f(t, y) and expand the exact solution over one step:

y(t+h)=y+h y′+h22y′′+h36y′′′+O(h4).y(t+h) = y + h\,y' + \frac{h^2}{2}y'' + \frac{h^3}{6}y''' + O(h^4).

Euler sets yn+1=yn+hf(tn,yn)=y+hy′y_{n+1} = y_n + h f(t_n, y_n) = y + h y'. It matches the series through the hh term and drops everything from h22y′′\tfrac{h^2}{2}y'' on, so the local truncation error (error made in one step) is O(h2)O(h^2). To reach a fixed time TT you take N=T/hN = T/h steps, and if the errors accumulate without being wildly amplified, the global error is

N⋅O(h2)=Th O(h2)=O(h).N \cdot O(h^2) = \frac{T}{h}\,O(h^2) = O(h).

That’s the meaning of “first order”: the global error is O(h1)O(h^1). In general, local error O(hp+1)O(h^{p+1}) means global error O(hp)O(h^p). This makes rigorous the error-and-order analysis you’ll do in 4.1 Local Error, Global Error & Order.

Midpoint. To do better we need the h2h^2 term, and y′′y'' requires the chain rule from 2.2 Linearization, the Chain Rule & the Jacobian: since y′=f(t,y(t))y' = f(t, y(t)),

y′′=ddtf(t,y(t))=ft+fy y′=ft+fyf.y'' = \frac{d}{dt} f(t, y(t)) = f_t + f_y\,y' = f_t + f_y f .

The midpoint method computes k1=f(t,y)k_1 = f(t, y) and k2=f ⁣(t+h2, y+h2k1)k_2 = f\!\left(t + \tfrac h2,\, y + \tfrac h2 k_1\right), then sets yn+1=y+hk2y_{n+1} = y + h k_2. Expand k2k_2 with the two-variable first-order Taylor expansion of ff (the subject of the next section):

k2=f+h2ft+h2k1fy+O(h2)=f+h2(ft+ffy)+O(h2).k_2 = f + \frac h2 f_t + \frac h2 k_1 f_y + O(h^2) = f + \frac h2\left(f_t + f f_y\right) + O(h^2).

So y+hk2=y+hf+h22(ft+ffy)+O(h3)=y+hy′+h22y′′+O(h3)y + h k_2 = y + h f + \tfrac{h^2}{2}(f_t + f f_y) + O(h^3) = y + h y' + \tfrac{h^2}{2} y'' + O(h^3). It matches through h2h^2: local error O(h3)O(h^3), global error O(h2)O(h^2). The midpoint method is second order because one well-placed extra evaluation of ff reproduces the h2h^2 Taylor term.

RK4 chooses four stages and their weights 16,13,13,16\tfrac16, \tfrac13, \tfrac13, \tfrac16 so that the expansion matches the true solution through h4h^4. Doing that by hand means matching eight “elementary differentials” like fyyf2f_{yy}f^2 and fy2ff_y^2 f. It’s tedious but uses nothing beyond what you just did. That’s the whole design principle of 4.3 Runge–Kutta Methods & Butcher Tableaux.

🐍 PythonMeasure the global order of Euler, midpoint and RK4
⌘/Ctrl + Enter
✏️ Pen & paperHeun's method is second order●●●

Heun’s method (improved Euler) is k1=f(t,y)k_1 = f(t, y), k2=f(t+h,y+hk1)k_2 = f(t + h, y + h k_1), yn+1=y+h2(k1+k2)y_{n+1} = y + \tfrac h2 (k_1 + k_2).

  1. Expand k2k_2 to first order in hh.
  2. Show that yn+1y_{n+1} matches the exact Taylor expansion of y(t+h)y(t+h) through the h2h^2 term.
  3. Conclude the local and global orders.
Show worked solution
  1. Using the two-variable Taylor expansion f(t+Δt,y+Δy)=f+Δt ft+Δy fy+O(Δ2)f(t + \Delta t, y + \Delta y) = f + \Delta t\, f_t + \Delta y\, f_y + O(\Delta^2) with Δt=h\Delta t = h and Δy=hk1=hf\Delta y = h k_1 = h f: k2=f+hft+hffy+O(h2)k_2 = f + h f_t + h f f_y + O(h^2).
  2. yn+1=y+h2(f+f+hft+hffy)+O(h3)=y+hf+h22(ft+ffy)+O(h3)y_{n+1} = y + \tfrac h2\left(f + f + h f_t + h f f_y\right) + O(h^3) = y + h f + \tfrac{h^2}{2}(f_t + f f_y) + O(h^3). The exact solution satisfies y′=fy' = f and y′′=ft+fyfy'' = f_t + f_y f (chain rule), so y(t+h)=y+hf+h22(ft+ffy)+O(h3)y(t+h) = y + h f + \tfrac{h^2}{2}(f_t + f f_y) + O(h^3). The two agree through h2h^2.
  3. The first mismatch is at h3h^3, so the local error is O(h3)O(h^3) and, after T/hT/h steps, the global error is O(h2)O(h^2): Heun is second order. (Euler, by comparison, misses the h22y′′\tfrac{h^2}{2}y'' term unless y′′≡0y'' \equiv 0, so it is only first order.)

Taylor in several variables

Fluid fields depend on xx, yy, zz and tt, so we need the multivariable version. There’s a neat trick that reduces it to one variable. To expand f(x)f(\mathbf x) about a\mathbf a in direction h=(h1,h2)\mathbf h = (h_1, h_2), look at ff along the line through a\mathbf a:

g(s)=f(a+s h),0≤s≤1.g(s) = f(\mathbf a + s\,\mathbf h), \qquad 0 \le s \le 1 .

Then g(0)=f(a)g(0) = f(\mathbf a) and g(1)=f(a+h)g(1) = f(\mathbf a + \mathbf h). By the chain rule,

g′(s)=fxh1+fyh2=∇f⋅h,g′′(s)=fxxh12+2fxyh1h2+fyyh22=hTH h,g'(s) = f_x h_1 + f_y h_2 = \nabla f \cdot \mathbf h, \qquad g''(s) = f_{xx}h_1^2 + 2 f_{xy}h_1 h_2 + f_{yy} h_2^2 = \mathbf h^{\mathsf T} H\, \mathbf h,

using Clairaut’s theorem (fxy=fyxf_{xy} = f_{yx}, from 2.1 Functions of Several Variables & Partial Derivatives). Here HH is the Hessian matrix of second partials that you met in 2.3 Gradient & Directional Derivatives. The one-variable expansion g(1)=g(0)+g′(0)+12g′′(0)+⋯g(1) = g(0) + g'(0) + \tfrac12 g''(0) + \cdots becomes:

Read it term by term. The constant is the value. The linear term ∇f⋅h\nabla f \cdot \mathbf h is the tangent plane, and it is largest when h\mathbf h points along the gradient. The quadratic term 12hTHh\tfrac12\mathbf h^{\mathsf T}H\mathbf h is a quadratic form, and its shape is governed by the eigenvalues of the symmetric matrix HH (1.5 Eigenvectors & Eigenvalues): both positive gives a bowl, both negative a dome, mixed signs a saddle. At a critical point the gradient term vanishes and the quadratic term is everything, which is exactly why the second-derivative test of the last lesson works.

Worked example

Expand f(x,y)=excos⁡yf(x, y) = e^x \cos y to second order about (0,0)(0, 0). The derivatives are fx=excos⁡yf_x = e^x\cos y, fy=−exsin⁡yf_y = -e^x \sin y, fxx=excos⁡yf_{xx} = e^x \cos y, fxy=−exsin⁡yf_{xy} = -e^x\sin y and fyy=−excos⁡yf_{yy} = -e^x\cos y. At the origin: f=1f = 1, ∇f=(1,0)\nabla f = (1, 0) and H=[100−1]H = \begin{bmatrix} 1 & 0 \\ 0 & -1\end{bmatrix}. So

excos⁡y≈1+x+12(x2−y2).e^x\cos y \approx 1 + x + \tfrac12\left(x^2 - y^2\right).

Check by multiplying 1D series: (1+x+x22)(1−y22)=1+x+x22−y22+(cubic and higher)(1 + x + \tfrac{x^2}{2})(1 - \tfrac{y^2}{2}) = 1 + x + \tfrac{x^2}{2} - \tfrac{y^2}{2} + (\text{cubic and higher}). ✓ The Hessian has eigenvalues ±1\pm1: near the origin the surface curves up along xx and down along yy, like a saddle superimposed on a tilt.

✏️ Pen & paperA quadratic model and its error●●○

Let f(x,y)=x2y+y3f(x, y) = x^2 y + y^3.

  1. Find the second-order Taylor polynomial QQ of ff about (1,1)(1, 1), in terms of h=x−1h = x - 1 and k=y−1k = y - 1.
  2. Use the linear part alone and then QQ to estimate f(1.1,0.9)f(1.1, 0.9). Compare with the exact value.
  3. Since ff is a cubic polynomial, its Taylor expansion ends at third order. Compute the third-order term and verify that it accounts for the remaining error exactly.
Show worked solution
  1. f(1,1)=2f(1,1) = 2. fx=2xy=2f_x = 2xy = 2, fy=x2+3y2=4f_y = x^2 + 3y^2 = 4. fxx=2y=2f_{xx} = 2y = 2, fxy=2x=2f_{xy} = 2x = 2, fyy=6y=6f_{yy} = 6y = 6. So Q=2+2h+4k+12(2h2+2⋅2hk+6k2)=2+2h+4k+h2+2hk+3k2Q = 2 + 2h + 4k + \tfrac12\left(2h^2 + 2\cdot 2hk + 6k^2\right) = 2 + 2h + 4k + h^2 + 2hk + 3k^2.
  2. With h=0.1h = 0.1, k=−0.1k = -0.1: the linear part gives 2+0.2−0.4=1.82 + 0.2 - 0.4 = 1.8. The quadratic terms add 0.01−0.02+0.03=0.020.01 - 0.02 + 0.03 = 0.02, so Q=1.82Q = 1.82. Exact: f(1.1,0.9)=1.21⋅0.9+0.729=1.089+0.729=1.818f(1.1, 0.9) = 1.21 \cdot 0.9 + 0.729 = 1.089 + 0.729 = 1.818. The linear error is 0.0180.018; the quadratic error is only 0.0020.002.
  3. The nonzero third partials are fxxy=2f_{xxy} = 2 and fyyy=6f_{yyy} = 6 (while fxxx=fxyy=0f_{xxx} = f_{xyy} = 0). The third-order term is 16g′′′(0)\tfrac16 g'''(0), where g′′′=fxxxh3+3fxxyh2k+3fxyyhk2+fyyyk3g''' = f_{xxx}h^3 + 3f_{xxy}h^2k + 3f_{xyy}hk^2 + f_{yyy}k^3 (the same binomial pattern as the second-order term), so it equals 16(3fxxyh2k+fyyyk3)=16(3⋅2⋅(0.01)(−0.1)+6(−0.001))=16(−0.006−0.006)=−0.002\tfrac16\left(3 f_{xxy} h^2 k + f_{yyy} k^3\right) = \tfrac16\left(3\cdot2\cdot(0.01)(-0.1) + 6(-0.001)\right) = \tfrac16(-0.006 - 0.006) = -0.002. Indeed 1.82−0.002=1.8181.82 - 0.002 = 1.818 exactly. The error of QQ is O(∥h∥3)O(\|\mathbf h\|^3), as promised.
✅ Concept check

At a point where $\nabla f = \mathbf 0$ and $H = \begin{bmatrix} 2 & 0 \\ 0 & 8 \end{bmatrix}$, the second-order model of $f(\mathbf a + \mathbf h) - f(\mathbf a)$ is…

Finite differences in 2D: the five-point Laplacian

Apply the 1D second-difference result in each direction on a grid with spacing hh:

f(x+h,y)−2f+f(x−h,y)h2=fxx+h212fxxxx+O(h4),\frac{f(x+h, y) - 2f + f(x-h, y)}{h^2} = f_{xx} + \frac{h^2}{12}f_{xxxx} + O(h^4),

and similarly in yy. Adding them:

∇2f≈fE+fW+fN+fS−4fCh2,error=h212(fxxxx+fyyyy)+O(h4),\nabla^2 f \approx \frac{f_E + f_W + f_N + f_S - 4 f_C}{h^2}, \qquad \text{error} = \frac{h^2}{12}\left(f_{xxxx} + f_{yyyy}\right) + O(h^4),

where CC is the center point and E,W,N,SE, W, N, S are its four neighbors. This is the five-point Laplacian, second-order accurate. It reads as “four times the difference between the neighbor average and the center”, the averaging picture you’ll develop in 2.11 The Laplacian & Vector Identities.

✏️ Pen & paperThe five-point Laplacian on a polynomial●●○
  1. Apply the five-point stencil to f(x,y)=x2+y2f(x, y) = x^2 + y^2 at any point with any hh. Compare with the exact ∇2f\nabla^2 f, and explain the result using the error term.
  2. Repeat for f(x,y)=x4f(x, y) = x^4 at (0,0)(0, 0).
Show worked solution
  1. fE+fW=(x+h)2+(x−h)2+2y2=2x2+2h2+2y2f_E + f_W = (x+h)^2 + (x-h)^2 + 2y^2 = 2x^2 + 2h^2 + 2y^2, and likewise fN+fS=2x2+2y2+2h2f_N + f_S = 2x^2 + 2y^2 + 2h^2. So the stencil gives (4x2+4y2+4h2−4(x2+y2))/h2=4\big(4x^2 + 4y^2 + 4h^2 - 4(x^2 + y^2)\big)/h^2 = 4. Exact: ∇2f=2+2=4\nabla^2 f = 2 + 2 = 4. The stencil is exact, because the error term involves fourth derivatives, which vanish for a quadratic.
  2. At the origin fC=0f_C = 0, fE=fW=h4f_E = f_W = h^4 and fN=fS=0f_N = f_S = 0, so the stencil gives 2h4/h2=2h22h^4/h^2 = 2h^2. The exact Laplacian is 12x2=012x^2 = 0 at the origin. The error is 2h22h^2, and the error formula predicts h212fxxxx=h212⋅24=2h2\tfrac{h^2}{12} f_{xxxx} = \tfrac{h^2}{12}\cdot 24 = 2h^2. ✓ (Here the formula is exact because all higher derivatives of x4x^4 vanish.)

Exercise: higher-order stencils and a convergence study

Now derive and test stencils yourself. The fourth-order 5-point formula for f′f' comes from applying Richardson to the central difference (or from undetermined coefficients with samples at x±h,x±2hx \pm h, x\pm 2h):

f′(x)≈−f(x+2h)+8f(x+h)−8f(x−h)+f(x−2h)12h,error=−h430f(5)(x)+O(h6).f'(x) \approx \frac{-f(x+2h) + 8f(x+h) - 8f(x-h) + f(x-2h)}{12h}, \qquad \text{error} = -\frac{h^4}{30}f^{(5)}(x) + O(h^6).
🧪 Coding exerciseExercise: implement stencils and measure their order
⌘/Ctrl + Enter

Key Takeaways

  • The degree-nn Taylor polynomial ∑k≤nf(k)(a)(x−a)k/k!\sum_{k\le n} f^{(k)}(a)(x-a)^k/k! matches ff‘s first nn derivatives at aa. Its error, the Lagrange remainder f(n+1)(ξ)hn+1/(n+1)!f^{(n+1)}(\xi)h^{n+1}/(n+1)!, looks like the first dropped term.
  • Big-O tracks how errors scale: an O(hp)O(h^p) error drops by 2p2^p when hh halves, and shows up as a line of slope pp on a log–log plot.
  • Finite differences come from combining Taylor expansions so unwanted terms cancel. Forward and backward differences are O(h)O(h); central and [1,−2,1]/h2[1,-2,1]/h^2 are O(h2)O(h^2); symmetry buys an order.
  • In floating point the total error is truncation plus round-off (∼ε/h\sim\varepsilon/h), giving a V-shaped curve with an optimal hh that is far from zero.
  • Richardson extrapolation combines two step sizes to cancel the known leading error term. The same idea powers error estimates in adaptive RK45.
  • A method’s order is set by the first Taylor term it gets wrong: local error O(hp+1)O(h^{p+1}) means global error O(hp)O(h^p). Euler matches through hh (order 1), midpoint and Heun through h2h^2, RK4 through h4h^4.
  • In several variables, f(a+h)≈f+∇f⋅h+12hTHhf(\mathbf a + \mathbf h) \approx f + \nabla f\cdot\mathbf h + \tfrac12\mathbf h^{\mathsf T}H\mathbf h: the gradient gives the tilt and the Hessian the curvature.
  • The five-point Laplacian and every other Navier–Stokes stencil is justified, and its accuracy predicted, by a Taylor expansion.
Finished this lesson?

Mark it complete to track your progress (saved in this browser). Reading every section does this for you.

Where this leads

These later lessons build directly on this one: