Flow Lab

Syllabus / Module 2: Multivariable & Vector Calculus

Lesson 2.3

Gradient & Directional Derivatives

The gradient points uphill and is perpendicular to level sets. Directional derivatives, gradient descent, and pressure-gradient forces.

Written byClaude Opus

🌊
Why this matters for simulation: Fluid accelerates from high to low pressure: the force per volume is −∇p.

By the end of this lesson you can…

  • Compute a directional derivative as the dot product ∇f · u and explain it with the chain rule.
  • Explain why the gradient points in the direction of steepest ascent and why it is perpendicular to level curves and level surfaces.
  • Use the gradient to write tangent lines to level curves and tangent planes to level surfaces.
  • Find critical points and classify them with the eigenvalues of the Hessian matrix.
  • Implement gradient descent, recognize it as forward Euler on x' = −∇f, and predict its largest stable step from the Hessian.
  • Derive the pressure-gradient force −∇p on a fluid parcel and connect it to the projection step of a fluid solver.
On this page
·
  1. Slopes in every direction: the directional derivative
  2. Worked example: a directional derivative
  3. The gradient points uphill
  4. The gradient is perpendicular to level sets
  5. Worked example: tangent lines and tangent planes from the gradient
  6. Critical points and the Hessian
  7. Why the eigenvalues decide
  8. Worked example: classifying critical points
  9. Gradient descent: rolling downhill
  10. It’s Euler’s method
  11. Why big steps blow up: an eigenvalue story
  12. Worked example: the narrow valley
  13. Your turn: implement it
  14. The pressure-gradient force
  15. Pen-and-paper problems
  16. Looking ahead
  17. Key Takeaways

Open a weather map and you’ll see isobars, the curves of equal pressure, wrapped around a big H and a big L. The wind tries to blow from the H toward the L, and it blows hardest where the isobars are packed close together. In a fluid simulator, that same push shows up as a single term in the momentum equation, −∇p-\nabla p, and in Stam’s solver an entire step (the projection) exists just to compute it. The symbol ∇p\nabla p is the gradient of the pressure: a vector built from partial derivatives that points in the direction where pp increases fastest.

In 2.1 Functions of Several Variables & Partial Derivatives you learned to measure slopes along the xx and yy axes, and in 2.2 Linearization, the Chain Rule & the Jacobian you learned that a differentiable function looks like a plane up close. This lesson asks the obvious next question: what is the slope in an arbitrary direction? The answer is a dot product (1.1 Vectors, Linear Combinations & Span), and from it comes everything else here. The gradient points uphill and cuts level curves at right angles. Following it downhill gives gradient descent, which turns out to be Euler’s method. And at the flat spots, the eigenvalues of a symmetric matrix tell you whether you’re at the bottom of a bowl, the top of a hill, or on a mountain pass.

▶ Video Part 1: slopes in every direction, the gradient, level curves and the Hessiancaptions only
Transcript
  1. In this lesson we meet the gradient: the arrow that tells you which way is uphill, and the single most useful vector in all of physics and optimization.
  2. Last time we linearized. Now take a function of two variables and picture it as a landscape: above every point x, y sits a height, f of x y.
  3. Stand at one point. You already know two slopes there: f x, heading east, and f y, heading north. But a hiker can set off in any direction at all.
  4. Pick a unit vector u for your heading. The slope you feel walking that way is the directional derivative, D u f. As u turns, the slope rises and falls.
  5. Here's the formula. Walk along the straight line a plus t u. By the chain rule, f changes at the rate f x times u one, plus f y times u two. That's a dot product.
  6. The vector of partial derivatives is the gradient, written nabla f. And because a dot product is length times length times cosine, the slope is the length of the gradient times cosine theta.
  7. So the slope is biggest when u points along the gradient. The gradient points straight uphill: the direction of steepest ascent, and its length is that steepest slope.
  8. Turn around, and you face steepest descent, minus the gradient. Turn a quarter turn, and the slope is exactly zero: you're walking along the hillside at constant height.
  9. Now float up and look straight down. The heights become level curves, the contour lines on a hiking map. Each one joins points at the same height.
  10. Draw the gradient at many points. Every arrow crosses the contours at a right angle, and points toward higher ground.
  11. Here's why. Walk along a level curve and your height never changes, so the directional derivative along the curve is zero. Zero dot product means perpendicular.
  12. Notice the lengths too. Where the contours crowd together the ground is steep and the arrows are long. Where they spread apart, the arrows shrink.
  13. Where the gradient is zero, the ground is flat in every direction. That's a critical point. It might be the bottom of a bowl,
  14. the top of a hill, or a mountain pass: a saddle, which goes up one way and down the other.
  15. The gradient is zero at all three, so first derivatives can't tell them apart. The second derivatives can. Collect them in the Hessian matrix. By Clairaut's theorem it is symmetric.
  16. A symmetric matrix has real eigenvalues and perpendicular eigenvectors. The eigenvectors are the principal directions, and each eigenvalue is the curvature along its direction. Here one curves up, the other curves down.
  17. So the rule is: both eigenvalues positive, a minimum. Both negative, a maximum. Opposite signs, a saddle. And if an eigenvalue is zero, the second derivative test is silent.
  18. To recap: the slope in direction u is the gradient dotted with u. The gradient points uphill, perpendicular to the level curves. Where it vanishes, the Hessian's eigenvalues classify the critical point.
  19. Next, we let a ball roll downhill. That's gradient descent, and it turns out to be Euler's method in disguise.

Slopes in every direction: the directional derivative

Stand at a point a⃗=(a1,a2)\vec a = (a_1, a_2) on the landscape z=f(x,y)z = f(x, y) and face in the direction of a unit vector u⃗=(u1,u2)\vec u = (u_1, u_2), with ∥u⃗∥=1\|\vec u\| = 1. Walk a distance tt in that direction and you reach a⃗+tu⃗\vec a + t\vec u. The directional derivative of ff at a⃗\vec a in the direction u⃗\vec u is the rate at which your height changes per unit distance walked:

Du⃗f(a⃗)=lim⁡t→0f(a⃗+tu⃗)−f(a⃗)t.D_{\vec u} f(\vec a) = \lim_{t \to 0} \frac{f(\vec a + t\vec u) - f(\vec a)}{t}.

The partial derivatives are special cases: fx=Du⃗ff_x = D_{\vec u} f with u⃗=(1,0)\vec u = (1, 0) (walk east), and fyf_y uses u⃗=(0,1)\vec u = (0, 1) (walk north).

You don’t need a new limit for every direction. Define the one-variable function g(t)=f(a1+tu1,  a2+tu2)g(t) = f(a_1 + t u_1,\; a_2 + t u_2). Then Du⃗f(a⃗)=g′(0)D_{\vec u} f(\vec a) = g'(0), and the multivariable chain rule from 2.2 Linearization, the Chain Rule & the Jacobian gives

g′(0)=∂f∂xdxdt+∂f∂ydydt=fx(a⃗) u1+fy(a⃗) u2.g'(0) = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt} = f_x(\vec a)\,u_1 + f_y(\vec a)\,u_2 .

That’s a dot product of the vector of partial derivatives with u⃗\vec u. The vector deserves a name.

The symbol ∇\nabla is called “nabla” or “del”. Think of it as the vector of derivative operators ∇=(∂x,∂y,∂z)\nabla = (\partial_x, \partial_y, \partial_z). Applied to a scalar field it gives the gradient; you’ll see it dotted with a vector field to make the divergence (2.7 Divergence: Sources & Sinks) and crossed with one to make the curl (2.8 Curl & Vorticity).

The gradient also packages the linearization from the last lesson in a compact form:

f(a⃗+h⃗)≈f(a⃗)+∇f(a⃗)⋅h⃗.f(\vec a + \vec h) \approx f(\vec a) + \nabla f(\vec a)\cdot \vec h .

Walking a small displacement h⃗\vec h changes the height by about ∇f⋅h⃗\nabla f \cdot \vec h. The directional derivative is just this with h⃗=tu⃗\vec h = t\vec u, divided by tt.

Worked example: a directional derivative

Find the rate of change of f(x,y)=x2+3xyf(x, y) = x^2 + 3xy at (1,2)(1, 2) in the direction of (3,4)(3, 4).

  1. Gradient: fx=2x+3yf_x = 2x + 3y, fy=3xf_y = 3x, so ∇f(1,2)=(2+6,  3)=(8,3)\nabla f(1,2) = (2 + 6,\; 3) = (8, 3).
  2. Unit direction: ∥(3,4)∥=5\|(3,4)\| = 5, so u⃗=(0.6,0.8)\vec u = (0.6, 0.8).
  3. Dot product: Du⃗f=8(0.6)+3(0.8)=4.8+2.4=7.2D_{\vec u} f = 8(0.6) + 3(0.8) = 4.8 + 2.4 = 7.2.

So walking in that direction, ff rises about 7.2 units per unit of distance. Sanity check with a small step t=0.001t = 0.001: f(1.0006,2.0008)≈7.0072f(1.0006, 2.0008) \approx 7.0072 versus f(1,2)=7f(1, 2) = 7, a change of 0.00720.0072, which divided by 0.0010.001 gives 7.2. ✓

✅ Concept check

For $f(x,y) = x^2 y$, compute the directional derivative at $(1, 3)$ in the direction of the unit vector $\vec u = (0.6, 0.8)$.

The gradient points uphill

Now use the geometric form of the dot product from 1.1 Vectors, Linear Combinations & Span. If θ\theta is the angle between ∇f\nabla f and u⃗\vec u, then because ∥u⃗∥=1\|\vec u\| = 1,

Du⃗f=∥∇f∥ ∥u⃗∥cos⁡θ=∥∇f∥cos⁡θ.D_{\vec u} f = \|\nabla f\|\,\|\vec u\|\cos\theta = \|\nabla f\| \cos\theta .

As you turn on the spot, the slope you feel traces out a cosine. That single formula has three consequences:

  • Steepest ascent. The slope is largest when cos⁡θ=1\cos\theta = 1, that is when u⃗\vec u points along ∇f\nabla f. The maximum slope is ∥∇f∥\|\nabla f\|. So the gradient points in the direction of steepest ascent, and its length is the steepest slope.
  • Steepest descent. The slope is most negative, −∥∇f∥-\|\nabla f\|, when u⃗\vec u points along −∇f-\nabla f.
  • Level directions. The slope is zero when θ=90∘\theta = 90^\circ: walking perpendicular to the gradient keeps you at constant height, to first order.

In the worked example above, ∥∇f(1,2)∥=64+9=73≈8.54\|\nabla f(1, 2)\| = \sqrt{64 + 9} = \sqrt{73} \approx 8.54. No direction gives a slope larger than 8.54, and the direction (3,4)/5(3, 4)/5 achieved 7.2 because it’s only about 32∘32^\circ away from the gradient.

✅ Concept check

At a point where $\nabla f = (3, -4)$, which statement is correct?

The gradient is perpendicular to level sets

A level curve of f(x,y)f(x, y) is a curve where ff equals a constant cc: a contour line on a hiking map, or an isobar on a weather map. In 3D, f(x,y,z)=cf(x,y,z) = c is a level surface, like a surface of constant temperature in a room.

Claim: at every point where ∇f≠0⃗\nabla f \neq \vec 0, the gradient is perpendicular to the level set through that point.

Derivation. Let r⃗(t)\vec r(t) be any curve that stays inside the level set, so f(r⃗(t))=cf(\vec r(t)) = c for all tt. Differentiate both sides with the chain rule:

ddtf(r⃗(t))=∇f(r⃗(t))⋅r⃗ ′(t)=0.\frac{d}{dt} f(\vec r(t)) = \nabla f(\vec r(t)) \cdot \vec r\,'(t) = 0 .

The vector r⃗ ′(t)\vec r\,'(t) is tangent to the level set, so the gradient is perpendicular to every tangent direction of the level set. In 2D that means perpendicular to the level curve; in 3D, ∇f\nabla f is a normal vector to the level surface.

Combined with steepest ascent, this gives a picture you can read off any contour map. The gradient crosses contours at right angles, pointing toward higher values. And since the slope is the change in height divided by the distance between contours, the gradient is long where contours are crowded and short where they are spread apart.

Worked example: tangent lines and tangent planes from the gradient

(a) Find the tangent line to the ellipse x2+4y2=8x^2 + 4y^2 = 8 at (2,1)(2, 1).

The ellipse is the level curve f=8f = 8 of f(x,y)=x2+4y2f(x,y) = x^2 + 4y^2. Check: 4+4=84 + 4 = 8. ✓ The gradient is ∇f=(2x,8y)=(4,8)\nabla f = (2x, 8y) = (4, 8) at (2,1)(2,1), and it’s normal to the curve. A line through (2,1)(2,1) with normal (4,8)(4, 8) is

4(x−2)+8(y−1)=0⟺x+2y=4.4(x - 2) + 8(y - 1) = 0 \quad\Longleftrightarrow\quad x + 2y = 4 .

Check by implicit differentiation: 2x+8y y′=02x + 8y\,y' = 0 gives y′=−x/(4y)=−1/2y' = -x/(4y) = -1/2 at (2,1)(2,1), and the line x+2y=4x + 2y = 4 has slope −1/2-1/2. ✓

(b) Find the tangent plane to the sphere x2+y2+z2=9x^2 + y^2 + z^2 = 9 at (1,2,2)(1, 2, 2).

The normal is ∇F=(2x,2y,2z)=(2,4,4)\nabla F = (2x, 2y, 2z) = (2, 4, 4). The plane through (1,2,2)(1,2,2) with this normal is 2(x−1)+4(y−2)+4(z−2)=02(x-1) + 4(y-2) + 4(z-2) = 0, which simplifies to x+2y+2z=9x + 2y + 2z = 9. The normal points straight out from the center, as it should for a sphere.

Now explore it yourself. The sim shows a contour map (color = height) with the gradient field drawn on top. The dashed line through the ball is the level curve’s tangent, and the amber arrow is ∇f\nabla f.

◆ Interactive Gradient landscape: click to place the ball, then run gradient descent

Loading simulation…

Things to try now (we’ll come back for the descent part):

  1. Click around the elongated bowl. Check that the amber arrow is always perpendicular to the dashed level-curve tangent, and that it’s longest at the top and bottom of the map, where the contours crowd together.
  2. Switch to Four wells. The gradient arrows all point away from the four minima and toward the single maximum near the middle. Find the points marked “saddle”: the arrows nearby point toward the saddle along one line and away from it along another.
  3. Turn on Exact gradient flow. The green curve follows −∇f-\nabla f, so it crosses every contour at a right angle on its way down.
✅ Concept check

On a contour map, the contour lines near point A are spaced 1 mm apart and near point B they are 4 mm apart (same height step). What can you say about the gradients?

Critical points and the Hessian

A critical point of ff is a point where ∇f=0⃗\nabla f = \vec 0. There the tangent plane is horizontal and every directional derivative is zero. For a smooth function, every local minimum and local maximum is a critical point. But not every critical point is an extremum, and in two dimensions there’s a third possibility, the saddle point, which goes up in some directions and down in others, like a mountain pass or a Pringle.

The first derivatives vanish at all three kinds, so we need second derivatives. Collect them in the Hessian matrix:

H(a⃗)=[fxxfxyfyxfyy].H(\vec a) = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix}.

By Clairaut’s theorem (2.1 Functions of Several Variables & Partial Derivatives), fxy=fyxf_{xy} = f_{yx} for smooth ff, so HH is symmetric. Notice what it is: the Jacobian matrix (2.2 Linearization, the Chain Rule & the Jacobian) of the gradient map x⃗↦∇f(x⃗)\vec x \mapsto \nabla f(\vec x). So near any point, ∇f(a⃗+h⃗)≈∇f(a⃗)+Hh⃗\nabla f(\vec a + \vec h) \approx \nabla f(\vec a) + H\vec h.

Why the eigenvalues decide

Near a critical point a⃗\vec a, the second-order Taylor expansion (derived carefully in 2.4 Taylor Series in One and Several Variables) is

f(a⃗+h⃗)≈f(a⃗)+∇f(a⃗)⋅h⃗⏟= 0+12 h⃗THh⃗.f(\vec a + \vec h) \approx f(\vec a) + \underbrace{\nabla f(\vec a)\cdot\vec h}_{=\,0} + \tfrac12\, \vec h^{\mathsf T} H \vec h .

So the shape near the critical point is the shape of the quadratic form 12h⃗THh⃗\tfrac12 \vec h^{\mathsf T} H \vec h. Here the spectral theorem from 1.6 Change of Basis & Diagonalization does all the work. A real symmetric HH has real eigenvalues λ1,λ2\lambda_1, \lambda_2 and perpendicular unit eigenvectors q⃗1,q⃗2\vec q_1, \vec q_2. Write the displacement in that basis, h⃗=s1q⃗1+s2q⃗2\vec h = s_1\vec q_1 + s_2 \vec q_2. Since Hq⃗i=λiq⃗iH\vec q_i = \lambda_i \vec q_i and q⃗1⋅q⃗2=0\vec q_1 \cdot \vec q_2 = 0,

12 h⃗THh⃗=12(λ1s12+λ2s22).\tfrac12\,\vec h^{\mathsf T} H \vec h = \tfrac12\left(\lambda_1 s_1^2 + \lambda_2 s_2^2\right).

In the principal directions q⃗1,q⃗2\vec q_1, \vec q_2, the surface is a parabola with curvature λ1\lambda_1 along one axis and λ2\lambda_2 along the other, with no cross term. That gives the second-derivative test:

In 2D there is a shortcut that avoids computing eigenvalues. From 1.5 Eigenvectors & Eigenvalues, det⁡H=λ1λ2\det H = \lambda_1\lambda_2 and tr⁡H=λ1+λ2\operatorname{tr} H = \lambda_1 + \lambda_2. Let D=fxxfyy−fxy2=det⁡HD = f_{xx}f_{yy} - f_{xy}^2 = \det H:

  • D>0D \gt 0: the eigenvalues have the same sign, and that sign is the sign of fxxf_{xx} (or of the trace). So it’s a min if fxx>0f_{xx} \gt 0 and a max if fxx<0f_{xx} \lt 0.
  • D<0D \lt 0: opposite signs, so it’s a saddle.
  • D=0D = 0: inconclusive.

This is the ”DD test” from Calc III textbooks. Now you know where it comes from.

Worked example: classifying critical points

Find and classify the critical points of f(x,y)=x3+y3−3xyf(x, y) = x^3 + y^3 - 3xy.

Critical points. ∇f=(3x2−3y,  3y2−3x)\nabla f = (3x^2 - 3y,\; 3y^2 - 3x). Setting both to zero gives y=x2y = x^2 and x=y2x = y^2. Substituting, x=x4x = x^4, so x(x3−1)=0x(x^3 - 1) = 0 and x=0x = 0 or x=1x = 1 (the only real roots). The critical points are (0,0)(0, 0) and (1,1)(1, 1).

Hessian. fxx=6xf_{xx} = 6x, fxy=−3f_{xy} = -3, fyy=6yf_{yy} = 6y:

H(x,y)=[6x−3−36y].H(x,y) = \begin{bmatrix} 6x & -3 \\ -3 & 6y \end{bmatrix}.

At (0,0)(0,0): H=[0−3−30]H = \begin{bmatrix} 0 & -3 \\ -3 & 0 \end{bmatrix}. The characteristic polynomial is λ2−9\lambda^2 - 9, so λ=±3\lambda = \pm 3. Opposite signs: saddle. The eigenvectors are (1,1)(1,1) for λ=−3\lambda = -3 and (1,−1)(1,-1) for λ=3\lambda = 3: the surface curves down along the diagonal y=xy = x and up along y=−xy = -x.

At (1,1)(1,1): H=[6−3−36]H = \begin{bmatrix} 6 & -3 \\ -3 & 6 \end{bmatrix}, with λ=6±3\lambda = 6 \pm 3, so λ=3\lambda = 3 and λ=9\lambda = 9. Both positive: local minimum, with f(1,1)=1+1−3=−1f(1,1) = 1 + 1 - 3 = -1. The DD test agrees: D=36−9=27>0D = 36 - 9 = 27 \gt 0 and fxx=6>0f_{xx} = 6 \gt 0.

It’s only a local minimum: along the line y=x→−∞y = x \to -\infty, f≈2x3→−∞f \approx 2x^3 \to -\infty. Pick the “Saddle + minimum” landscape in the sim above to see both points and the open valley running off to the lower left.

✅ Concept check

At a critical point, the Hessian is $\begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}$. What kind of point is it?

Gradient descent: rolling downhill

Suppose you want the minimum of ff: the lowest-energy configuration of a structure, the best-fitting parameters of a model, or (soon) the pressure that makes a velocity field divergence-free. Since −∇f-\nabla f is the direction of steepest descent, the simplest algorithm is to take a step that way, recompute, and repeat:

x⃗n+1=x⃗n−α ∇f(x⃗n).\vec x_{n+1} = \vec x_n - \alpha\,\nabla f(\vec x_n).

The positive number α\alpha is the step size (machine learning calls it the learning rate). This is gradient descent.

▶ Video Part 2: gradient descent as Euler's method, step-size stability, and the pressure-gradient forcecaptions only
Transcript
  1. Part two. We follow the gradient downhill, find out when that goes wrong, and then see the gradient pushing real fluid around.
  2. Here's the contour map of a long, narrow valley: f equals x squared plus five y squared. Drop a ball on the hillside and let it roll down.
  3. Gradient descent: from where you stand, take a step of size alpha times the gradient, downhill. x new equals x old minus alpha grad f. Repeat.
  4. Look closely at that update. It is forward Euler, with time step alpha, applied to the differential equation x dot equals minus grad f. The green curve is that equation's exact solution, the gradient flow.
  5. Along the exact flow, f can only go down: by the chain rule, d f d t is the gradient dotted with the velocity, which is minus the gradient's length squared.
  6. Bigger steps get there faster, up to a point. With alpha equal to point one five, the ball overshoots the valley floor every step and zigzags its way down.
  7. With alpha equal to point two two, each zigzag is bigger than the last, and the ball is flung right out of the valley. The method has gone unstable.
  8. Why point two two? For this bowl the gradient is the Hessian times x. So each step multiplies x by the matrix I minus alpha H, whose eigenvalues are one minus alpha times each eigenvalue of H.
  9. Every one of those multipliers must be smaller than one in size. The steep direction, with lambda equal to ten, sets the limit: alpha below two over ten. It's exactly the rule for Euler's method on a stiff problem.
  10. And notice the trap: the steep direction caps the step, so progress along the gentle direction crawls. A narrow valley, with very different eigenvalues, is slow to descend. That ratio is called the condition number.
  11. Now physics. Here is a pressure field: high pressure on the left, low pressure on the right. The circles are isobars, level curves of the pressure.
  12. Take a tiny box of fluid. Pressure pushes inward on every face. The push on the left face is stronger than the push on the right, so the net force points right: from high pressure toward low.
  13. The net force is p at x minus p at x plus d x, times the face area. That's minus d p d x times the volume. Do the same in every direction, and the force per unit volume is minus the pressure gradient.
  14. Draw minus grad p everywhere. The arrows run from high to low, crossing the isobars at right angles, strongest where the isobars crowd together. That is what drives the wind.
  15. In a fluid solver, the pressure gradient is the enforcer. Stam's projection step solves for a pressure, then subtracts its gradient from the velocity, which is exactly what keeps the simulated fluid incompressible.
  16. To recap: gradient descent is Euler's method on the gradient flow, stable only when alpha is below two over the largest Hessian eigenvalue. And fluid is pushed by minus the pressure gradient.
  17. Next time: Taylor series in several variables, where the gradient and the Hessian turn out to be the first two terms.

It’s Euler’s method

Look at the update next to the ODE

x⃗˙=−∇f(x⃗).\dot{\vec x} = -\nabla f(\vec x).

Forward Euler with time step hh gives x⃗n+1=x⃗n+h (−∇f(x⃗n))\vec x_{n+1} = \vec x_n + h\,(-\nabla f(\vec x_n)), which is gradient descent with α=h\alpha = h. Gradient descent is forward Euler applied to the gradient flow. The exact solutions of the gradient flow are the curves of steepest descent, the green curves in the sim, and they always go downhill. By the chain rule,

ddtf(x⃗(t))=∇f⋅x⃗˙=−∥∇f∥2≤0,\frac{d}{dt} f(\vec x(t)) = \nabla f \cdot \dot{\vec x} = -\|\nabla f\|^2 \le 0 ,

with equality only at critical points. So ff itself acts as an energy that the flow steadily drains. That idea, a function that can only decrease along trajectories, becomes the Lyapunov function in 7.8 Conservative Systems & Lyapunov Functions.

Everything you learned about Euler in your Simulations course now applies to optimization. Small steps track the smooth flow. Big steps can overshoot, and too-big steps blow up. Better integrators suggest better optimizers too: momentum methods behave like integrating a damped second-order ODE.

Why big steps blow up: an eigenvalue story

Near a minimum x⃗∗\vec x^*, linearize the gradient using the Hessian: ∇f(x⃗)≈H(x⃗−x⃗∗)\nabla f(\vec x) \approx H(\vec x - \vec x^*). Write e⃗n=x⃗n−x⃗∗\vec e_n = \vec x_n - \vec x^* for the error. Then

e⃗n+1=e⃗n−αHe⃗n=(I−αH) e⃗n.\vec e_{n+1} = \vec e_n - \alpha H\vec e_n = (I - \alpha H)\,\vec e_n .

This is the matrix iteration from 1.5 Eigenvectors & Eigenvalues. I−αHI - \alpha H has the same eigenvectors as HH and eigenvalues 1−αλi1 - \alpha\lambda_i. In the eigenbasis of HH the iteration decouples: the error component along q⃗i\vec q_i gets multiplied by 1−αλi1 - \alpha\lambda_i every step. At a minimum all λi>0\lambda_i \gt 0, and the iteration converges exactly when every multiplier has size less than 1:

∣1−αλi∣<1 for all i⟺0<α<2λmax⁡.|1 - \alpha\lambda_i| \lt 1 \text{ for all } i \quad\Longleftrightarrow\quad 0 \lt \alpha \lt \frac{2}{\lambda_{\max}} .

That’s the same bound as Euler’s h<2/∣λ∣h \lt 2/|\lambda| on y′=λyy' = \lambda y. It’s no coincidence: the linearized gradient flow is e⃗˙=−He⃗\dot{\vec e} = -H\vec e, a linear system with eigenvalues −λi-\lambda_i.

The multiplier also tells you how each component behaves:

αλi\alpha\lambda_imultiplier 1−αλi1 - \alpha\lambda_ibehavior in direction q⃗i\vec q_i
between 0 and 1between 0 and 1smooth, monotone approach
exactly 10lands on the minimum in one step
between 1 and 2between −1 and 0overshoots, zigzags, but shrinks
greater than 2less than −1zigzags with growing amplitude: divergence

Worked example: the narrow valley

Run gradient descent on f(x,y)=x2+5y2f(x,y) = x^2 + 5y^2 from (−3,1.5)(-3, 1.5).

Here ∇f=(2x,10y)\nabla f = (2x, 10y) and H=diag⁡(2,10)H = \operatorname{diag}(2, 10), so the update is exactly linear:

xn+1=(1−2α) xn,yn+1=(1−10α) yn.x_{n+1} = (1 - 2\alpha)\,x_n, \qquad y_{n+1} = (1 - 10\alpha)\,y_n .
  • Stability limit: λmax⁡=10\lambda_{\max} = 10, so we need α<0.2\alpha \lt 0.2.
  • α=0.15\alpha = 0.15: the multipliers are 0.70.7 and −0.5-0.5. The first step is ∇f(−3,1.5)=(−6,15)\nabla f(-3, 1.5) = (-6, 15), so x⃗1=(−3,1.5)−0.15(−6,15)=(−2.1,−0.75)\vec x_1 = (-3, 1.5) - 0.15(-6, 15) = (-2.1, -0.75). Then x⃗2=(−1.47,0.375)\vec x_2 = (-1.47, 0.375). The yy-coordinate flips sign every step while it halves: that’s the zigzag.
  • α=0.22\alpha = 0.22: the yy-multiplier is 1−2.2=−1.21 - 2.2 = -1.2, so ∣yn∣|y_n| grows by 20% per step even though xnx_n shrinks. Divergence.
  • Best fixed step: the slowest component decides the convergence rate, max⁡(∣1−2α∣,∣1−10α∣)\max(|1 - 2\alpha|, |1 - 10\alpha|). This is smallest when the two are equal, 1−2α=10α−11 - 2\alpha = 10\alpha - 1, giving α=1/6\alpha = 1/6 and a rate of 2/32/3 per step.

In general the best rate is κ−1κ+1\dfrac{\kappa - 1}{\kappa + 1}, where κ=λmax⁡/λmin⁡\kappa = \lambda_{\max}/\lambda_{\min} is the condition number of HH. Here κ=5\kappa = 5 gives 4/6=2/34/6 = 2/3. ✓ A long, narrow valley has a large κ\kappa: the steep walls cap the step size while progress along the gentle floor crawls.

Back to the sim. On the elongated bowl, set α=0.08\alpha = 0.08 and press Run descent. Then try 0.150.15, 0.190.19 and 0.210.21, and watch the multipliers in the readout. On the curved valley, the Hessian changes from point to point, so the safe step changes too: the readout shows the local αλmax⁡\alpha\lambda_{\max} as the ball moves.

🐍 PythonStep size vs. stability on x² + 5y²
⌘/Ctrl + Enter
✅ Concept check

Gradient descent is run on $f(x,y) = 3x^2 + \tfrac12 y^2$. What is the largest step size $\alpha$ for which it converges (from any starting point)? Give a decimal.

Your turn: implement it

🧪 Coding exerciseExercise: gradient descent and Hessian classification
⌘/Ctrl + Enter

The pressure-gradient force

Now the physics that motivated the lesson. Take a tiny box of fluid with sides dxdx, dydy, dzdz, centered at (x,y,z)(x, y, z). The surrounding fluid pushes on every face with force per area equal to the pressure pp, always directed into the box.

Look at the two faces perpendicular to xx. Each has area dy dzdy\,dz. The left face, at x−dx/2x - dx/2, is pushed in the +x+x direction with force p(x−dx/2) dy dzp(x - dx/2)\,dy\,dz. The right face, at x+dx/2x + dx/2, is pushed in the −x-x direction with force p(x+dx/2) dy dzp(x + dx/2)\,dy\,dz. The net xx-force is

Fx=[p(x−dx2)−p(x+dx2)] dy dz≈−∂p∂x dx dy dz=−∂p∂x dV.F_x = \Big[p\big(x - \tfrac{dx}{2}\big) - p\big(x + \tfrac{dx}{2}\big)\Big]\,dy\,dz \approx -\frac{\partial p}{\partial x}\,dx\,dy\,dz = -\frac{\partial p}{\partial x}\,dV .

The same argument in yy and zz gives the other components, so the net pressure force per unit volume is

f⃗pressure=−∇p.\vec f_{\text{pressure}} = -\nabla p .

A uniform pressure, however large, produces no net force: only differences in pressure push fluid around. The force points from high pressure toward low, perpendicular to the isobars, and it’s strongest where the isobars are packed together. That’s exactly the weather-map rule from the introduction. (On Earth, the Coriolis effect then deflects the wind until it blows nearly along the isobars, but −∇p-\nabla p is the driver.)

Newton’s second law for a fluid parcel of density ρ\rho, ignoring viscosity and gravity, becomes

ρ Du⃗Dt=−∇p,\rho\,\frac{D\vec u}{Dt} = -\nabla p ,

where D/DtD/Dt is the material derivative from 2.2 Linearization, the Chain Rule & the Jacobian. That is the Euler equation of fluid motion, which you’ll derive properly in 11.3 Momentum: Pressure & the Euler Equations.

Hydrostatics. In a fluid at rest under gravity, the pressure force must balance the weight: −∇p+ρg⃗=0⃗-\nabla p + \rho\vec g = \vec 0 with g⃗=(0,0,−g)\vec g = (0, 0, -g). That gives ∂p/∂z=−ρg\partial p/\partial z = -\rho g, so pressure increases linearly with depth, about one atmosphere for every 10 m of water.

🐍 PythonThe pressure-gradient force on a grid
⌘/Ctrl + Enter

Pen-and-paper problems

✏️ Pen & paperDirectional derivatives and steepest ascent●○○

Let f(x,y)=xeyf(x, y) = x e^{y} and P=(2,0)P = (2, 0).

  1. Find ∇f(P)\nabla f(P).
  2. Find the rate of change of ff at PP in the direction toward the point (5,4)(5, 4).
  3. In which direction does ff increase fastest at PP, and what is that rate?
  4. Find the directions in which the rate of change is zero.
Show worked solution
  1. fx=eyf_x = e^y and fy=xeyf_y = x e^y, so ∇f(2,0)=(1,2)\nabla f(2, 0) = (1, 2).
  2. The direction from (2,0)(2,0) to (5,4)(5,4) is (3,4)(3, 4), with length 5, so u⃗=(0.6,0.8)\vec u = (0.6, 0.8). Then Du⃗f=1(0.6)+2(0.8)=2.2D_{\vec u} f = 1(0.6) + 2(0.8) = 2.2.
  3. Along ∇f/∥∇f∥=(1,2)/5\nabla f / \|\nabla f\| = (1, 2)/\sqrt5, at the rate ∥∇f∥=5≈2.236\|\nabla f\| = \sqrt5 \approx 2.236. (The direction in part 2 was close to this, which is why 2.2 is nearly the maximum.)
  4. Perpendicular to (1,2)(1, 2): u⃗=±(2,−1)/5\vec u = \pm(2, -1)/\sqrt5. These are the two directions along the level curve xey=2x e^y = 2 through PP.
✏️ Pen & paperTangent plane from a level surface●●○

The paraboloid z=x2+y2z = x^2 + y^2 can be written as the level surface F(x,y,z)=x2+y2−z=0F(x, y, z) = x^2 + y^2 - z = 0.

  1. Find a normal vector to the paraboloid at (1,2,5)(1, 2, 5).
  2. Write the equation of the tangent plane there and solve it for zz.
  3. Compare with the linearization L(x,y)=f(1,2)+fx(1,2)(x−1)+fy(1,2)(y−2)L(x, y) = f(1,2) + f_x(1,2)(x - 1) + f_y(1,2)(y - 2) of f(x,y)=x2+y2f(x, y) = x^2 + y^2 from 2.2 Linearization, the Chain Rule & the Jacobian.
Show worked solution
  1. ∇F=(2x,2y,−1)=(2,4,−1)\nabla F = (2x, 2y, -1) = (2, 4, -1) at (1,2,5)(1, 2, 5).
  2. 2(x−1)+4(y−2)−(z−5)=02(x - 1) + 4(y - 2) - (z - 5) = 0, so z=2x+4y−5z = 2x + 4y - 5.
  3. f(1,2)=5f(1,2) = 5, fx=2x=2f_x = 2x = 2 and fy=2y=4f_y = 2y = 4, so L(x,y)=5+2(x−1)+4(y−2)=2x+4y−5L(x,y) = 5 + 2(x-1) + 4(y-2) = 2x + 4y - 5. They’re the same plane. In general, the graph z=f(x,y)z = f(x,y) is the level surface f(x,y)−z=0f(x,y) - z = 0, whose normal is (fx,fy,−1)(f_x, f_y, -1), and that’s exactly the tangent plane of the linearization.
✏️ Pen & paperClassify the critical points●●○

Find and classify all critical points of f(x,y)=x4+y4−4xyf(x, y) = x^4 + y^4 - 4xy. Give the Hessian eigenvalues at each.

Show worked solution

∇f=(4x3−4y,  4y3−4x)\nabla f = (4x^3 - 4y,\; 4y^3 - 4x). Setting it to zero gives y=x3y = x^3 and x=y3x = y^3, so x=x9x = x^9, i.e. x(x8−1)=0x(x^8 - 1) = 0. The real solutions are x=0,1,−1x = 0, 1, -1, giving the critical points (0,0)(0,0), (1,1)(1,1) and (−1,−1)(-1,-1).

The Hessian is

H=[12x2−4−412y2].H = \begin{bmatrix} 12x^2 & -4 \\ -4 & 12y^2 \end{bmatrix}.
  • At (0,0)(0,0): H=[0−4−40]H = \begin{bmatrix} 0 & -4 \\ -4 & 0\end{bmatrix}, with eigenvalues ±4\pm 4. Saddle, with f=0f = 0.
  • At (1,1)(1,1): H=[12−4−412]H = \begin{bmatrix} 12 & -4 \\ -4 & 12\end{bmatrix}, with eigenvalues 12±412 \pm 4, i.e. 88 and 1616. Local minimum, with f(1,1)=1+1−4=−2f(1,1) = 1 + 1 - 4 = -2.
  • At (−1,−1)(-1,-1): the same Hessian, so also a local minimum with f=−2f = -2.

Since f→+∞f \to +\infty in every direction (the quartic terms win), these two are the global minima. The surface is two bowls joined by a pass at the origin.

✏️ Pen & paperTuning the step size●●●

Let f(x,y)=32x2+xy+32y2f(x, y) = \tfrac32 x^2 + xy + \tfrac32 y^2.

  1. Show that the origin is the only critical point and that it is a minimum. Find the eigenvalues and eigenvectors of the Hessian.
  2. What is the largest step size for which gradient descent converges?
  3. Which fixed step size gives the fastest guaranteed convergence, and what is the error reduction factor per step?
  4. Starting from (1,1)(1, 1) with α=0.25\alpha = 0.25, compute one step. Explain the result with eigenvectors.
Show worked solution
  1. ∇f=(3x+y,  x+3y)\nabla f = (3x + y,\; x + 3y). Setting this to zero gives a linear system with determinant 9−1=8≠09 - 1 = 8 \neq 0, so the only solution is (0,0)(0,0). The Hessian is constant, H=[3113]H = \begin{bmatrix} 3 & 1 \\ 1 & 3\end{bmatrix}, with eigenvalues 44 (eigenvector (1,1)(1,1)) and 22 (eigenvector (1,−1)(1,-1)). Both are positive, so it’s a minimum, and the eigenvectors are perpendicular as the spectral theorem promises.
  2. α<2/λmax⁡=2/4=0.5\alpha \lt 2/\lambda_{\max} = 2/4 = 0.5.
  3. The rate is max⁡(∣1−2α∣,∣1−4α∣)\max(|1 - 2\alpha|, |1 - 4\alpha|). The first term decreases and the second eventually increases as α\alpha grows, so the best α\alpha balances them: 1−2α=4α−11 - 2\alpha = 4\alpha - 1 gives α=1/3\alpha = 1/3. The rate is then 1/31/3 per step, which matches (κ−1)/(κ+1)(\kappa - 1)/(\kappa + 1) with κ=2\kappa = 2.
  4. ∇f(1,1)=(4,4)\nabla f(1,1) = (4, 4), so x⃗1=(1,1)−0.25(4,4)=(0,0)\vec x_1 = (1,1) - 0.25(4,4) = (0, 0). It lands exactly on the minimum. The start (1,1)(1,1) is an eigenvector of HH with λ=4\lambda = 4, so the only multiplier involved is 1−αλ=1−0.25⋅4=01 - \alpha\lambda = 1 - 0.25\cdot 4 = 0. From a generic starting point, the (1,−1)(1,-1) component would still shrink only by 1−0.5=0.51 - 0.5 = 0.5 per step.
✏️ Pen & paperWhy vortex cores have low pressure●●○

A fluid of density ρ\rho rotates like a rigid body with angular speed Ω\Omega: the velocity is u⃗=Ω(−y,x)\vec u = \Omega(-y, x), so a parcel at radius rr moves in a circle at speed Ωr\Omega r. Suppose the pressure is p(x,y)=p0+12ρ Ω2(x2+y2)p(x, y) = p_0 + \tfrac12\rho\,\Omega^2 (x^2 + y^2).

  1. Compute the pressure force per unit volume −∇p-\nabla p. Which way does it point, and how large is it at radius rr?
  2. A parcel moving in a circle of radius rr at speed vv needs a centripetal acceleration v2/rv^2/r toward the center. Show that −∇p/ρ-\nabla p/\rho supplies exactly this.
  3. Where is the pressure lowest? What does this suggest about the core of a tornado, or a vortex in a smoke simulation?
Show worked solution
  1. ∇p=ρΩ2(x,y)\nabla p = \rho\Omega^2(x, y), so −∇p=−ρΩ2(x,y)-\nabla p = -\rho\Omega^2(x, y). It points toward the center, with magnitude ρ Ω2r\rho\,\Omega^2 r.
  2. The needed acceleration is v2/r=(Ωr)2/r=Ω2rv^2/r = (\Omega r)^2/r = \Omega^2 r toward the center. The acceleration provided is ∥−∇p∥/ρ=Ω2r\|-\nabla p\|/\rho = \Omega^2 r, also toward the center. They match exactly, so this pressure field is precisely what keeps the parcels on their circles.
  3. The pressure is lowest at the center, p=p0p = p_0, and grows like r2r^2 outward. Any swirling flow needs an inward pressure force, so vortex cores are low-pressure regions. That’s why tornado funnels and wingtip vortices often show condensation (lower pressure means cooler air), and why pressure minima are a quick way to locate vortices in simulation output.

Looking ahead

The gradient is the first of the three great differential operators. Lesson 2.4 Taylor Series in One and Several Variables will show that ∇f\nabla f and the Hessian are the first- and second-order terms of the multivariable Taylor expansion. That’s the result we borrowed in the critical-point test, and it’s the tool behind every finite-difference error estimate. In 2.6 Vector Fields, Flow Lines & Streamlines, gradient fields F⃗=∇ϕ\vec F = \nabla\phi get their own name, conservative fields, and their flow lines are the steepest-descent curves you’ve been drawing. And in 7.7 Nonlinear Systems & Linearization, linearizing x⃗˙=−∇f\dot{\vec x} = -\nabla f at a critical point gives the Jacobian −H-H, so a minimum becomes a stable equilibrium and a saddle stays a saddle.

Key Takeaways

  • The directional derivative is a dot product: Du⃗f=∇f⋅u⃗D_{\vec u} f = \nabla f\cdot\vec u for a unit vector u⃗\vec u. It follows from the chain rule along the line a⃗+tu⃗\vec a + t\vec u.
  • Since Du⃗f=∥∇f∥cos⁡θD_{\vec u} f = \|\nabla f\|\cos\theta, the gradient points in the direction of steepest ascent, and its length is the steepest slope. −∇f-\nabla f is steepest descent.
  • The gradient is perpendicular to level curves and level surfaces, so it supplies normal vectors for tangent lines and tangent planes. It’s long where contours crowd together.
  • At a critical point (∇f=0⃗\nabla f = \vec 0), the eigenvalues of the symmetric Hessian are the principal curvatures: all positive means a minimum, all negative a maximum, mixed signs a saddle, and a zero eigenvalue is inconclusive.
  • Gradient descent x⃗n+1=x⃗n−α∇f(x⃗n)\vec x_{n+1} = \vec x_n - \alpha\nabla f(\vec x_n) is forward Euler on the gradient flow x⃗˙=−∇f\dot{\vec x} = -\nabla f.
  • Near a minimum, each step multiplies the error by I−αHI - \alpha H, so descent converges only for α<2/λmax⁡\alpha \lt 2/\lambda_{\max}, and it’s slow when the condition number λmax⁡/λmin⁡\lambda_{\max}/\lambda_{\min} is large.
  • The net pressure force per unit volume is −∇p-\nabla p: fluid is pushed from high to low pressure, and the projection step of a fluid solver subtracts exactly this gradient.
Finished this lesson?

Mark it complete to track your progress (saved in this browser). Reading every section does this for you.

Where this leads

These later lessons build directly on this one: