Open a weather map and you’ll see isobars, the curves of equal pressure, wrapped around a big H and a big L. The wind tries to blow from the H toward the L, and it blows hardest where the isobars are packed close together. In a fluid simulator, that same push shows up as a single term in the momentum equation, , and in Stam’s solver an entire step (the projection) exists just to compute it. The symbol is the gradient of the pressure: a vector built from partial derivatives that points in the direction where increases fastest.
In 2.1 Functions of Several Variables & Partial Derivatives you learned to measure slopes along the and axes, and in 2.2 Linearization, the Chain Rule & the Jacobian you learned that a differentiable function looks like a plane up close. This lesson asks the obvious next question: what is the slope in an arbitrary direction? The answer is a dot product (1.1 Vectors, Linear Combinations & Span), and from it comes everything else here. The gradient points uphill and cuts level curves at right angles. Following it downhill gives gradient descent, which turns out to be Euler’s method. And at the flat spots, the eigenvalues of a symmetric matrix tell you whether you’re at the bottom of a bowl, the top of a hill, or on a mountain pass.
Transcript
- In this lesson we meet the gradient: the arrow that tells you which way is uphill, and the single most useful vector in all of physics and optimization.
- Last time we linearized. Now take a function of two variables and picture it as a landscape: above every point x, y sits a height, f of x y.
- Stand at one point. You already know two slopes there: f x, heading east, and f y, heading north. But a hiker can set off in any direction at all.
- Pick a unit vector u for your heading. The slope you feel walking that way is the directional derivative, D u f. As u turns, the slope rises and falls.
- Here's the formula. Walk along the straight line a plus t u. By the chain rule, f changes at the rate f x times u one, plus f y times u two. That's a dot product.
- The vector of partial derivatives is the gradient, written nabla f. And because a dot product is length times length times cosine, the slope is the length of the gradient times cosine theta.
- So the slope is biggest when u points along the gradient. The gradient points straight uphill: the direction of steepest ascent, and its length is that steepest slope.
- Turn around, and you face steepest descent, minus the gradient. Turn a quarter turn, and the slope is exactly zero: you're walking along the hillside at constant height.
- Now float up and look straight down. The heights become level curves, the contour lines on a hiking map. Each one joins points at the same height.
- Draw the gradient at many points. Every arrow crosses the contours at a right angle, and points toward higher ground.
- Here's why. Walk along a level curve and your height never changes, so the directional derivative along the curve is zero. Zero dot product means perpendicular.
- Notice the lengths too. Where the contours crowd together the ground is steep and the arrows are long. Where they spread apart, the arrows shrink.
- Where the gradient is zero, the ground is flat in every direction. That's a critical point. It might be the bottom of a bowl,
- the top of a hill, or a mountain pass: a saddle, which goes up one way and down the other.
- The gradient is zero at all three, so first derivatives can't tell them apart. The second derivatives can. Collect them in the Hessian matrix. By Clairaut's theorem it is symmetric.
- A symmetric matrix has real eigenvalues and perpendicular eigenvectors. The eigenvectors are the principal directions, and each eigenvalue is the curvature along its direction. Here one curves up, the other curves down.
- So the rule is: both eigenvalues positive, a minimum. Both negative, a maximum. Opposite signs, a saddle. And if an eigenvalue is zero, the second derivative test is silent.
- To recap: the slope in direction u is the gradient dotted with u. The gradient points uphill, perpendicular to the level curves. Where it vanishes, the Hessian's eigenvalues classify the critical point.
- Next, we let a ball roll downhill. That's gradient descent, and it turns out to be Euler's method in disguise.
Slopes in every direction: the directional derivative
Stand at a point on the landscape and face in the direction of a unit vector , with . Walk a distance in that direction and you reach . The directional derivative of at in the direction is the rate at which your height changes per unit distance walked:
The partial derivatives are special cases: with (walk east), and uses (walk north).
You don’t need a new limit for every direction. Define the one-variable function . Then , and the multivariable chain rule from 2.2 Linearization, the Chain Rule & the Jacobian gives
That’s a dot product of the vector of partial derivatives with . The vector deserves a name.
The symbol is called “nabla” or “del”. Think of it as the vector of derivative operators . Applied to a scalar field it gives the gradient; you’ll see it dotted with a vector field to make the divergence (2.7 Divergence: Sources & Sinks) and crossed with one to make the curl (2.8 Curl & Vorticity).
The gradient also packages the linearization from the last lesson in a compact form:
Walking a small displacement changes the height by about . The directional derivative is just this with , divided by .
Worked example: a directional derivative
Find the rate of change of at in the direction of .
- Gradient: , , so .
- Unit direction: , so .
- Dot product: .
So walking in that direction, rises about 7.2 units per unit of distance. Sanity check with a small step : versus , a change of , which divided by gives 7.2. ✓
The gradient points uphill
Now use the geometric form of the dot product from 1.1 Vectors, Linear Combinations & Span. If is the angle between and , then because ,
As you turn on the spot, the slope you feel traces out a cosine. That single formula has three consequences:
- Steepest ascent. The slope is largest when , that is when points along . The maximum slope is . So the gradient points in the direction of steepest ascent, and its length is the steepest slope.
- Steepest descent. The slope is most negative, , when points along .
- Level directions. The slope is zero when : walking perpendicular to the gradient keeps you at constant height, to first order.
In the worked example above, . No direction gives a slope larger than 8.54, and the direction achieved 7.2 because it’s only about away from the gradient.
The gradient is perpendicular to level sets
A level curve of is a curve where equals a constant : a contour line on a hiking map, or an isobar on a weather map. In 3D, is a level surface, like a surface of constant temperature in a room.
Claim: at every point where , the gradient is perpendicular to the level set through that point.
Derivation. Let be any curve that stays inside the level set, so for all . Differentiate both sides with the chain rule:
The vector is tangent to the level set, so the gradient is perpendicular to every tangent direction of the level set. In 2D that means perpendicular to the level curve; in 3D, is a normal vector to the level surface.
Combined with steepest ascent, this gives a picture you can read off any contour map. The gradient crosses contours at right angles, pointing toward higher values. And since the slope is the change in height divided by the distance between contours, the gradient is long where contours are crowded and short where they are spread apart.
Worked example: tangent lines and tangent planes from the gradient
(a) Find the tangent line to the ellipse at .
The ellipse is the level curve of . Check: . ✓ The gradient is at , and it’s normal to the curve. A line through with normal is
Check by implicit differentiation: gives at , and the line has slope . ✓
(b) Find the tangent plane to the sphere at .
The normal is . The plane through with this normal is , which simplifies to . The normal points straight out from the center, as it should for a sphere.
Now explore it yourself. The sim shows a contour map (color = height) with the gradient field drawn on top. The dashed line through the ball is the level curve’s tangent, and the amber arrow is .
Things to try now (we’ll come back for the descent part):
- Click around the elongated bowl. Check that the amber arrow is always perpendicular to the dashed level-curve tangent, and that it’s longest at the top and bottom of the map, where the contours crowd together.
- Switch to Four wells. The gradient arrows all point away from the four minima and toward the single maximum near the middle. Find the points marked “saddle”: the arrows nearby point toward the saddle along one line and away from it along another.
- Turn on Exact gradient flow. The green curve follows , so it crosses every contour at a right angle on its way down.
Critical points and the Hessian
A critical point of is a point where . There the tangent plane is horizontal and every directional derivative is zero. For a smooth function, every local minimum and local maximum is a critical point. But not every critical point is an extremum, and in two dimensions there’s a third possibility, the saddle point, which goes up in some directions and down in others, like a mountain pass or a Pringle.
The first derivatives vanish at all three kinds, so we need second derivatives. Collect them in the Hessian matrix:
By Clairaut’s theorem (2.1 Functions of Several Variables & Partial Derivatives), for smooth , so is symmetric. Notice what it is: the Jacobian matrix (2.2 Linearization, the Chain Rule & the Jacobian) of the gradient map . So near any point, .
Why the eigenvalues decide
Near a critical point , the second-order Taylor expansion (derived carefully in 2.4 Taylor Series in One and Several Variables) is
So the shape near the critical point is the shape of the quadratic form . Here the spectral theorem from 1.6 Change of Basis & Diagonalization does all the work. A real symmetric has real eigenvalues and perpendicular unit eigenvectors . Write the displacement in that basis, . Since and ,
In the principal directions , the surface is a parabola with curvature along one axis and along the other, with no cross term. That gives the second-derivative test:
In 2D there is a shortcut that avoids computing eigenvalues. From 1.5 Eigenvectors & Eigenvalues, and . Let :
- : the eigenvalues have the same sign, and that sign is the sign of (or of the trace). So it’s a min if and a max if .
- : opposite signs, so it’s a saddle.
- : inconclusive.
This is the ” test” from Calc III textbooks. Now you know where it comes from.
Worked example: classifying critical points
Find and classify the critical points of .
Critical points. . Setting both to zero gives and . Substituting, , so and or (the only real roots). The critical points are and .
Hessian. , , :
At : . The characteristic polynomial is , so . Opposite signs: saddle. The eigenvectors are for and for : the surface curves down along the diagonal and up along .
At : , with , so and . Both positive: local minimum, with . The test agrees: and .
It’s only a local minimum: along the line , . Pick the “Saddle + minimum” landscape in the sim above to see both points and the open valley running off to the lower left.
Gradient descent: rolling downhill
Suppose you want the minimum of : the lowest-energy configuration of a structure, the best-fitting parameters of a model, or (soon) the pressure that makes a velocity field divergence-free. Since is the direction of steepest descent, the simplest algorithm is to take a step that way, recompute, and repeat:
The positive number is the step size (machine learning calls it the learning rate). This is gradient descent.
Transcript
- Part two. We follow the gradient downhill, find out when that goes wrong, and then see the gradient pushing real fluid around.
- Here's the contour map of a long, narrow valley: f equals x squared plus five y squared. Drop a ball on the hillside and let it roll down.
- Gradient descent: from where you stand, take a step of size alpha times the gradient, downhill. x new equals x old minus alpha grad f. Repeat.
- Look closely at that update. It is forward Euler, with time step alpha, applied to the differential equation x dot equals minus grad f. The green curve is that equation's exact solution, the gradient flow.
- Along the exact flow, f can only go down: by the chain rule, d f d t is the gradient dotted with the velocity, which is minus the gradient's length squared.
- Bigger steps get there faster, up to a point. With alpha equal to point one five, the ball overshoots the valley floor every step and zigzags its way down.
- With alpha equal to point two two, each zigzag is bigger than the last, and the ball is flung right out of the valley. The method has gone unstable.
- Why point two two? For this bowl the gradient is the Hessian times x. So each step multiplies x by the matrix I minus alpha H, whose eigenvalues are one minus alpha times each eigenvalue of H.
- Every one of those multipliers must be smaller than one in size. The steep direction, with lambda equal to ten, sets the limit: alpha below two over ten. It's exactly the rule for Euler's method on a stiff problem.
- And notice the trap: the steep direction caps the step, so progress along the gentle direction crawls. A narrow valley, with very different eigenvalues, is slow to descend. That ratio is called the condition number.
- Now physics. Here is a pressure field: high pressure on the left, low pressure on the right. The circles are isobars, level curves of the pressure.
- Take a tiny box of fluid. Pressure pushes inward on every face. The push on the left face is stronger than the push on the right, so the net force points right: from high pressure toward low.
- The net force is p at x minus p at x plus d x, times the face area. That's minus d p d x times the volume. Do the same in every direction, and the force per unit volume is minus the pressure gradient.
- Draw minus grad p everywhere. The arrows run from high to low, crossing the isobars at right angles, strongest where the isobars crowd together. That is what drives the wind.
- In a fluid solver, the pressure gradient is the enforcer. Stam's projection step solves for a pressure, then subtracts its gradient from the velocity, which is exactly what keeps the simulated fluid incompressible.
- To recap: gradient descent is Euler's method on the gradient flow, stable only when alpha is below two over the largest Hessian eigenvalue. And fluid is pushed by minus the pressure gradient.
- Next time: Taylor series in several variables, where the gradient and the Hessian turn out to be the first two terms.
It’s Euler’s method
Look at the update next to the ODE
Forward Euler with time step gives , which is gradient descent with . Gradient descent is forward Euler applied to the gradient flow. The exact solutions of the gradient flow are the curves of steepest descent, the green curves in the sim, and they always go downhill. By the chain rule,
with equality only at critical points. So itself acts as an energy that the flow steadily drains. That idea, a function that can only decrease along trajectories, becomes the Lyapunov function in 7.8 Conservative Systems & Lyapunov Functions.
Everything you learned about Euler in your Simulations course now applies to optimization. Small steps track the smooth flow. Big steps can overshoot, and too-big steps blow up. Better integrators suggest better optimizers too: momentum methods behave like integrating a damped second-order ODE.
Why big steps blow up: an eigenvalue story
Near a minimum , linearize the gradient using the Hessian: . Write for the error. Then
This is the matrix iteration from 1.5 Eigenvectors & Eigenvalues. has the same eigenvectors as and eigenvalues . In the eigenbasis of the iteration decouples: the error component along gets multiplied by every step. At a minimum all , and the iteration converges exactly when every multiplier has size less than 1:
That’s the same bound as Euler’s on . It’s no coincidence: the linearized gradient flow is , a linear system with eigenvalues .
The multiplier also tells you how each component behaves:
| multiplier | behavior in direction | |
|---|---|---|
| between 0 and 1 | between 0 and 1 | smooth, monotone approach |
| exactly 1 | 0 | lands on the minimum in one step |
| between 1 and 2 | between −1 and 0 | overshoots, zigzags, but shrinks |
| greater than 2 | less than −1 | zigzags with growing amplitude: divergence |
Worked example: the narrow valley
Run gradient descent on from .
Here and , so the update is exactly linear:
- Stability limit: , so we need .
- : the multipliers are and . The first step is , so . Then . The -coordinate flips sign every step while it halves: that’s the zigzag.
- : the -multiplier is , so grows by 20% per step even though shrinks. Divergence.
- Best fixed step: the slowest component decides the convergence rate, . This is smallest when the two are equal, , giving and a rate of per step.
In general the best rate is , where is the condition number of . Here gives . ✓ A long, narrow valley has a large : the steep walls cap the step size while progress along the gentle floor crawls.
Back to the sim. On the elongated bowl, set and press Run descent. Then try , and , and watch the multipliers in the readout. On the curved valley, the Hessian changes from point to point, so the safe step changes too: the readout shows the local as the ball moves.
Your turn: implement it
The pressure-gradient force
Now the physics that motivated the lesson. Take a tiny box of fluid with sides , , , centered at . The surrounding fluid pushes on every face with force per area equal to the pressure , always directed into the box.
Look at the two faces perpendicular to . Each has area . The left face, at , is pushed in the direction with force . The right face, at , is pushed in the direction with force . The net -force is
The same argument in and gives the other components, so the net pressure force per unit volume is
A uniform pressure, however large, produces no net force: only differences in pressure push fluid around. The force points from high pressure toward low, perpendicular to the isobars, and it’s strongest where the isobars are packed together. That’s exactly the weather-map rule from the introduction. (On Earth, the Coriolis effect then deflects the wind until it blows nearly along the isobars, but is the driver.)
Newton’s second law for a fluid parcel of density , ignoring viscosity and gravity, becomes
where is the material derivative from 2.2 Linearization, the Chain Rule & the Jacobian. That is the Euler equation of fluid motion, which you’ll derive properly in 11.3 Momentum: Pressure & the Euler Equations.
Hydrostatics. In a fluid at rest under gravity, the pressure force must balance the weight: with . That gives , so pressure increases linearly with depth, about one atmosphere for every 10 m of water.
Pen-and-paper problems
Let and .
- Find .
- Find the rate of change of at in the direction toward the point .
- In which direction does increase fastest at , and what is that rate?
- Find the directions in which the rate of change is zero.
Show worked solution
- and , so .
- The direction from to is , with length 5, so . Then .
- Along , at the rate . (The direction in part 2 was close to this, which is why 2.2 is nearly the maximum.)
- Perpendicular to : . These are the two directions along the level curve through .
The paraboloid can be written as the level surface .
- Find a normal vector to the paraboloid at .
- Write the equation of the tangent plane there and solve it for .
- Compare with the linearization of from 2.2 Linearization, the Chain Rule & the Jacobian.
Show worked solution
- at .
- , so .
- , and , so . They’re the same plane. In general, the graph is the level surface , whose normal is , and that’s exactly the tangent plane of the linearization.
Find and classify all critical points of . Give the Hessian eigenvalues at each.
Show worked solution
. Setting it to zero gives and , so , i.e. . The real solutions are , giving the critical points , and .
The Hessian is
- At : , with eigenvalues . Saddle, with .
- At : , with eigenvalues , i.e. and . Local minimum, with .
- At : the same Hessian, so also a local minimum with .
Since in every direction (the quartic terms win), these two are the global minima. The surface is two bowls joined by a pass at the origin.
Let .
- Show that the origin is the only critical point and that it is a minimum. Find the eigenvalues and eigenvectors of the Hessian.
- What is the largest step size for which gradient descent converges?
- Which fixed step size gives the fastest guaranteed convergence, and what is the error reduction factor per step?
- Starting from with , compute one step. Explain the result with eigenvectors.
Show worked solution
- . Setting this to zero gives a linear system with determinant , so the only solution is . The Hessian is constant, , with eigenvalues (eigenvector ) and (eigenvector ). Both are positive, so it’s a minimum, and the eigenvectors are perpendicular as the spectral theorem promises.
- .
- The rate is . The first term decreases and the second eventually increases as grows, so the best balances them: gives . The rate is then per step, which matches with .
- , so . It lands exactly on the minimum. The start is an eigenvector of with , so the only multiplier involved is . From a generic starting point, the component would still shrink only by per step.
A fluid of density rotates like a rigid body with angular speed : the velocity is , so a parcel at radius moves in a circle at speed . Suppose the pressure is .
- Compute the pressure force per unit volume . Which way does it point, and how large is it at radius ?
- A parcel moving in a circle of radius at speed needs a centripetal acceleration toward the center. Show that supplies exactly this.
- Where is the pressure lowest? What does this suggest about the core of a tornado, or a vortex in a smoke simulation?
Show worked solution
- , so . It points toward the center, with magnitude .
- The needed acceleration is toward the center. The acceleration provided is , also toward the center. They match exactly, so this pressure field is precisely what keeps the parcels on their circles.
- The pressure is lowest at the center, , and grows like outward. Any swirling flow needs an inward pressure force, so vortex cores are low-pressure regions. That’s why tornado funnels and wingtip vortices often show condensation (lower pressure means cooler air), and why pressure minima are a quick way to locate vortices in simulation output.
Looking ahead
The gradient is the first of the three great differential operators. Lesson 2.4 Taylor Series in One and Several Variables will show that and the Hessian are the first- and second-order terms of the multivariable Taylor expansion. That’s the result we borrowed in the critical-point test, and it’s the tool behind every finite-difference error estimate. In 2.6 Vector Fields, Flow Lines & Streamlines, gradient fields get their own name, conservative fields, and their flow lines are the steepest-descent curves you’ve been drawing. And in 7.7 Nonlinear Systems & Linearization, linearizing at a critical point gives the Jacobian , so a minimum becomes a stable equilibrium and a saddle stays a saddle.
Key Takeaways
- The directional derivative is a dot product: for a unit vector . It follows from the chain rule along the line .
- Since , the gradient points in the direction of steepest ascent, and its length is the steepest slope. is steepest descent.
- The gradient is perpendicular to level curves and level surfaces, so it supplies normal vectors for tangent lines and tangent planes. It’s long where contours crowd together.
- At a critical point (), the eigenvalues of the symmetric Hessian are the principal curvatures: all positive means a minimum, all negative a maximum, mixed signs a saddle, and a zero eigenvalue is inconclusive.
- Gradient descent is forward Euler on the gradient flow .
- Near a minimum, each step multiplies the error by , so descent converges only for , and it’s slow when the condition number is large.
- The net pressure force per unit volume is : fluid is pushed from high to low pressure, and the projection step of a fluid solver subtracts exactly this gradient.