Calculus Primer

Imagine turning a knob and watching an error meter. We want the error to fall but do not know the best setting. A small change could tell us which way the meter responds.

Calculus makes that question precise: how does one quantity respond when another changes?

Everything here rests on comparing two changes. An output changes as an input changes, and the question is always how much of the one you get for the other. On a straight line that comparison has a name, the slope, and you compute it from any two points by dividing the rise, how far up the line went between them, by the run, how far across.

slope = rise / run

run   = 6 − 2 = 4
rise  = 11 − 3 = 8
slope = 8 / 4 = 2

Written with the points as (x₁, y₁) and (x₂, y₂), that is the whole formula.

slope = (y₂ − y₁) / (x₂ − x₁)

The sign carries meaning of its own. Had the line dropped between the two points the rise would come out negative, and so would the slope, so a downhill line has a negative slope. A flat line has a slope of zero, because it rises by nothing however far it runs. Those three cases, positive, negative and zero, are the whole vocabulary, and every later section reuses them.

Rise over run can be dragged. The two points below start at the worked pair, and the triangle between them remeasures as either moves.

Now the reframing that turns a fact about a drawing into something worth having. A slope is not only a tilt. When the two axes carry units, rise over run carries units too, and the slope becomes a rate.

100 miles / 2 hours = 50 miles per hour

Notice what kind of number that is. It is an average across the whole trip. At no single moment did the speedometer have to read 50, since you may have been stopped at a light for part of it and doing 70 for the rest. An average across a stretch and a reading at an instant are different quantities, and closing the gap between them is the entire subject of the next three sections.

A straight line has one slope everywhere, which is what made it easy. A curve does not. It might fall steeply in one place, flatten out, and climb somewhere else, so no single number describes its steepness. What we can still do is exactly what we did with the drive, pick two points and measure the average between them.

The straight line drawn through two points of a curve is called a secant, and its slope is the average rate of change of the curve across that stretch. Let us make that concrete on the curve in the box below.

f(x) = 0.5·x² − x + 2.5

where f(x) is just the name for the curve’s height at input x.

f(3) = 0.5·9  − 3 + 2.5 = 4
f(5) = 0.5·25 − 5 + 2.5 = 10

average slope = (10 − 4) / (5 − 3) = 3

That is what the box below shows on load. The base point starts at x = 3, the gap is 2, and the solid line through the two points has slope 3.00, exactly the number we just computed by hand. The dashed line and the second readout will be explained shortly. For now, drag the base point around and watch the average slope change as the stretch of curve under it changes.

An average across a stretch is still not what we set out to find. We wanted the steepness of the curve at a single point, the speedometer reading rather than the trip average. And a single point, on its own, gives us nothing to divide, since one point has no rise and no run. It looks like a dead end.

Here is the move everything else is built on. If the average across a stretch is the wrong answer because the stretch is too wide, then shrink the stretch. Keep the base point at x = 3 and pull the second point closer, computing the same rise over run every time. Each average is taken across a shorter piece of curve, so each is a better stand-in for the steepness right at the base.

f(4) = 0.5·16 − 4 + 2.5 = 6.5
average slope = (6.5 − 4) / (4 − 3) = 2.5 / 1 = 2.5

Working every row the same way gives this, and every one of them is a position of the gap slider in the box above.

The averages are not wandering. They are settling, and the smaller the gap the closer they settle. 3, then 2.5, then 2.25, then 2.05, then 2.01. Nothing here yet says what they settle on, or gives that value a name. What the table shows is only that they are heading somewhere, and that where they are heading does not depend on how we got there.

The value those averages close in on, 2 here, is the instantaneous rate of change of the curve at x = 3. That is what a derivative is, and it is worth saying plainly that you have already computed one. The name arrives after the idea rather than before it.

Writing the gap as h, the definition says to take the average slope across a gap of h and ask what value it approaches as h shrinks toward zero.

(f(x + h) − f(x)) / h,   as h approaches 0

Written the conventional way, with a prime mark for the derivative and lim for the approaching, that is

f'(x) = lim(h → 0)  (f(x + h) − f(x)) / h

The picture of the same process is the two lines in the box above. While the second point slides in, the solid secant rotates, and it settles onto the dashed line, the tangent, the one straight line that grazes the curve at the base point and runs alongside it there. The secant measures an average and the tangent is what the averages become, so the slope of the tangent is the derivative. That is why the two readouts meet as the gap closes.

What the shrinking-gap experiment found was one number, the slope at x = 3. Doing that again at x = 4, and again at x = 4.1, would be a miserable way to work, and it would never finish, because a curve has infinitely many points.

It matters because of what the slope is for. The walk in section 11 reads the slope where it stands, moves, and reads it again somewhere new, at x = 5, then at 3.8, then at 2.96, and none of those places was known before the walk set out. A slope found once at x = 3 answers none of those requests. What the walk needs is an instrument it can point at any x and read on the spot, not a reading taken in advance.

The remarkable thing is that the shrinking-gap calculation can be carried out with the gap left as the letter h and the base left as the letter x, which does every point at once. What comes out is not a number but a formula, a rule that returns the slope at whichever point you hand it. That is the next section, and it is ordinary algebra throughout.

The notation keeps the two apart. f’(3) is a number, the 2 that the shrinking gaps settled on. f’(x), with the x left in, is the rule itself, and f’(3) is what the rule says when handed 3. One derivation produces the rule, and every reading after that is a substitution into it.

The recipe says to take the height at x + h, subtract the height at x, and divide by h. Here is that carried out on our curve, one operation per line, with nothing hidden.

The second-to-last line says that the average slope differs from the derivative by half the gap. At a base point of three, the derivative is two.

average slope = x − 1 + h/2
at x = 3 and h = 1:   3 − 1 + 1/2 = 2.5
at x = 3 and h = 0.1: 3 − 1 + 0.1/2 = 2.05

These are the values in section 4’s table. Letting the gap approach zero removes the extra term and leaves the derivative.

What is left is one clean formula for the slope at every point of the curve at once. Drag the base point anywhere in the box above and the derivative readout will be x − 1 to the last digit, because that is exactly what the server computes.

A derivative is not merely another formula to carry around. It describes the curve, and three readings cover most of what anyone needs from it. Take f’(x) = x − 1 and evaluate it in a few places.

So the sign gives the direction, negative for falling and positive for rising. The magnitude gives the steepness, so a derivative of 2 is twice as steep as one of 1 regardless of which way either points. And zero marks a flat place, where the curve has stopped doing one and not yet begun the other.

The magnitude has a use beyond comparison. A rate multiplied by a run predicts a rise, as 50 miles per hour times 2 hours predicts the 100 miles of section 2, and a derivative is a rate, so it predicts how much the height will change over a short run from the point it was read at.

predicted rise = 2 × 0.02 = 0.04
actual rise    = f(3.02) − f(3) = 4.0402 − 4 = 0.0402

predicted change = −1 × 0.02 = −0.02
actual change    = f(0.02) − f(0) = 2.4802 − 2.5 = −0.0198

from x = 3 over a run of 0.5
predicted rise = 2 × 0.5 = 1.0
actual rise    = f(3.5) − f(3) = 5.125 − 4 = 1.125

Those three readings are enough to find the bottom of a curve, which is what the rest of this primer does with them.

At an interior minimum of a differentiable curve, the derivative is zero. That gives us candidate locations to check. We still need to establish whether a flat point is a minimum, maximum, or neither.

f'(x) = x − 1 = 0   →   x = 1

f(1) = 0.5·1 − 1 + 2.5 = 2

For this example, finding the point with zero slope takes one short equation.

f′(x) = 0
x − 1 = 0
x = 1

Other functions need not be so cooperative. A model’s loss can depend on many parameters at once, and setting all its derivatives to zero may produce equations we cannot solve directly. That motivates taking repeated steps guided by the gradient.

So we keep the insight, that the bottom is where the slope is zero, and give up on jumping there in one algebraic step. Instead we walk, and we use the derivative the other way, not as an equation to solve but as a direction to follow.

Picture standing on a hillside in thick fog. You cannot see the valley floor, only the ground at your feet, but the slope underfoot still tells you which way is down. Stand where the slope is negative, the ground falling to your right, and downhill is to the right, so you want to increase x. Stand where the slope is positive and downhill is to the left, so you want to decrease x. In both cases you move against the sign of the slope.

That is what the rule says, and now it should read as inevitable rather than arbitrary.

x  ←  x − η · f'(x)

The minus sign is the “against”. The η, a number you choose, is how boldly to act on what the slope told you, and it has a section of its own shortly.

One update, worked all the way through, on the curve we have been using throughout. Start at x = 5 with η = 0.3.

Carrying on from there, the next few steps go like this.

Each row is made from the one above it by the same three operations, reading the slope at the new location, scaling it by the rate, and moving against it. From the end of step 1,

slope at 3.8  = 3.8 − 1 = 2.8
movement      = −0.3 × 2.8 = −0.84
new location  = 3.8 − 0.84 = 2.96

which is row 2, and every later row checks the same way.

Nothing tells the walk to slow down. It slows because the ground flattens, and the step is sized by the slope it reads.

The one choice left in the walk is η, how far to move on each step. The slope gives a direction and a magnitude, and η decides how boldly to act on them. It is easier to see what it does than to be told, so start with the box below.

Click anywhere on the curve to choose a start and watch the walk come down. Each dot is one step, and the dashed line is the tangent at the start, the very line the earlier sections built. Now move the rate slider and four distinct behaviours appear.

The third case is the surprising one. It looks alarming and it is still converging, and the fourth looks similar for a step or two before it leaves. Something changes between 1.9 and 2.5, and for this particular bowl we can say exactly what.

x  ←  x − η·(x − 1)

(x − 1)  ←  (1 − η) · (x − 1)

That is the trade the learning rate sets. Too small wastes steps, too large diverges, and the workable range in between is found by exactly the kind of experiment the slider lets you run.

Switch the box above to the two-valley curve, and the caution from section 9 becomes something you can watch. Start on the left of the central hill and the walk settles in the left valley. Start on the right and it settles in the right one. Same curve, same rule, different answer.

With sufficiently small steps, this walk stays in the valley where it starts. Its local slope gives it no reason to cross the hill to inspect the other valley. The bottom it reaches is a local minimum; the lowest point across the whole curve is the global minimum.

Everything so far has had one adjustable value. A model has more, and the step from one to many is the last idea this primer owes you.

With one setting there is a loss L(θ), one derivative, and two directions to move, left or right. With two settings there is a loss L(θ₁, θ₂), and the picture becomes a surface rather than a line. There is no single slope on a surface, because the ground can tilt one way as you walk east and another as you walk north.

What there is instead is one slope per direction, found by asking how the loss changes as you vary one setting while holding the other still. Each of those is called a partial derivative.

Collect them and you have the gradient, written with a nabla.

∇L = ( ∂L/∂θ₁,  ∂L/∂θ₂,  …,  ∂L/∂θₙ )

The update rule then reads exactly as it did before, with the single slope replaced by the whole collection and the single value replaced by all of them at once.

θ  ←  θ − η·∇L