Backpropagation

Backpropagation reverses something, so start with the thing it reverses. One row, (1, 2), goes into the 2-3-1 network. Each hidden neuron forms a score from the row, the rectifier applies ReLU to each score, the output neuron forms a score from the three activations, its activation function is the identity so that score is the prediction, and the prediction is measured against the target of 1 by half the squared miss.

…

Six kinds of intermediate value, and every one of them is kept. That is not tidiness. The backward pass is about to ask for each of them by name, and a forward pass that discarded its scores would have to be run again to recover them.

A gradient-descent step needs one number per weight, the slope of the loss with respect to that weight. In plain terms, if this weight went up a little, would the loss rise or fall, and how sharply?

∂L/∂w, for every w in the network

On the logistic regression page that question had a short answer, because a single neuron’s weight sat one step from the loss. A hidden weight here does not. Its effect travels through its own neuron’s score and activation function, then through the output neuron’s score, then through the prediction, and only then into the loss. The widget draws that route for whichever hidden weight you choose, with the values the forward pass kept at every stop, and calculates nothing.

…

Take one neuron rather than the whole layer, and take the output neuron because its route is shortest. Three separate questions can be asked about it. How much does the loss respond to the neuron’s output? How much does the neuron’s output respond to its score? How much does its score respond to one particular weight? Each has a small, local answer, and the chain rule multiplies them.

…

∂L/∂w = (∂L/∂a) · (∂a/∂z) · (∂z/∂w)

The walk has to start somewhere, and it starts at the loss, which for this page is half the squared miss.

L = ½ (ŷ − y)²            ∂L/∂ŷ = ŷ − y

The half is there so the derivative comes out as a clean subtraction. Two different numbers live at this point and the page will keep them apart. The loss is a value, how wrong the network is. The slope of the loss is a rate, how fast that wrongness changes when the prediction moves.

miss = prediction − target = 5 − 1 = 4
loss = ½ × miss² = ½ × 4² = 8
∂loss/∂prediction = prediction − target = 4

The output neuron is a useful place to learn the backward calculation because its activation function is the identity: it returns its score unchanged.

output a = g(z) = z
activation derivative g′(z) = 1

Call the gradient with respect to the score the delta. To obtain it, multiply the arriving output gradient by the activation derivative.

δ = (∂L/∂a) · g′(z)

Then pick one edge, the weight from h₁ into the output. The score is a weighted sum, so its slope with respect to that weight is whatever the weight multiplied, and the weight’s gradient is the delta times that.

z = w₁h₁ + w₂h₂ + w₃h₃ + b
∂z/∂w₁ = h₁
∂L/∂w₁ = δ · h₁

…

delta = 4 × 1 = 4
weight gradients = 4 × (3, 0, 1) = (12, 0, 4)
bias gradient = 4 × 1 = 4

The output neuron can now update its own weights. The hidden layer still knows nothing, because nobody has told it how its outputs affected the loss. So ask one concrete question. How did changing h₁ affect the output score? By w₁ per unit, since h₁ appears in the score once, multiplied by w₁. The loss’s sensitivity to h₁ is therefore the output delta times w₁, and the same for the other two.

∂z_out/∂h₁ = w₁            ∂L/∂h₁ = δ_out · w₁

…

hidden output gradients = 4 × (1, −1, 2) = (4, −4, 8)

The hidden neurons have an activation function that is not the identity, and this is what changes. For each of them, take the arriving gradient, recall the score the forward pass kept, evaluate the rectifier’s derivative at that score, and multiply.

…

The middle row is the one to look at. h₂ scored −1, the rectifier turned that into 0 going forward, and going backward its slope at −1 is 0, so an arriving −4 becomes a delta of 0. This neuron receives no update from this row. Nothing it does reaches the loss on this row, so nothing on this row can tell it what to do. Whether a neuron can get stuck that way for good belongs to the activation and regularisation material, not here.

Once its delta is known, a hidden neuron follows the same rule as the output neuron. The first hidden neuron has delta 4 and received inputs 1 and 2.

weight gradients = 4 × (1, 2) = (4, 8)
bias gradient = 4 × 1 = 4

…

Everything done by hand above is four lines, and now that each has been used they can be written as a summary rather than a rule to decode.

δ = arriving ⊙ g′(z)
∂L/∂W = δ xᵀ
∂L/∂b = δ
∂L/∂x = Wᵀ δ

Two names are worth having now. The arriving block is the slope of the loss at a layer’s outputs, which came from the loss for the top layer and from the layer above for everyone else. The passed-down block is the slope at its inputs, which is the next layer down’s arriving block. One variable threads the whole walk.

Calculating gradients and applying them are two phases, and they must not overlap. In the first, backpropagation computes and stores every gradient while no parameter moves. Moving one early would change the scores and activations the remaining gradients are still being computed from. In the second, every parameter takes one step against its own gradient, scaled by the learning rate.

θ_new = θ_old − η · ∂L/∂θ

…

h₁ weights: (1, 1) − 0.05 × (4, 8) = (0.8, 0.6)
h₁ bias: 0 − 0.05 × 4 = −0.2

h₁ score: 0.8 × 1 + 0.6 × 2 − 0.2 = 1.8
h₃ score: −1.4 × 1 + 0.2 × 2 − 0.4 = −1.4
prediction: 0.4 × 1.8 + (−1) × 0 + 1.8 × 0 − 0.2 = 0.52
loss: ½ × (0.52 − 1)² = 0.1152

The chain rule’s answer can be checked without the chain rule. Take one weight, nudge it up by a tiny ε and measure the loss, nudge it down by ε and measure again, and divide the difference by the width of the nudge.

[L(w + ε) − L(w − ε)] / 2ε  ≈  ∂L/∂w

…

That is why the nudge is useful, and why it is a check rather than a method. It costs two forward passes per parameter, so it is the right tool for confirming that an implementation of the four lines is correct and the wrong tool for training anything, where one backward walk finds every slope at once for about the price of a second forward pass.

Every calculation so far was on scalars, one neuron and one weight at a time. A layer does the same thing for several neurons at once, and the scalars become vectors and matrices. The input x is a column of the layer’s inputs, W has a row per neuron and a column per input, z and a are columns with one entry per neuron, δ is a column of the same length, and the weight gradient has exactly W’s shape, because there is one slope per weight.

…

Read δ xᵀ back against section 5. A column of deltas times a row of inputs is an outer product, and its entry in row j and column i is δⱼ xᵢ, which is the scalar rule for the weight from input i into neuron j. Read Wᵀ δ back against section 6. Input i’s entry is Σⱼ w[j, i] δⱼ, the sum over every neuron of that neuron’s delta times the weight on the edge, which is section 6’s rule with the routes summed because in general an input feeds every neuron.

The stack then walks its layers from the top down, handing each one the block the layer above passed down.

With the mechanism settled, one question about the layer interface is worth settling too. Every layer must return a gradient with respect to its inputs, because the backward walk has to continue through it. Only layers with trainable parameters have parameter gradients. Dense and convolutional layers return both. Flattening, pooling and dropout return an input gradient and no parameter gradient, and they say so with None rather than a block of zeros.

The two meanings must stay distinct. None means this layer has no trainable parameters. A zero tensor would mean this layer has parameters and their gradients happen to be zero, which is what h₂ reported in section 8 and is a fact about this row, not about the layer. The rule cuts the other way as well.

A dense layer handed None when asked to step refuses, because it does have parameters and a missing gradient for them is a mistake in whoever built the backward pass. The stack’s report of the largest slope anywhere skips the parameterless layers rather than counting them as zero, so a network made of nothing but pooling reports no movement at all, which is the true answer.

Take one layer. Each row of its weight matrix belongs to one neuron. The matrix multiplication produces the scores, and the activation function acts on each score separately.

scores:  z = Wx + b
outputs: a = g(z)

Suppose the gradient of the loss with respect to every output is already known. Because this activation acts separately on each score, its backward calculation multiplies each arriving gradient by the corresponding activation derivative.

∂L/∂zⱼ = ∂L/∂aⱼ · g′(zⱼ)   =   δⱼ

Score j is the sum over i of w[j, i] x[i], plus b[j], so its derivative with respect to w[j, i] is x[i] and with respect to b[j] is 1, and the chain rule again gives the two lines the layer keeps. Input i appears in every score, so the loss reaches it through every neuron, and the chain rule sums the routes.

∂L/∂w[j, i] = δⱼ · xᵢ
∂L/∂bⱼ = δⱼ
∂L/∂xᵢ = Σⱼ δⱼ · w[j, i]

That last line is the induction. The slope of the loss with respect to this layer’s inputs is, by definition, the slope of the loss with respect to the outputs of the layer beneath, which is exactly the arriving block that layer needs to run the same three lines. The base case is the loss itself, whose slope at the final output section 4 worked out, and the walk ends at the row, whose slope nobody needs but which costs nothing to report.

Every layer is visited once, every slope is computed once, and the arithmetic is a handful of matrix products against blocks the forward pass already built, which is why the forward responses keep their scores rather than only their outputs.