Residual Connections

A new layer may only need to improve a representation we already have. A residual connection carries the input forward so the layer can learn a correction rather than having to recreate the entire representation.

10 sections · 10 questions · 4 coding challenges

1 / 10

The arrow keys move between sections too.

Why Did Adding More Layers Make Training Worse?

BOTIE

If a network already gives a useful representation, adding layers seems as though it should at least preserve that result. The new layers could simply pass the representation through. In practice, learning that apparently simple behaviour was not always easy, and deeper networks could have worse training error.

He and colleagues addressed this degradation problem in their 2015 residual-network paper. A block was asked to learn a change relative to its input, with a shortcut carrying the input alongside the learned transformation.

The output combines the original representation with the block's proposed adjustment. If little needs changing, a small adjustment is easier to express than rebuilding the whole representation. The shortcut also supplies a direct route for information and derivatives. It helps optimisation, but it does not mean every extra layer improves a model or that incompatible shapes can be added without a projection.

Sources for this history

He and colleagues (2015), Deep Residual Learning for Image Recognition

Do We Need to Rebuild a Useful Representation?

An earlier layer may already produce useful information. A later layer might need only a small correction, yet an ordinary chain asks it to produce the entire next representation.

A residual connection carries the input forward and lets another route learn an adjustment.

Start with information worth keeping

Imagine a representation already describing an image reasonably well. The next stage may need to refine it without rebuilding every useful part from scratch.

Input representation

Existing information

A representation can already be useful before the next block.

1 of 4

Learn a correction around an explicit identity route

The worked block shows the addition and its backward pass. It also distinguishes an identity shortcut from a learned projection used when dimensions differ.

A residual block still requires suitable initialization, training, and evaluation.

Part 1. Learn an Adjustment

1. Keep the input available

Call the original input x and the branch's transformation F. The residual output is their sum:

y = x + F(x)

The branch can increase, decrease, or otherwise modify coordinates. It is not limited to making a small change. "Residual" describes how we express the desired result relative to the input, not a constraint on the branch's magnitude.

An adjustment to one value

The playground uses a one-neuron branch with tanh activation and zero bias. At the default input and weight:

Input = 1
Branch weight = 0.5
F(1) = tanh(0.5 × 1) ≈ 0.4621
Residual output = 1 + 0.4621 ≈ 1.4621

The input contributes directly as well as influencing the branch. The result is therefore different from simply sending the input through that neuron.

2. Match the arrangements before adding

Each input coordinate must have a corresponding branch-output coordinate. A residual addition therefore requires the same shape on both routes. A flattened image and a two-dimensional image can hold the same number of values without having the same arrangement.

The SDK checks exact input and output shapes when constructing the wrapper. It does not silently reshape them or learn a projection on the shortcut. Larger architectures can introduce such projections explicitly when dimensions change.

Part 2. The Two Routes Through the Calculation

3. Add both input derivatives

Training moves every weight by the slope of the loss with respect to it, and that slope reaches a weight only by travelling back down through every layer above it. A plain layer hands the slope on through its own derivative, which for a tanh neuron is never larger than the size of its weight, and in the playground’s branch at the default setting it is 0.3932. The shortcut was added for what it does on this way back, so the second route has to be followed as carefully as the first.

A change to the input affects the result through the branch and through the shortcut. Backpropagation must include both contributions. For a scalar input:

dy/dx = 1 + F′(x)

For F(x) = tanh(w x):
F′(x) = w × (1 − tanh²(w x))

At x = 1 and w = 0.5:
F′(1) ≈ 0.5 × (1 − 0.4621²) ≈ 0.3932
dy/dx ≈ 1 + 0.3932 = 1.3932

Both routes, measured on the playground branch

Send a slope of one back into the playground’s branch on its own and it hands down 0.3932, the branch derivative above. Wrap the same branch in the shortcut, send the same slope of one, and what arrives at the input is 1.3932.

slope handed down by the branch alone        ≈ 0.3932
slope handed down by branch and shortcut   ≈ 1 + 0.3932 = 1.3932

The difference between the two is exactly the one the shortcut contributes. Nothing about the branch changed between the two measurements, which is the point of the wrapper. It leaves the branch as it was and adds a route beside it.

The shortcut contributes a derivative of one. It provides a direct route around the branch's transformations. For vectors, the corresponding expression uses the identity matrix and the branch's Jacobian.

4. Keep parameter learning inside the branch

The direct shortcut has no parameters. Its gradient contribution concerns the input. The branch still calculates the derivatives of its own weights and biases, just as it would without the wrapper.

input_gradient = arriving_gradient + branch_input_gradient
parameter_gradient = branch_parameter_gradient

The weight’s own slope, with and without the shortcut

The playground’s branch has one weight. Moving that weight moves the branch output through the tanh, and the shortcut takes no part, because the shortcut carries the input and the input does not depend on the weight. With a slope of one arriving at the output, the weight’s slope therefore comes out the same whichever way the branch is wrapped.

dy/dw = x × (1 − tanh²(w x))

At x = 1 and w = 0.5, with the tanh taken at full precision:
dy/dw = 1 × (1 − tanh²(0.5)) ≈ 0.7864

weight slope, branch alone          ≈ 0.7864
weight slope, branch and shortcut   ≈ 0.7864

So a training step moves the branch’s weight by the same amount with the shortcut as without it. What the shortcut changes is what the layers beneath receive, 1.3932 in the previous step against 0.3932, and in a deep chain that difference is the whole story, which Part 3 measures.

The implementation returns the branch's parameter gradient and rebuilds the wrapper around the updated branch after a training step. The forward response retains the branch response, so its backward calculation uses the values from the same pass.

Part 3. Why Deep Networks Use Shortcuts

A long chain asks its derivative to pass through every transformation. Residual connections introduce additional routes through the chain and make it straightforward for a block to preserve its input. This helps explain their usefulness in deep networks, including image models and transformers.

The failure the shortcut repairs is a product. On the way back, the slope arriving at the top of a chain is multiplied at each plain layer by that layer’s own derivative, and a product of many factors each smaller than one is very small. The bottom of such a chain receives almost no slope, learns almost nothing, and reports no error, because the loss at the top still falls while the layers nearer the top do the learning. A shortcut turns the factor at each block into a sum, one plus the branch’s derivative, and a chain of such sums has a route from the top to the bottom along which nothing is multiplied by any weight at all.

The playground neuron, stacked thirty deep

Take the playground’s branch at its default weight of 0.5, stack copies of it so that each reads what the one before answered, and hand the chain an input of one. The derivative at the bottom of the plain chain is one factor per layer multiplied together, and each factor is a tanh slope times 0.5, so no factor exceeds a half. In the residual chain each block adds its input back, so each factor is one plus a branch derivative instead.

depth    plain chain dy/dx     residual chain dy/dx
  1      0.3932                 1.3932
  2      0.1865                 1.8188
  5      0.0229                 2.5053
 10      0.0007                 2.5763
 30      6.8 × 10⁻¹⁰            2.5768

Thirty plain layers hand the bottom layer less than a billionth of the slope that arrived at the top, so it is barely trained however long the run goes on, and nothing in the loss says so. Thirty residual layers hand it 2.5768. The residual chain’s output has grown as well, to 29.68, since once the branch’s tanh saturates each block adds almost exactly one to what it read. The branch’s derivative there is near zero, so each block passes the slope through at almost exactly one, which is why the residual figure stops moving after about ten layers.

Read forward, the same addition says that a block whose branch has learned nothing yet is harmless. At a branch weight of zero the playground’s output is its input, where a plain neuron at a weight of zero answers zero whatever it was given. A network can therefore be made deeper without first being made worse, which is the observation the residual formulation was introduced to explain.

Keep in mind

The complete derivative is a sum. A negative branch derivative can cancel the shortcut's contribution, and repeated residual blocks can still amplify or reduce gradients. A shortcut improves the structure of the learning problem; it does not promise that every gradient or every training run will behave well.

The playground can show the cancellation, though not at every input. At an input of one the branch derivative never falls below −0.4477, which it reaches at a weight of about −0.77, so the combined derivative never falls below 0.5523 there. At an input of zero the branch derivative is simply the weight, so a weight of −1 gives a combined derivative of exactly 0 and a weight of −2 gives −1, a branch pulling against its own shortcut.

The residual formulation is developed in Deep Residual Learning for Image Recognition. The skip connections on the U-Net page serve a related purpose of preserving information, but that model concatenates selected feature maps rather than applying this elementwise addition.