Normalisation Layers

Take the crowd network the dropout page trains, sixteen rectified hidden units reading a person’s height and weight, and one output unit reading those sixteen. The output unit’s weights are being tuned against the sixteen numbers the hidden layer hands it for each person, and every step of training also moves the hidden layer, so the numbers the output unit was tuned against last epoch are not the numbers it reads this epoch.

I measured how far they move. Over four hundred epochs on the whole crowd, the mean of one hidden unit over the twenty-five people travels from 0.439 to 1.529, and its deviation from 0.819 to 3.097.

… the first visit trains a hundred and thirty small networks, which takes ten to twenty seconds

Read the two readouts against each other. The largest mean shift from the first epoch to the last is 1.0897, and the largest move any unit’s mean makes in a single epoch is 0.1375, so the block the output layer reads is being moved underneath the output layer at every step rather than settling, by weights the output layer has no say in. That is the moving target the original paper named internal covariate shift, and on this network the chart above is the measurement of it.

The feature scaling page put height and weight on a common footing once, before the model saw them, and the crowd network here reads them that way. That settles the block the first layer reads and nothing after it. The block the output layer reads is produced by weights that change, so there is no one-off standardisation of it to make, and the only place a standardising step can go is inside the model, between the two layers, recomputed on every pass from whatever is flowing through at that moment.

Switch the chart above to the batch layer and the sixteen lines start at exactly zero and one, because that is what standardising does to them. What happens afterwards is the claim this page has to be plain about. The block the output layer reads still moves, and by more. The largest mean shift over training is 1.3204 against the plain network’s 1.0897, and the largest single-epoch move is 0.3113 against 0.1375.

What differs is what moved it. Under the batch layer the mean of what the output layer reads equals the layer’s shift to within 1.8e−15 at every epoch, so the location and spread of the block are two explicit parameters per unit that the gradient moves on purpose, rather than a by-product of sixteen weight vectors being adjusted for other reasons.

Picture four people’s readings at two hidden units, chosen so that every statistic comes out whole: (1, 7), (1, 9), (7, 1) and (23, 7). The playground opens on them. A batch layer reads down each column, so for the second unit it has the four numbers 7, 9, 1 and 7 to work with, and the first thing it does is find their mean.

7 + 9 + 1 + 7 = 24,   24 / 4 = 6

The deviations from 6 are 1, 3, −5 and 1, whose squares are 1, 9, 25 and 1, and the variance divides their total by the number of rows rather than one fewer, which is the biased form and is the one every implementation of this layer uses on the way forward, for a reason Part 4 comes back to.

1 + 9 + 25 + 1 = 36,   36 / 4 = 9

(7 − 6) / 3 =  0.3333
(9 − 6) / 3 =  1.0000
(1 − 6) / 3 = −1.6667
(7 − 6) / 3 =  0.3333

The playground does not print a deviation of 3. It prints 3.0000017, and the 1.0000 in the worked column is 0.9999994 rounded. That is a small constant, 0.00001 here, added to the variance inside the square root before the layer divides by it. The reason is a column whose four numbers are all the same, which has a variance of zero and would otherwise be a division by zero, and that is an ordinary thing for a hidden block to contain, since a rectified unit that is off for every person in the batch produces exactly that column.

deviation = √(variance + ε)
√(9 + 0.00001) = 3.0000017

Adding epsilon under the square root keeps the denominator positive even when a column has zero variance. The derivative of the reciprocal square root is also finite at zero variance for a fixed positive epsilon. The normalized variance is consequently a little below one.

normalized variance = v / (v + ε)
worked column: 9 / (9 + 0.00001) ≈ 0.9999989

Type five into every cell of a column. Centring produces zeros, and the normalized output is zero for each row. A learned shift can still move those outputs afterward.

Standardising alone would be a constraint. A layer that could only answer with mean zero and spread one would be forbidden from ever passing on an off-centre signal, and the next layer sometimes needs exactly that. So every normalising layer computes a second line, one learned multiplier and one learned offset per feature, which begin at one and zero so that a fresh layer standardises and does nothing more.

standardised = (value − mean) / √(variance + ε)
answer       = scale · standardised + shift

Press Undo exactly and the sliders jump to the column’s own deviation and mean, 3.0000017 and 6 for the second feature, and the answer is the raw column back again. The largest gap between the answer and the raw block is 8.9e−16, and the first row’s 1 comes back as 0.9999999999999991, which is a rounding of the last bit rather than anything the standardising left behind.

Two learned numbers per feature can undo it completely, so the layer costs the network nothing it could represent before. What changed is how the numbers are parameterised. The weights beneath used to have to produce a signal at the right level and the right spread by themselves, and now they produce a standardised one while the scale and shift set the level and the spread.

γ = √(v + ε),   β = m
γ · (x − m) / √(v + ε) + β = (x − m) + m = x

Every statistic the batch layer uses is a fact about the batch rather than about the row being answered, and everything awkward about the layer follows from that. In the playground, take rows away under the batch layer down to the two it keeps. The first row, (1, 7), answered −0.7778 and 0.3333 among the four; among the two it answers 0.0000 and −1.0000, though not a digit of the reading changed, only the crowd it was standardised by.

That is unusable at prediction time. A request may hold one person or a thousand, and a person’s answer cannot be allowed to depend on who else happened to be in the same request, so the layer needs to know which of two things it is doing. While training it standardises by the batch, since that is what the gradient will be taken through. While predicting it standardises by something remembered instead, which the next two steps build.

Push the first row through on its own while training and the batch is that one row. Its mean is itself, its deviation is the root of zero plus ε, and the layer answers zeros for both features, which is the same answer the constant column got in step 4 and for the same reason: a batch of one carries no spread to standardise by.

Nothing is refused, because a row of zeros is what the arithmetic honestly says, and that is worth knowing, since a training loop fed one row at a time through this layer trains on nothing and raises nothing.

While predicting the same row is fine, because the divisor is no longer read from the row. A fresh layer holds a running mean of zero and a running variance of one, so it answers 0.999995 and 6.999965, the row itself to within the epsilon, and a layer that has been stepped a hundred times on the four-row batch answers −0.7778 and 0.3334, which is the batch answer of step 3 to three places.

What prediction standardises by is a running mean and a running variance, one pair per feature, moved a little toward each batch on every step and consulted only when predicting. They begin at zero and one, which is what an untrained layer believes about its inputs and is why predicting through one is close to an identity rather than an error.

running = 0.9 · running + 0.1 · batch

running mean     = 0.9 · 0 + 0.1 · 6 = 0.6
running variance = 0.9 · 1 + 0.1 · 9 = 1.8,   √(1.8 + 0.00001) = 1.3416445
(7 − 0.6) / 1.3416445 = 4.7702651

Same layer, same row, different statistics, and only more steps bring the two answers together. The chart follows them over a hundred steps of the same batch. The widest gap between the predicting answer and the training answer is 5.7333 after one step, 1.0433 after ten and 0.0238 after forty-four; it first falls under a hundredth at step 53 and under a thousandth at step 75, and after a hundred steps it is 6.5e−05.

…

A tenth of the way each step is the default momentum of 0.9, which keeps nine tenths of what it had, and it is worth saying that some frameworks name the same update by the other number, so a momentum of 0.1 there is this 0.9 and not a different rate. A larger batch makes each step’s figures a better estimate of the whole and does nothing to the number of steps the average needs, which is why a batch layer is trained for a good while before anyone trusts its predictions.

On twenty of the crowd the running mean after one step is 15.19 centimetres against a batch mean of 151.9, and a person predicted through that layer would be standardised by a crowd that does not exist.

Most implementations move the running figures inside the forward pass, by changing the layer as it answers. The layer measured here does not change anything as it answers. Its training pass carries the batch’s mean and variance on the response it hands back, the backward pass carries them on the gradient beside the slopes for the scale and shift, and the step that builds the next layer is where the running figures move, at the same moment the scale and the shift do.

In an ordinary training loop that is once per batch, which is where a mutating implementation would have put it.

The cost of that choice is stated rather than hidden. A forward pass that is never followed by a step contributes nothing to the running figures, so a training pass made for a look rather than for a step leaves prediction where it was. For a training loop that is the correct behaviour, since a batch that was never learned from should not shape what the model believes about its inputs; for a caller used to other frameworks it is a difference, and this is where it is written down.

The playground’s predicting toggle takes one real step with a slope of zero arriving, which moves neither the scale nor the shift and folds the batch a tenth of the way in.

Now the blame has to pass back through the layer, and the tempting reading is that a standardised value is just the input minus a number over another number, so the slope with respect to the input is one over the deviation. It is wrong in a way that trains. Change one person’s reading at a hidden unit and that unit’s mean over the batch moves, and with it every other person’s standardised value at that unit; the variance moves too.

So one input reaches the loss by three routes, directly through its own standardised value, through the mean, and through the variance, and the gradient has to collect all three.

Write n for the number of rows, z for the standardised block, s for the deviation of each column with the epsilon inside, and d for the slope arriving at the layer’s outputs multiplied by the scale. Collecting the three routes gives one line, with the two sums running down each column.

passed down = (n · d − Σ d − z · Σ (d · z)) / (n · s)

Σ d        = 2 + 4 + 6 + 8 = 20
Σ (d · z)  ≈ 2 × 0.3333 + 4 × 1 + 6 × (−1.6667) + 8 × 0.3333 ≈ −2.6667
n · s      ≈ 4 × 3 = 12

row 1   (4 × 2 − 20 − 0.3333 × (−2.6667)) / 12       ≈ −0.9259
row 2   (4 × 4 − 20 − 1 × (−2.6667)) / 12            ≈ −0.1111
row 3   (4 × 6 − 20 − (−1.6667) × (−2.6667)) / 12    ≈ −0.0370
row 4   (4 × 8 − 20 − 0.3333 × (−2.6667)) / 12       ≈  1.0741

column sum    −0.9259 − 0.1111 − 0.0370 + 1.0741 ≈ 0
naive form    d / s ≈ (0.6667, 1.3333, 2, 2.6667), which sums to 6.6667

The two parameter slopes are the easy half and come out the same whichever form of the passed-down block is used, which matters in step 13. The variance in that line is the biased one because the derivative is taken with respect to the quantity actually computed, so the forward pass and the backward pass have to agree about which variance that was.

A finite difference is the oracle, because it knows nothing about routes. Nudge one input up and down by a small step, run the forward pass both ways, and difference the objective, which here is the arriving slope times the outputs, summed. The widget does that for every cell of the whole-number four under a slope that counts up from one across the block, and puts the passed-down block, the naive block and the finite difference side by side.

The passed-down block agrees with the finite difference to 2.1e−10 across every cell. The naive block misses it by 2.0370, on a block whose largest true slope is 1.0741, so the error is larger than the largest slope it was reporting, and every entry of it is the wrong sign in the first column. There is also a check that needs no oracle at all.

Shifting a whole column by a constant cannot change what a centring layer answers, so the true block’s columns must sum to zero, and they do, to 2.2e−16; the naive block’s sum to 1.7778 and 6.6667.

The scale slope on the worked batch is 8.0 and −2.6667, and the shift slope is 16 and 20, and neither number depends on which passed-down form was used, since both are sums over the standardised block and the arriving slope alone. So a check that only looked at what the layer learns for itself would pass the naive form, and a network built on it would train, with plausible magnitudes and a falling loss, on a gradient that is not the gradient of the loss for every layer beneath.

That is why the check has to be on the block passed down, and why the tests behind this page run the finite difference rather than trusting the derivation.

Everything awkward in Part 3 came from reaching across the batch, and none of it was about normalising. Layer normalisation makes the identical move along the row. The mean and the deviation are read from the numbers one row already holds, so the answer is the same whether the row arrives alone or among a thousand others, and it is the same while training and while predicting because there is nothing that could differ. No running figures, no purpose, no batch. Press Layer in the playground and the bands turn sideways.

(1 − 4) / 3.0000017 = −0.9999994,   (7 − 4) / 3.0000017 = 0.9999994

The whole difference between the layers is the index the sum runs over. Write the block as xᵢⱼ, row i and feature j, with n rows and d features.

batch   mⱼ = (1/n) Σᵢ xᵢⱼ,   vⱼ = (1/n) Σᵢ (xᵢⱼ − mⱼ)²     over the rows, one figure per feature
layer   mᵢ = (1/d) Σⱼ xᵢⱼ,   vᵢ = (1/d) Σⱼ (xᵢⱼ − mᵢ)²     over the features, one figure per row

The three routes are the same three, with the two sums now running along the row and divided by d rather than n, and the structural check turns with them, since shifting a whole row by a constant cannot change what the layer answers, so the true passed-down block’s rows sum to zero. The widget shows it on the three-feature block, because on a row of two numbers something stranger happens that step 16 is about.

On the three-feature block the passed-down block agrees with the finite difference to 1.5e−09 against a largest true slope of 0.5123, and its rows sum to zero to 4.5e−16, where the naive block misses by 5.3404 and its rows sum to as much as 14.1203. Switch the block to the whole-number four and the true block is zero in every cell, with a naive block that says otherwise by 2.0, which is the row of two numbers having nothing to pass down rather than a fault in the check.

A row of two numbers has a mean halfway between them and a deviation of half their difference, so each number standardises to exactly one or minus one, whichever side of the mean it was on. That is why the worked row came back as ±0.9999994 whatever its values, and it is what happens if the row layer is put where the batch layer was in step 3, reading a person’s height and weight.

Twenty of the crowd go in and twenty rows of 1.0000 and −1.0000 come out, every person’s height being the larger number, and the slope through it is zero in every cell, so a network below could not learn anything either.

…

The widest distance from ±1 across all forty values is 2.6e−09, which is the epsilon, and the twenty people have become one row. Down the columns the same twenty people keep their differences, because a column of twenty heights has a spread worth dividing by. The row is the right axis when the row is wide and its entries are alike in kind, sixteen hidden units reading the same person, or the hundreds of numbers that describe one token in a sequence model, and it is the wrong axis for two measurements in different units.

Which is the sentence that decides between the two layers, and it is a sentence about the data.

RMS normalisation goes one step further along the row and subtracts nothing. It divides the row by its root mean square as it stands, so the row’s level survives and only its size is fixed. The finding behind it was empirical, the re-scaling was doing the work and the re-centring was not, and the reward is one reduction instead of two on the way forward and one route fewer on the way back, since there is no mean for an input to reach the loss through.

(1 + 49) / 2 = 25,   √(25 + 0.00001) = 5.000001
1 / 5.000001 = 0.2000,   7 / 5.000001 = 1.4000

Twenty of the crowd under RMS come out as 1.3622 and 0.3799 for the first person, 1.3390 and 0.4549 for the second, rows that stay apart because what the layer kept is each row’s proportions and not only which entry was larger. Conventionally the shift is dropped as well as the centring; the layer here keeps a shift defaulting to zero, which is the conventional model exactly and lets one shape serve the family.

The fourth layer with normalisation in its name standardises nothing at runtime. Its forward pass is a dense layer’s, value for value. What it changes is where the parameters live. Instead of a weight vector per neuron it stores a direction and a length, and rebuilds the vector from the two on the way past. A weight vector has a length and a direction whether anyone stores them apart or not, and they do different jobs, since the direction decides which way the neuron’s boundary faces and the length decides how sharply it answers.

weights = magnitude · direction / ‖direction‖

Written the ordinary way neither is a parameter, so gradient descent can only move the length by moving every component at once and hoping the total comes out right, and any step that changes the length also swings the direction. Written this way one number per neuron owns the length, and a step of the direction cannot change the length at all, because the length was divided out.

Write w for the slope with respect to the effective weights, which is a dense layer’s ordinary slope and is computed exactly that way, and u for the unit direction. Then per neuron the length’s slope is the part of w lying along u, which is the whole of what lengthening the vector can achieve, and the direction’s slope is the rest of w, rescaled, with that radial part projected out.

magnitude slope = w · u
direction slope = (magnitude / ‖direction‖) · (w − (w · u) u)

Dropping the projection is the mistake this layer is about. It leaves a block with the right shape and plausible magnitudes that trains, and the check is the same finite difference as Part 4. Three neurons read five of the crowd in standard units through a hyperbolic tangent, with stored directions whose lengths are 1, 2.5 and 0.5.

… the first visit trains a hundred and thirty small networks, which takes ten to twenty seconds

The two layers’ answers differ by exactly 0, because the forward pass is the same arithmetic. The direction slope agrees with the finite difference to 3.6e−10 and the length slope to 1.1e−10, against a largest true direction slope of 2.9973; the unprojected form misses by 1.8286, which is about six tenths of the largest slope it was trying to report.

And there is a structural check here too. Scaling a stored direction by any positive constant leaves every answer unchanged, so the loss is flat along the direction itself and the slope can have no part along it. The projected direction slope dotted with the direction is 8.9e−16, and the unprojected form’s is 3.6914.

A reparameterisation that moved the parameters to the same place would be a renaming, so the last thing to check is that it does not. One step at a stride of 0.1 from the same slope lands the weight-normalised layer’s effective weights 0.0364 away from where the dense layer’s land. The lengths say where the difference went.

Neuron 1 goes from a length of 1 to 1.2286 under weight normalisation and to 1.2347 under the dense layer, neuron 3 from 0.5 to 0.3040 and 0.3247, and under the dense layer part of each step that was meant for the direction changed the length as a side effect, where under weight normalisation only the length’s own slope could move it.

Whether any of this helps is a measurement, and the arrangement is the dropout page’s. Twenty-five people are too few to hold out a fixed group, so each run deals the crowd into five folds, trains on twenty and holds out five, and repeats until every person has been held out once, so the held-out accuracy is counted over all twenty-five.

Heights and weights are put into standard units using the training rows of each fold only. The whole of that is repeated under five seeds and five arrangements of the same sixteen-unit network, no normalisation, a batch layer between the hidden and output layers, a layer-normalising layer there, an RMS layer there, and a weight-normalised hidden layer with nothing inserted, which is a hundred and twenty-five training runs of four hundred full-batch epochs.

Two honest limits before the numbers. The stride of one was chosen for the plain network on the dropout page and is held the same under every arrangement, so what is compared is the arrangement and not a tuned version of each. And the batch layer comes after the rectifier rather than before it, because here an activation function belongs to the neuron and a normalising layer can only go between layers; the original paper standardises the score before the activation function, and both orders are used in practice.

… the first visit trains a hundred and thirty small networks, which takes ten to twenty seconds

Across the five seeds the plain network trains to 0.992 and holds out at 0.744, the dropout page’s numbers, since it is the same network under the same seeds. The batch layer trains to 0.904 and holds out at 0.640, and its seeds spread from 0.60 to 0.68, every one of them below the plain network’s mean.

Layer normalisation trains to 0.910 and holds out at 0.696, RMS to 0.870 and 0.720, and weight normalisation to 0.986 and 0.752, spread 0.68 to 0.80, which is the one arrangement whose seeds are level with the plain network’s rather than below them.

The textbook claim is that normalising the hidden block makes a network train better, and on this crowd it did not hold. The three activation normalisers all trained to a lower accuracy and a higher loss than the plain network at the same stride, and the batch layer held out worse by more than the seed spread can excuse. Weight normalisation, which changes the surface and not the function, finished 0.008 above the plain network on held-out accuracy and 0.0054 below it on loss, which is inside what the seeds move by.

The original claim was about speed rather than accuracy, and that is a different measurement. The curves are one seed on the whole crowd, the loss the backward pass measured after every epoch, and each is timed against a loss of 0.3.

… the first visit trains a hundred and thirty small networks, which takes ten to twenty seconds

The two row layers get there first, at epoch 16 against the plain network’s 77, so the speed claim held for them at the start of training. It did not hold for the batch layer, which crossed 0.3 at epoch 119, and it did not carry through to the end for either row layer, where layer normalisation finished at a loss of 0.1041 against the plain 0.0813 and RMS at 0.4284 with an accuracy of 0.76.

Weight normalisation and the plain network start from the identical loss of 0.9863, because the directions are drawn from the same stream in the same order, cross 0.3 on the same epoch, and part only at the end, 0.0753 against 0.0813.

The lower chart is the batch layer asked two ways after each step, properly by its running figures and again by the batch’s own statistics, and the two disagree on 88 of the 400 epochs; at epoch 10 it answers 0.76 one way and 0.72 the other. That is the lag of Part 3 seen from inside a training loop, and it is why a loss curve read while training is not quite the curve the deployed model would report.

The network the batch layer was built for was dozens of layers deep, trained on mini-batches of a large dataset, and its drift compounded through every layer. The network here has one hidden layer, trains on its whole twenty-row batch at once, and can only overfit the mixed middle of the crowd, which no method can generalise from.

Between those two lies most of practice, and what the measurements support is narrower than the folklore. A normalising layer changes the surface the gradient descends, and at a given stride that surface can come out easier or harder, so whether it helped is measured across seeds rather than read off the name.

A complete implementation states which axis the statistics run along, whether the variance is biased, where the epsilon goes, that the scale and shift exist and what they start at, what the two purposes are and which is the default, how the running figures start and by what rule and at what moment they move, whether the running variance is biased or not, which three routes the backward pass collects, what the parameter slopes are, and, for weight normalisation, that the direction slope has its radial part projected out and what a zero direction means.

Every row below was run rather than reasoned about, and the behaviours fall into three kinds, refused with a typed error, accepted with an answer that follows from the arithmetic, and accepted with an answer worth knowing about.

The single row while training deserves the last word, because it is the edge that is accepted rather than refused and is documented rather than defended. A training loop that feeds one row at a time through a batch layer trains on a block of zeros and raises nothing, which follows from the arithmetic and is still a mistake, and the remedy is the axis of Part 5 rather than a guard.