Dropout

Take the tangled crowd from the bagging page, twenty-five people measured by height and weight, twelve of them children and thirteen adults, with the middle of the crowd mixed on purpose so that no smooth boundary separates the two. Give a network sixteen rectified hidden units, which for twenty rows of two numbers is far more than it needs, and train it hard, four hundred full-batch steps at a stride of one.

It learns the mixed middle person by person. Held out fold by fold, five folds so every person is left out once and averaged over five seeds, it calls 99.2 percent of the rows it trained on correctly and 74.4 percent of the people it did not see.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

Under the first seed, whose folds the chart pools, the training accuracy reaches 0.89 by epoch 100, 0.96 by 200 and 0.99 by 400, and the held-out accuracy climbs too, from 0.64 to 0.76 to 0.84. Memorising did not visibly cost held-out accuracy on this seed, and I am not going to pretend it did; what it produced is the gap between the two lines, a quarter of the crowd across the five seeds, which is the network reporting a score it will not keep.

Here is what memorising looks like from inside the layer. Silence one hidden unit at prediction time, leaving the other fifteen as they were, and ask how much of the training accuracy survives. If every unit had learned something useful on its own, losing one would cost a little; if the units had learned to work in particular pairs, losing the wrong one would take its partner’s contribution down with it.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

On the network trained without dropout, eleven of the sixteen units can be silenced at no cost at all, and the whole fit turns out to rest on two of them. Silencing the thirteenth unit drops the training accuracy from 1.00 to 0.76 and silencing the twelfth drops it to 0.84, with the sixteenth, third and fourteenth costing 0.12, 0.08 and 0.04.

The right-hand panel is the same measurement on a network trained with dropout at one half, and Part 7 comes back to it; for now notice that silencing any one of its sixteen units changes its accuracy by exactly nothing.

The ensemble cure, many networks averaged, fixes this by making the partnerships disagree with one another, and it costs as many trainings as there are members. Dropout gets the same effect from one training run by breaking the partnerships as they form. On every training pass, each unit in the layer is independently silenced with some probability, so a unit that only means something beside its neighbour finds the neighbour missing on a share of the passes and is pushed toward carrying weight by itself.

Nothing is penalised and no weight is shrunk. The layer is made unreliable while it learns, and section 23 measures what the units did about that.

Each pass therefore trains a different thinned network, and since the thinnings share one set of weights, the thing that is used afterwards is something like the average of all of them. That framing is the paper’s and it explains what the method is worth; the framing that explains the mechanism is the one above, that a feature which works only alongside a specific partner now gets a bad score a share of the time.

The box at the top of the page holds one row of four units, the kind of numbers a dense layer answers with, and shows it twice. On the left is one training pass. A coin was tossed over every unit, the ones that lost are greyed out and send nothing on, and the ones that survived have been raised, so a bar that would have stood at 1 stands at 2. On the right is the same row while predicting, where no coin is tossed and nothing is scaled.

training     out_j = value_j · m_j / (1 − p),   m_j ∈ {0, 1}
predicting   out_j = value_j

keep probability = 1 − 0.5 = 0.5
survivor multiplier = 1 / 0.5 = 2

1 · 2 = 2,   2 · 0 = 0,   3 · 2 = 6,   4 · 0 = 0
training reads    2, 0, 6, 0
predicting reads  1, 2, 3, 4

A layer answers a whole batch at once, one row per person, and the coin is tossed separately over every entry of the block rather than once per unit for the batch. The grid below pushes the worked row through three times as three rows of one block, and the mask differs from row to row.

Under seed 10 the three rows keep two, three and two units, and no two rows keep the same set. A mask shared across the batch would let a unit rely on its neighbour for the whole batch and only lose it on the next one, which is a much weaker version of the rule; drawing per row means the partnership is broken for some people on every pass.

The probability can sit anywhere in the half-open interval from zero up to but not including one, and the scale follows it. At zero nothing is silenced, the scale is 1, and the layer is an expensive way of doing nothing, which is exactly what a control in a search over this setting needs. At one every unit would be silenced and the scale would be a division by zero, and the layer refuses to be built.

The raising is the half of the idea that is easy to skip. The next layer reads a weighted sum of these units, and silencing half of them lowers that sum by half on average. On the worked row the four units total 10, and the first draw’s survivors, 1 and 3, total 4 before any scaling. A network trained on sums of that size and then asked to predict with every unit present would meet inputs about twice as large as any it was trained on.

So the survivors are raised by one over the keep probability, and the doubled survivors total 8, which is nearer the 10 the whole row would have sent. Nearer and not equal, because the claim is about the average over many draws and not about this one.

E[out_j] = p · 0 + (1 − p) · value_j / (1 − p) = value_j

A claim about an average is checked by averaging. The chart under the bars at the top of the page follows the mean of each unit’s training output over four hundred draws from one seeded layer, and each line settles onto the dashed line at that unit’s predicting output.

u1   200 / 400 · 2 = 1.00
u2   192 / 400 · 4 = 1.92
u3   190 / 400 · 6 = 2.85
u4   204 / 400 · 8 = 4.08

The worked example is the derivation’s cancellation with a count in place of a probability, 192 of 400 in place of one half, which is why the means land near the targets rather than on them. The survival share of any finite run is close to 1 − p and not equal to it.

What the derivation does not say is that any single pass is close. Over the four hundred draws every unit of the worked row was silenced at least once and doubled at least once, so a unit holding 4 sent 0 on some passes and 8 on others, while its average settled at 4.08. The spread of one pass is on the order of the value itself, and it shrinks only in the average, roughly as one over the square root of the number of draws.

E[out²]   = (1 − p) · (v / (1 − p))²  =  v² / (1 − p)
Var[out]  = v² / (1 − p) − v²        =  v² · p / (1 − p)
sd[out]   = v · √(p / (1 − p))

at p = 0.5     sd[out] = v

unit    value    sd of one pass    sd of the mean of 400    gap measured
u1      1        1                 1 / 20 = 0.05            0.00
u2      2        2                 2 / 20 = 0.10            0.08
u3      3        3                 3 / 20 = 0.15            0.15
u4      4        4                 4 / 20 = 0.20            0.08

There are two places to put the correction, and they are not equally good. The original recipe trained with silencing alone and scaled the weights down by the keep probability at prediction time, which makes a saved model impossible to evaluate without knowing the probability it was trained with. The inverted form scales the survivors up while training, and prediction becomes the identity, so the layer disappears once learning is over.

original     training  out = value · m          predicting  out = value · (1 − p)
inverted     training  out = value · m / (1 − p)   predicting  out = value

Both agree in expectation, by the same cancellation, and the right-hand bars at the top of the page are the proof that the inverted form is the one running here. The predicting pass returns the row bit for bit, and the endpoint behind the widget returns that answer rather than asserting it.

The layer has to remember which entries it silenced, because the backward pass needs the same mask on the way down, and there is a tempting shortcut. The outputs are right there, a zero in them looks like a dropped unit, so why not read the mask back off the answers. The block below shows why not. Its rows hold exact zeros, the kind a rectified layer hands upward, and a rose outline marks every entry where the mask read off the outputs disagrees with the mask the layer actually drew.

Under seed 10 four of the twelve entries are misfiled, and every one of them is a zero that survived. The first row holds 0, 2, 0 and 4, the draw keeps its first and third units, and the row sends on 0, 0, 0, 0, because doubling nothing is nothing. Read off the outputs, that row would be recorded as entirely dropped, its blame set to zero everywhere, and two units that were present on the pass would learn nothing from it.

This would be a curiosity if exact zeros were rare where the layer lives, and they are the ordinary case. A rectified unit answers 0 for every negative score, and most of a scanned digit’s border is 0, so a dropout layer above either of those meets zeros on every pass. On the crowd network trained with dropout at one half, one training pass through the hidden layer produces four hundred readings, sixteen units for each of twenty-five people.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

Of those four hundred readings, 282 are exactly zero, and the mask keeps 142 of them. Reading the mask back off the outputs would misfile 142 of 400 entries on that one pass, more than a third, quietly and with nothing raised, and the network would still train and its loss would still fall.

So the layer carries the mask on its response, already scaled and ready to multiply, and its backward step refuses a response that arrives without one rather than guess. The pooling page’s layer took the opposite decision and recomputes its winners instead of storing them, and the difference between the two is the whole point. A pooling winner is a function of numbers the response already holds. A mask is a draw, and nothing that survives the forward pass can reproduce it.

A seeded dropout layer must be reproducible across a sequence of passes while still producing new masks on later passes. Its random generator therefore advances after every draw.

The layer keeps that generator state when a training step occurs. Reconstructing a seeded generator from its initial seed after every step would repeat the same mask, permanently removing the same contributions instead of varying them across training passes.

The chart uses four hundred successive draws from one generator. Under the same seed and call sequence, repeating the experiment reproduces those draws. An unseeded construction does not promise the same sequence on another run.

The backward pass hands every layer the slope of the loss at its outputs and asks for the slope at its inputs. For this layer the answer is one multiplication. A unit that was silenced contributed nothing to the loss, so it is owed nothing back, and a unit that survived was amplified by the scale on the way up, so its blame is amplified by the same scale on the way down. Both of those are multiplication by the scaled mask, which is why the mask is stored already scaled.

passed_down_j = arriving_j · m_j / (1 − p)

With a block of ones arriving, what is passed down is the mask itself. Under seed 10 the three rows of the worked block pass down 2, 0, 2, 0 and 2, 0, 2, 2 and 0, 2, 2, 0. The layer has no weights of its own, so it reports no parameter gradient at all rather than a block of zeros, which would be a small lie about having something to learn.

A backward step that is wrong by a constant factor still trains. Forgetting the scale on the way back gives a slope too small by the keep probability everywhere, the network still converges, and the effective learning rate is quietly not the one that was configured. The check that settles it is the one the backpropagation page uses everywhere, a finite difference. Nudge one input up and down by a hair, rerun the forward pass under the same seed so the same mask is drawn, and difference a weighted total of the outputs.

The widget in section 14 shows the finite difference beside the passed-down block and reports the largest gap between them, 2.1e−09 on the worked block and 4.1e−09 on the rectified one. An implementation that forgot the scale would disagree by a factor of two at every kept entry, and one that read the mask off the outputs would disagree at every kept zero.

Now the rectified block through the backward step. Its first row sent on nothing but zeros, and the blame passed down to it is 2, 0, 6, 0 under an arriving slope of 1, 2, 3, 4, because the first and third units were present on the pass and their blame is the arriving slope times the scale of 2. The two entries with the identical output of zero, one kept and one dropped, are owed entirely different amounts, and nothing in the answers distinguishes them.

arriving slope         1,  2,  3,  4

mask the layer drew    2,  0,  2,  0
passed down            1 × 2,  2 × 0,  3 × 2,  4 × 0   =   2, 0, 6, 0

mask read off outputs  0,  0,  0,  0
passed down            1 × 0,  2 × 0,  3 × 0,  4 × 0   =   0, 0, 0, 0

That is the whole argument for storing the mask, made with numbers. Press the rectified block button in section 14 and the passed-down grid shows the zeros that were still paid, outlined in rose where the shortcut would have paid nothing.

A dense layer, a convolution, a pooling window and a flattening all do exactly the same thing whether the network is learning or answering. This layer does not, and it cannot be written without being told which is happening. Silencing units while answering would make the model’s prediction a matter of luck, so the silencing has to stop, and the only way it can stop is if the pass says what it is for. The batch normalisation layer has the same need from the other direction, standardising by the batch in front of it while learning and by remembered figures while answering.

So every forward pass here carries a purpose, one of exactly two, training or predicting, and a layer that does not care ignores it. It is a closed pair of names rather than a true-or-false switch, so that a call site reads as what it means and a wrong value is a mistake in the caller’s source rather than a plausible-looking boolean.

There are two ways to get the purpose wrong, and they are not equally bad. Forget to say training and the network never drops anything, so it trains as the plain network would. Forget to say predicting and every answer the model gives is drawn from a coin.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

The two mode errors have different effects. First, take the network trained with a dropout probability of one half. In prediction mode, its classification accuracy on this crowd is 0.84.

Leaving dropout active makes the prediction depend on a fresh mask. Across twenty calls, accuracy is 0.84 on fifteen calls and 0.80 on five. The class decisions change less than the probabilities: one observation's predicted adult probability ranges from 0.29 to 1.00 across those calls, compared with 0.996 in prediction mode.

Second, omitting dropout during training means the intended regularization never takes place. The zero-dropout and no-dropout versions in the matched experiment produce identical outputs. The mode setting therefore determines whether the model actually receives the intervention being studied.

Given that asymmetry the default is chosen to protect against the worse mistake. A pass that does not say why it is running is a predicting pass, so a caller who forgets gets a slightly slower descent rather than a random model. The backward pass states training for itself, since a backward pass is training by definition and a dropout layer that answered deterministically inside one would have the gradient describe a network the step is not about to build, so an ordinary training loop never thinks about the purpose at all.

The only place a caller says it is when reading a training-mode answer on purpose, which is what the experiment above did to measure the swing.

Whether dropout helps is a measurement, and at this scale the measurement has to be arranged carefully or the answer is noise. Twenty-five people are too few to hold out a fixed group, so each run deals the crowd into five folds, trains on twenty and holds out five, and repeats until every person has been held out once, so the held-out accuracy is counted over all twenty-five.

Heights and weights are put into standard units using the training rows of each fold only, so no held-out person’s measurements reach the fit. The whole of that is repeated under five seeds, each seeding the weights, the deal and the draws together, at three settings, no dropout layer at all, dropout at 0.2, and dropout at 0.5, which is seventy-five training runs of four hundred epochs.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

Across the five seeds the plain network trains to 0.992 and holds out at 0.744. Dropout at 0.2 trains to 0.950 and holds out at 0.752, and dropout at 0.5 trains to 0.886 and holds out at 0.728. The gap between training and held-out accuracy narrows from 0.248 to 0.198 to 0.158, and it narrows almost entirely from the training side; the held-out means are 0.008 above and 0.016 below the plain network’s, where the seeds at any one setting spread from 0.64 to 0.84.

The textbook claim is that dropout raises held-out accuracy, and on this crowd it did not hold in any way the seed spread can distinguish from chance. What it did do is stop the network reporting a training score it would not keep.

The curves of the first seed show the other thing dropout changes, which is the loss the training loop can see. The backward pass measures its loss on the way forward, and under dropout that is the loss of one thinned network, a different one every epoch.

… the first visit trains seventy-five small networks, which takes ten to twenty seconds

Without dropout the measured loss rises between consecutive readings on only 1 of 99 occasions. At 0.2 it rises on 40 of 99 and at 0.5 on 48 of 99, about as often as it falls, while the accuracy read while predicting still climbs underneath. At the end of the four hundred epochs the plain network stands at 0.99 trained and 0.84 held out on this seed, dropout at 0.2 at 0.97 and 0.80, and dropout at 0.5 at 0.89 and 0.72.

This is the seed on which the plain network did best of the five, and on it dropout helped least, which is worth seeing beside the averages.

The silencing measurement of section 2 is where dropout shows what it did, whatever the held-out score says. Trained on the whole crowd without it, the network rests on two units, and silencing the thirteenth alone costs 0.24 of its training accuracy. Trained with dropout at one half, the same network stands at 0.84 with every unit present and at 0.84 with any one of the sixteen silenced, a mean drop of 0.000 and a largest drop of 0.000, and none of the sixteen units is dead, so the insensitivity is not a network that has nothing left to lose.

That is the co-adaptation of the paper’s title, removed and measured. The plain network came to rest on two units, and losing the thirteenth costs it 0.24; the dropped network can lose any one of its sixteen and stay at 0.84. On this crowd the second network is not measurably better at new people, and it is a great deal less fragile.

The network the method was built for had sixty million parameters and two dense layers of 4096 units each above a stack of convolutions, and the paper reports it overfitting badly without dropout and not with it. The network on this page has sixteen hidden units and twenty-five people, and the only thing it can overfit is the mixed middle of the crowd, which no method can generalise from because there is nothing there to generalise.

Between those two lies most of practice, and the rule of thumb the measurements support is that dropout is a way of spending capacity you already have more carefully. It is not a substitute for having enough data, and on data this small its visible effect is on the gap and on the fragility rather than on the score.

A complete implementation states the drop probability and its legal range, whether the scaling happens at training or at prediction time, that the draw is independent per entry and per row, what the layer does under each purpose and which purpose is the default, that the mask is carried on the response and what the backward step does when it is missing, what the backward step passes down and that it reports no parameter gradient, how the generator is seeded and what stepping the layer does to it, and that the layer reads and answers the same arrangement so it can be placed anywhere in a sound stack.

Every row below was run rather than reasoned about. A refusal is a typed error with a message written to be read; an acceptance is what the layer did instead.

The two accepted edges in the backward step are the ones worth knowing about. The arrangement check settles that the arriving block and the response belong together, and a non-finite slope is a fault of the layer above rather than of this one; the same-shape case is a response carrying no record of which layer produced it, which a stack never produces because it pairs each layer with its own response by construction.