Convolutional Networks

Every picture on this page is sixteen pixels on a side, one channel of brightness, and holds one of four shapes, a cross, a square outline, a filled disc or a diagonal bar. Each shape is drawn at a random position, at a random size within a range, at a random brightness against a random background, with a little noise on every pixel, so that no single pixel can give the answer away.

The network learns from 240 of them, sixty of each kind, and is scored on another 240 it never sees while training. The box at the top of the page holds the first held-out picture of each kind.

The obvious first attempt reads the 256 pixels as a row and hands them to dense layers. A cross in the top left corner and the same cross in the bottom right share no lit pixel, so to a layer that reads a row they are two unrelated patterns, and the network has to learn every kind again at every position it might appear.

Sixty pictures of each kind is nowhere near enough for that. I built a dense network of about the same size as the convolutional one on this page, 256 pixels to 9 units to 16 to 4 scores, 2541 parameters, and trained it with the same loop on the same pictures.

It names about two held-out pictures in three, it does not finish learning even the pictures it was trained on, and moving a picture by one pixel changes its answer on about a third of them. That is the failure the rest of the page is measured against.

The convolutional network does two rounds of the same thing and then answers. A convolution sweeps small grids of weights across the picture, a pooling keeps the largest value in each two by two window and halves the picture, and the pair is repeated on what the first pair produced. The last maps are flattened into a row of 128 numbers, a dense layer turns that into 16, and a final layer gives one score per kind.

A convolution’s count does not depend on the size of the picture, since each filter is one small grid used at every position. It is the number of filters times the weights in one filter, which reads every channel beneath it, plus one bias each.

parameters  =  filters × (channels × 3 × 3 + 1)

Count the weights and biases in each layer separately. Each convolutional filter has one weight per input channel and kernel position, plus one bias. Each dense output has one weight per input and one bias.

first convolution  = 4 × (1 × 9 + 1) = 40
second convolution = 8 × (4 × 9 + 1) = 296
hidden dense layer = 128 × 16 + 16 = 2,064
output dense layer = 16 × 4 + 4 = 68
total parameters   = 40 + 296 + 2,064 + 68 = 2,468

Most of the parameters belong to the hidden dense layer.

Every number in the maps at the top of the page comes from the same small sum. Take the held-out cross, and the one place where any first-layer filter answers most strongly, which is the third filter at row 11, column 11, counting from zero at the top left. The filter lays its nine weights over the three by three patch centred there, multiplies each weight by the pixel beneath it, adds the nine products and the filter’s bias, and keeps the result if it is above zero.

The weights are large on the left column and small or negative on the right, and the patch is bright on its left and dim on its right, which is why this is where the filter answers most. One of the cross’s bars, two pixels thick, covers the left two columns of the patch.

The network is trained with the plain loop from the training page, 40 passes over the 240 training pictures in shuffled batches of 16 with a step size of 0.1, every layer corrected at once from the loss at the top. After each pass I read how many pictures it names correctly in both halves.

…

At weight seed 0 the loss over the first pass is 1.3691, which is close to the loss of spreading the probability evenly over four kinds, since that is the natural logarithm of 4, 1.3863. After the first pass the network names 0.346 of the held-out pictures. It first names every training picture at the end of pass 33, when the loss is 0.0347, and finishes at a loss of 0.0190 and a held-out accuracy of 0.971.

The starting weights are drawn at random, and a network this small ends somewhere different for each draw. Everything on this page that depends on training is reported at three weight seeds, and the widgets draw seed 0 unless a button says otherwise.

The three differ by 0.05 on the held-out half, from 221 to 233 pictures of 240, and the lowest final loss belongs to seed 2 while the best held-out score belongs to seed 0, so a lower loss on the training half did not order them. Seed 1 reached 0.938 at pass 27 and then fell back to 0.921 while its loss kept falling.

All three networks name every training picture by the end, and none of them names every held-out one. After the training half is perfect, the loss keeps falling, from 0.0347 at pass 33 to 0.0190 at pass 40 at seed 0, which is the network growing more certain about pictures it already names rather than learning anything new about the kinds. The training page covers the two curves and when to stop; what matters here is that 240 pictures are enough for this network to memorise, so the held-out half is the only score on the page that measures anything.

The usual account is that the first layer of a trained convolutional network learns edge detectors, the same kind of grid the filters and edges page builds by hand. To test it I took the Sobel grid from that page, turned it in eighths of a turn so that there is one for every direction an edge can face, and asked which of the eight each trained filter is closest to in shape.

Closeness is the cosine between the two grids of nine numbers after each filter’s own mean is taken off, which is 1 for the same shape and 0 for an unrelated one. At seed 0 the four filters come out at 0.675, 0.543, 0.814 and 0.794. That sounds like edges until it is set beside what nine random numbers manage, since with eight directions to choose from one of them is always fairly close.

Over 2000 random grids the closest Sobel grid is at a median of 0.466, and one draw in ten comes closer than 0.724.

The board also shows each filter as it was before training, and that changes the reading. The fourth filter, the one closest to an edge that is brighter to the lower right, was already at 0.754 when its weights were drawn, and training moved its nine weights by 0.09 of their starting length. It looks like an edge detector because the random draw put one there.

The third filter did move, by 1.86 times its starting length, and its closeness rose from 0.594 to 0.814, so of the two edge-like filters one was learned and one was inherited.

The second and fourth filters at seed 0 hardly moved, and the reason is the activation function after the convolution, which passes a score through when it is above zero and answers zero otherwise. Where it answers zero it also passes no slope back, so a filter whose scores are below zero almost everywhere receives almost no correction.

Over every position of every held-out picture, the second filter answers above zero on 0.08% of them and the fourth on 2.3%. Their weights sum below zero, to −3.395 for the second and −0.738 for the fourth, and on pictures that are brighter than zero nearly everywhere that is enough to keep them quiet from the first step.

A Sobel grid’s nine weights sum to zero, so a patch of even brightness gives it nothing however bright it is. The two filters that did train at seed 0 have weights summing to 3.256 and 3.766, with biases of −0.589 and −0.963, so each answers to a patch that is bright overall, more strongly if the brightness sits on one side.

That is closer to a detector for the edge of a lit region than to a pure edge detector, and on these pictures, where every shape is lit against a darker ground, the two are nearly the same thing.

The box at the top of the page carries one held-out picture of each kind through the network. The first convolution gives four maps the size of the picture, one per filter, bright where that filter’s pattern is. Pooling halves each to eight by eight, keeping the largest value in each window, so a bright answer survives even if it moved within its window.

The second convolution reads all four pooled maps at once through its own three by three windows, 36 weights and a bias per filter, and gives eight maps, and the second pooling halves those to four by four.

On the cross, the maps after the first convolution follow the bars, and the two nearly silent filters from section 10 are dark almost everywhere. By the second pooling each of the eight maps is sixteen numbers, and no longer looks like the picture at all. What it holds is where in the picture, to within four pixels, each of eight second-layer patterns was found.

The flattening lays the eight four by four maps out as a row of 128, the dense layer turns them into sixteen numbers between −1 and 1, and the last layer gives four scores that become probabilities. The sixteen numbers are the vector the vector for a picture page reads off this same network.

Three of the four are named with near certainty. The disc is named correctly by a margin of 0.048, with nearly half the probability on a cross, and the maps in the box show why a small filled disc and the middle of a thick cross are similar things to a filter three pixels wide.

A unit in the first map reads a three by three patch. A unit after the first pooling reads a two by two block of those, whose patches overlap, so it depends on a four by four patch of pixels. Each layer widens the patch by an amount set by its window and by how far apart its inputs sit, and pooling doubles how far apart the next layer’s inputs sit, so every layer after a pooling widens the view twice as fast.

field after  =  field before + (window − 1) × spacing before

spacing after  =  spacing before × stride

The geometry says which pixels a unit could depend on. To check it I brightened each of the 256 pixels of the held-out cross by 0.5, one at a time, and recorded which of them moved a chosen unit at all. The drawing shows the units stacked over the middle of the picture, one at each depth, and the box at the top of the page does the same for any unit you click.

…

No pixel outside a window ever moved its unit, at any depth. Inside, the counts are 9 of 9, 14 of 16, 63 of 64 and 92 of 100. The shortfall is the pooling and the activation function. A maximum passes on only the largest value in its window, so a pixel that feeds only a losing position does not reach the unit, and a pixel that feeds only a unit whose activation function answered zero does not either.

For the unit in the corner of the last maps the window runs from three pixels outside the picture to six inside, so 49 of its 100 pixels exist and the rest are the padding’s zeros; 45 of the 49 moved it.

Without pooling, the same two convolutions give a field of 3 and then 5, since the second convolution’s inputs are still one pixel apart. The pooling between them is what lets the second layer’s three by three window cover eight pixels instead of five, and the second pooling takes it to ten.

A single convolution with a ten-by-ten kernel sees ten pixels across at once. On a one-channel image, eight such filters would need:

large-kernel parameters = 8 × (10 × 10 + 1) = 808
two small convolutions = 40 + 296 = 336

The larger filters would look for a complete pattern directly. The two smaller convolutions build later features from the earlier features, using fewer parameters in this example.

Taking the two poolings out leaves both convolutions reading and answering at full size, so the last maps are eight of sixteen by sixteen and the flattening hands the dense layer 2048 numbers instead of 128. No arrangement removes pooling and leaves everything else as it was, since the pooling is what decides that width.

So the network without pooling has 13.4 times the parameters and does 3.9 times the arithmetic per picture, and anything measured on it measures the loss of pooling and the gain in parameters together.

Trained with the same loop from the same three seeds, the network without pooling names every training picture, as the network does, and does worse on the held-out half at every seed.

…

The held-out scores are 0.875, 0.821 and 0.817 against 0.971, 0.921 and 0.967. The gap, between 0.096 and 0.150 at matching seeds, is larger than the spread of either arrangement across its own seeds, so it is a difference in the arrangement. The larger network fitted the training half just as completely and carried less of it over to pictures it had not seen.

The reason usually given for pooling is that it makes a network indifferent to where a pattern sits, within a window. To test that I moved every held-out picture one, two and three pixels right, left, down and up, keeping only the moves that left the whole shape inside the frame and filling the strip left behind from the picture’s own edge, and counted how often the network’s answer changed.

…

With pooling a one-pixel move changes the answer on 5.5 to 11.5 percent of pictures, and without it on 16.8 to 24.9 percent, so pooling cut the sensitivity to a shift by between two and four times. It did not remove it. At seed 0 the network names 0.970 of the pictures moved by one pixel before the move and 0.939 after it.

A maximum over a window is unchanged by a move only if the largest value stays inside the same window. The windows are fixed to the grid, two pixels wide, so a move of one pixel carries half the positions across a window boundary, and wherever the winner crosses, the pooled value changes. After two poolings the grid of windows is four pixels wide, and the dense layer that reads the result gives each of the 128 numbers its own weights, so it is not indifferent to which window a pattern landed in at all.

Section 2 opened the page on the dense network, and here it is beside the others. Taking the convolutions away entirely and keeping the parameter count close, 2541 against 2468, the dense network names 0.663 to 0.729 of the held-out pictures against the network’s 0.921 to 0.971, and it is the only arrangement that fails to fit its own training half in 40 passes.

…

Its trouble shows on the moved pictures as well. A one-pixel move changes its answer on 30.8 to 37.9 percent of them, more than pooling taken away, and at seed 0 about five times the network’s 6.7 percent.

…

Parameters are only one cost. Every weight in a dense layer is used once per picture, and every weight in a convolution is used at every position it is swept to, so the two counts come apart.

The network and the dense one hold nearly the same number of weights, and the network does 11.8 times as much arithmetic on each picture, 9,216 multiplications in the first convolution and 18,432 in the second against 2,048 in its dense layer. That arithmetic is where the held-out accuracy came from. The training chart in section 5 prints how long the training took on the machine that answered, which moves from run to run while these counts do not.

One more arrangement asks what the second round of convolve and pool adds. Keeping only the first round leaves four maps of eight by eight, 256 numbers, for the dense layer, which now holds 4112 weights, so the network with one round has more parameters than the network with two, 4220 against 2468.

The second round raised the held-out accuracy at every seed, by 0.029 to 0.109, and lowered the share of answers a one-pixel move changed, while taking 1752 parameters away. In the geometry of section 14, one round leaves each number seeing four pixels by four, and two rounds leave it seeing ten, which is most of a shape.

A convolution and a pooling can read a picture of any size, since both are defined window by window. The dense layer cannot. Its weights are one per number of the flattened row, and the length of that row is fixed by the picture the network was built for.

A seventeen-pixel picture happens to fit, because a two by two pooling of an odd side keeps only whole windows and leaves the last row and column out, which is a quiet way to lose a pixel. The usual repair for other sizes is to pool each final map down to a single number, whatever its size, which makes the row as long as the number of maps; the network here does not do that, and so it reads sixteen by sixteen and sixteen by sixteen only.

The last layer turns four scores into four probabilities that sum to one, so every picture is shared out among the four kinds and no picture can be given to none of them. Shown sixty rings, a kind it never trained on, the network has no way to say that a ring is none of these.

…

It called 48 of the 60 rings a square outline, 9 a cross and 3 a filled disc, which is a reasonable guess in shape, a closed outline with nothing inside. Its confidence dropped, and not by much. The probability of its call averaged 0.861 on the rings against 0.948 on the held-out pictures of the kinds it knows, and 63% of the rings were called with more than 0.9.

Shown a picture with nothing on it at all, an even brightness of 0.15, it calls it a diagonal bar with a probability of 0.892.

Each of these is a fact about stacking windows, strides and a dense layer, and each is a decision any convolutional network faces, whatever builds it.