Convolution

The grid on the left of the playground is eight cells by eight, with a four by four square painted in the middle of it, and every cell holds either a zero or a one. That is sixty-four numbers, and it is the picture every claim on this page is made about. The small grid in the middle holds nine weights.

The grid on the right is what comes back when those nine weights are swept across the picture, and it is six cells by six because a three by three window fits into an eight by eight picture in six places along each side.

The square sits in rows 2 to 5 and columns 2 to 5, counting from zero, so its left edge is the boundary between column 1 and column 2 and its right edge the boundary between column 5 and column 6. Every worked number below comes from that arrangement, and the widgets on the page all start from it, so a reader can hold one picture in mind for the whole of the page rather than a fresh one per section.

A dense layer takes a list of numbers and answers with another list, and each of its answers is a weighted sum of the whole input. To hand it this picture we first have to flatten it, so the eight rows are laid end to end into a list of sixty-four. Cell row 0 column 0 becomes entry 0, cell row 0 column 1 becomes entry 1, and cell row 1 column 0 becomes entry 8.

Once that has happened the layer has no way of knowing which entries were neighbours. Entry 0 and entry 1 were beside each other and entry 0 and entry 8 were one above the other, and entry 0 and entry 37 were nowhere near each other, but all three of those pairs look identical to a weighted sum.

The layer is not forbidden from learning the geometry, and given enough examples it can work out that certain entries tend to agree; it simply has not been told, and everything it comes to know about the arrangement has to be paid for out of the data.

flattened[8 · row + column]  =  picture[row, column]

answer[j]  =  bias[j]  +  Σ over i of  weight[j, i] · flattened[i]

There is a sharper way of putting the same thing. Take any fixed shuffling of the sixty-four positions, apply it to every picture the layer will ever see, and apply it to the columns of the weight matrix as well. Nothing the layer answers changes, because the sum is over the same set of products in a different order. A reader who found that hard to believe should look at the next section, where it is measured rather than argued.

Rather than take that on trust, we can run it. A fixed shuffle moves the picture’s sixty-four cells about, and the same shuffle is applied to every weight of a layer that reads a row. Then both readers are given both pictures, the original and the shuffle, and asked what they answer. The measurement is what the widget below prints on load.

The rewired row reader answers what it answered before. Its largest disagreement across all thirty-six of its outputs is 2.2 × 10⁻¹⁶, which is one rounding step in double precision and not a difference of meaning; the answers are not bit for bit identical only because the products were added up in a different order. The same two pictures put through one convolution give answers 6 apart at the worst cell, with 27 of the 36 cells changed, on a map whose largest entry at rest is 4.

The shuffle moved 18 of the 64 cells, so it was not a violent rearrangement, and it was still enough to destroy every edge in the picture. Look at the two swept maps side by side. The one on the original has a clean column of positive answers where the square begins, nothing across its interior, and a clean column of negative answers where it ends. The one on the shuffle has neither.

The repair is to stop giving the layer the whole picture at once. Choose a small window, three cells by three, place it over the top left corner of the picture, and form one weighted sum of the nine cells beneath it. Then move the window one cell to the right and form another, using the same nine weights. Carry on until the window reaches the right edge, drop down a row, and start again from the left.

Two things about that procedure are worth stating before any arithmetic. A weight only ever multiplies a cell inside its own window, so whatever the nine weights say, they say it about a neighbourhood; that is locality. And the same nine numbers answer at every position, so what they have learned to notice they notice everywhere rather than once per place; that is weight sharing. The rest of this page is those two facts followed through.

The nine weights are called a kernel, and one kernel with one bias is a filter. The bias is added once to every answer, not once per window position, which is worth saying early because it is where the count of parameters gets its second term. Sweep the filter across the picture and what comes back is a map, one number for every place the window fitted.

The kernel the page opens with has a negative left column, a zero middle column and a positive right column. A window that is dark on the left and lit on the right therefore sums to something large and positive, a window that is the same shade all the way across sums to zero because the two columns cancel, and a window that goes from lit to dark sums to something negative. It is a detector for a vertical edge, and it is the vertical half of the pair usually called the Sobel operator.

answer[i, j]  =  bias  +  Σ over u, v of  kernel[u, v] · picture[i + u, j + v]

with u and v each running over 0, 1, 2

Nothing in that sum depends on i and j except which cells are read. The kernel entries are the same numbers whatever i and j are, which is the sharing written down as an equation rather than as a sentence.

Take the answer cell at row 2, column 0, which is the one the playground shows whenever nothing is hovered. Its window covers picture rows 2 to 4 and columns 0 to 2. The first two of those columns are outside the square and dark, and the third is the square’s left edge and lit, so the window reads three zeros, three zeros and three ones.

the window        the kernel         the products
0  0  1          −1   0   1          0  0  1
0  0  1     ×    −2   0   2     =    0  0  2
0  0  1          −1   0   1          0  0  1

sum of the nine products  =  1 + 2 + 1  =  4

Six of the nine products are zero because the cell beneath them is zero. The three that survive are the kernel’s right column, 1 and 2 and 1, each multiplied by a lit cell, and they add to 4. The sweep answers 4 at that position, exactly, and the playground will show the nine products underneath if the cell is hovered.

Move one cell to the right and the window reads columns 1 to 3, still dark on the left and lit on the right, so it answers 4 again. Two cells further along the window lies entirely inside the square, every cell is lit, and the two columns cancel to 0. At column 4 the window straddles the square’s right edge, lit on the left and dark on the right, and the sum is −4.

Rows 0 and 5 answer 1 rather than 4 because their windows only overlap the square by one of their three rows, and rows 1 and 4 answer 3 because they overlap it by two. The map is a picture of where the vertical edges are and how much of the window is sitting on them, and it is the same nine weights that produced all thirty-six of those numbers.

Nothing about the sweep is particular to edge detection. The same procedure with nine different numbers answers a different question about every neighbourhood, and the playground’s preset buttons are four more of them. Switching the kernel does not change the picture, the window, the stride or the size of the answer, only what the nine weights are asking.

Turn the kernel on its side and it finds the square’s top and bottom edges instead, with the interior at zero and the vertical edges invisible. Set all nine weights to a ninth and it answers the mean of the window, so the interior comes back at 1, the corner where only one cell is lit comes back at a ninth, and the map is a blurred copy of the picture.

Paint the diagonal instead of the square and both edge kernels find it, each at half the strength the square gave them, because a diagonal presents each of them with only part of an edge.

There is a naming inaccuracy here that is universal and worth knowing about. In signal processing a convolution flips the kernel end over end, top to bottom and left to right, before sliding it, and the sum with the kernel unflipped is called a cross-correlation. What this layer computes, and what every machine learning framework computes under the name convolution, is the unflipped sum.

cross-correlation   answer[i, j]  =  Σ  kernel[u, v] · picture[i + u,     j + v]
convolution         answer[i, j]  =  Σ  kernel[u, v] · picture[i − u + 2, j − v + 2]

The consequence for a reader is small but real. A kernel copied out of a signal processing reference has to be flipped before it will do here what the reference says it does, and a kernel that is symmetric under that flip, which the blur is and the vertical edge kernel is not, will behave the same either way.

Take the sharing away for a moment and keep only the window. Imagine a layer that gives every one of its thirty-six answer cells its own private set of nine weights and its own bias, but still lets each of them read only the nine cells under its own window. That layer is local and shares nothing, and counting it separates the two savings.

window, nothing shared  =  36 positions × (9 weights + 1 bias)  =  360
dense, equal width      =  36 outputs × (64 weights + 1 bias)  =  2,340

Locality alone takes 2,340 down to 360, which is a factor of six and a half, and the ratio is exactly the ratio of 65 to 10, what one output reads plus its bias in each case. That is worth having, and it is nothing like the whole of the saving the page is about.

Now add the sharing back. The thirty-six sets of nine become one set of nine, and the thirty-six biases become one bias. The layer holds ten numbers and answers with the same thirty-six it answered with before.

The thing to notice is what the ten does not depend on. It does not depend on the size of the picture, since the same nine weights sweep a picture of any size. It does not depend on where in the picture anything is. It depends on the window and on how many input channels there are, and on nothing else, which is why the count in the geometry table of Part 5 stays flat down a column where every other number moves.

combined saving factor = locality factor × sharing factor
                       = 6.5 × 36 = 234

Written out in general the count has two terms, and both are short. A filter holds one weight for each cell of its window in each input channel, plus one bias, and a layer holds as many filters as it was asked for.

convolution  =  n_filters × (channels × k × k)  +  n_filters
dense        =  n_outputs × n_inputs  +  n_outputs

A colour picture makes the difference between the two terms visible. Three channels of an eight by eight picture through one filter of three by three costs 28 parameters, which is 27 weights and a bias, because a filter reads every input channel at once rather than one per channel. The picture grew by a factor of three and the filter grew by a factor of three, so the layer got a little larger; a dense layer reading the same picture would have grown by a factor of three in its width as well.

The convolution’s count is flat in the size of the picture and the dense layer’s is quadratic in it, since the dense layer’s count is how many numbers went in times how many come out and both of those grow with the picture. So the ratio between them grows without any change to the layer.

The example carried through these pages is a twenty-eight by twenty-eight picture, the size of a handwritten digit, read by eight filters of three by three. The convolution holds 80 parameters. A layer keeping the window and sharing nothing would hold 54,080. A dense layer reading the same 784 cells and answering the same 5,408 numbers holds 4,245,280, which is 53,066 times what the convolution holds, for an answer of exactly the same size.

Move the picture side slider and watch which bar moves. The convolution’s bar does not, because nothing in its count mentions the picture; the other two climb. The bars are on a log scale, since at the default setting the first is 80 and the last is over four million and a linear scale would leave the first invisible.

A kernel can only ever say something about a neighbourhood, so one layer of them cannot tell a face from a scrambled arrangement of the same eyes and mouth, since every window contains at most part of one feature and no window contains two of them at a distance. A wider kernel would answer that complaint and would hand back the parameter saving while doing so, since the count is the window area times the channels. The route taken instead is a second convolution laid on top of this one.

A second three by three window placed over the first layer’s map covers a five by five patch of the original picture, because each of its nine inputs was itself formed from a three by three patch and those patches overlap. A third covers seven by seven. That growing patch is called the receptive field, and stacking layers is what widens it while each layer keeps its own small count of weights.

receptive field after L layers of a k by k window at stride 1

    side  =  1  +  L · (k − 1)

k = 3, L = 1   side = 3
k = 3, L = 2   side = 5
k = 3, L = 3   side = 7

Take the picture, move every cell one column to the right, and sweep it again. The answer we get is the answer we had, moved one position to the right. Not approximately, and not usually; every cell of the map that has somewhere to move to agrees with the moved answer exactly, and the widget below reports the largest disagreement as 0.

That is a statement about the arithmetic, not about the training, so it holds whatever nine numbers the kernel is carrying. Switch the kernel with the buttons and the maps change completely while the gap stays at 0. It is the property Fukushima was after in 1980, and it is what makes a filter that has learned an edge in one place a filter that has learned it everywhere.

The square really is moving under it. At a shift of one, 8 of the picture’s cells change and 18 of the map’s thirty-six do; at a shift of two, 16 cells change and all 36 of the map’s do. The map is not staying still, it is travelling with the square, which is a different and stronger claim.

The comparison that makes the point is a dense layer built to answer the same thirty-six numbers from the same sixty-four cells. Two of them sit beside the sweep in the widget above. The first holds a single detector, which is the page’s own nine weights written into one window position of the picture and zeros in all the other fifty-five places.

At rest that detector answers 4, since the square’s edge is sitting exactly under it, and it is the same 4 the sweep answers at that position. Move the picture one column and it answers 0. Move it two and it answers 0. Move it three and it answers 0. It has learned the edge at one place and has learned nothing at all about the same edge one cell over.

The second is an ordinary draw of thirty-six neurons over sixty-four cells, laid out as a six by six block purely so it can be printed beside the sweep’s map, and compared against its own resting answer moved by the same amount. At a shift of one its answers move by 1.753 where the largest answer it gives anywhere on that picture is 0.896. The change is larger than the thing being changed, and all 36 of its outputs move at every distance.

The proof is one line of algebra and worth doing, because it shows exactly which assumptions it rests on. Let the moved picture be the original read one column to the left, so that the moved picture at column j holds what the original held at column j minus one. Substitute that into the sum.

moved[row, column]  =  picture[row, column − 1]

answer_moved[i, j]  =  Σ  kernel[u, v] · moved[i + u, j + v]
                    =  Σ  kernel[u, v] · picture[i + u, j + v − 1]
                    =  answer[i, j − 1]

The middle line is the whole of it. The kernel entries did not change when the picture moved, because they do not depend on the position, and so the sum at position j of the moved picture is character for character the sum at position j minus one of the original. A layer whose weights differed by position could not take that step, which is why the locally connected layer of section 10 has no such property.

It is easy to read the last three sections as saying the layer does not care where things are, and that is not what they say. The answer moved. A layer above this one still sees a different map for a moved picture, and if what it needs is a single verdict that does not change under a shift then something else has to supply it.

The two usual somethings are pooling, which summarises a small neighbourhood of the map and so absorbs a small movement inside it, and a summary taken over the whole map at the end, which throws the position away entirely. Both are separate layers with their own costs, and the pooling page measures how much of a shift the first of them actually absorbs.

There is a second limit, at the edges. A square sitting against the right border and moved one column further right loses part of itself, so nothing about the sweep can recover an answer for a thing that is no longer in the picture. The widget compares only the columns the move leaves inside the map for exactly that reason.

One filter answers one question about every neighbourhood, and a picture has more than one question worth asking of it. So a layer holds a bank of filters rather than one, each with its own nine weights and its own bias, and each sweeping the same picture independently of the others. The answer is one map per filter, stacked.

Three filters over the page’s picture answer with three maps of six by six, which is written as an arrangement of (3, 6, 6), and the whole bank costs 30 parameters, ten per filter. The vertical edge filter answers most strongly at row 2 column 0, on the square’s left edge. The horizontal one answers most strongly at row 0 column 2, on its top edge. The blur answers most strongly in the middle, at row 2 column 2, where the whole window is inside the square.

Tick and untick the filters and watch the arrangement change in the readout. Only the first extent moves, since adding a filter adds a map and changes nothing about how large a map is. That first extent is the number of channels the next layer will read.

That stack of maps is what the next convolution reads, and it reads it the way this one read a colour picture. A filter in the second layer holds one three by three grid of weights for each of the maps beneath it, and its answer at one position is the sum over all of them at once. So the second layer’s filters are wider than the first layer’s even though their windows are the same size.

reads    (channels, height, width)
answers  (n_filters, out_height, out_width)

one filter's weights  (channels, k, k)
the whole bank        (n_filters, channels, k, k)

That four-dimensional arrangement is kept on the layer itself, and only flattens it to two dimensions when a gradient block is handed out and back, which section 26 returns to. What matters here is that a filter is never a flat grid once there is more than one channel, and that the first extent of what a layer answers is always the number of filters it holds.

first layer = 8 × (1 × 3 × 3 + 1) = 80 parameters
second layer = 16 × (8 × 3 × 3 + 1) = 1,168 parameters
ratio = 1,168 / 80 = 14.6

A dense layer’s output width is a number somebody chose. A convolution’s is arithmetic, and every term in it is known before any picture arrives. A window of side k on a picture of side n, with p rows of padding added at each edge and the window moved s cells at a time, fits at a position every s cells until its far edge runs out of picture.

out  =  ⌊(n − k + 2p) / s⌋  +  1

n = 8, k = 3, p = 0, s = 1     ⌊5 / 1⌋ + 1  =  6
n = 8, k = 3, p = 1, s = 1     ⌊7 / 1⌋ + 1  =  8
n = 8, k = 3, p = 0, s = 2     ⌊5 / 2⌋ + 1  =  3
n = 8, k = 3, p = 1, s = 2     ⌊7 / 2⌋ + 1  =  4
n = 8, k = 5, p = 0, s = 1     ⌊3 / 1⌋ + 1  =  4

The floor is where a stride that does not divide evenly loses its last partial step, and it is the only place in the formula where information is thrown away. The table below builds a layer for every combination of window, stride and padding at whatever picture size is chosen, and reads the answer off the layer’s own shape without sweeping a single cell.

Look down the parameters column while the picture side slider moves. It does not change. Every other number in the table depends on the picture and that one does not, which is section 11’s claim shown rather than argued.

Without padding the border of the picture is under-read, and the amount is easy to count. On the page’s eight by eight picture under a three by three window at stride one, the top left cell is inside exactly 1 window position and the middle cell is inside 9 of them. The corner contributes to a ninth as many answers as the middle does, so whatever is happening at the corner is nine times less able to influence what the layer says.

Adding one row and column of zeros at each edge before the sweep fixes both problems at once. The answer comes back at eight by eight rather than six by six, so the picture stops shrinking, and the corner’s coverage goes from 1 window position to 4. The geometry table above reports both counts, and the rows where the answer is the same size as the picture are shaded.

Zeros specifically, and the reason is that a zero contributes nothing to any sum it enters, so the border adds no evidence of its own. It is still an assumption, and the assumption is that the world outside the picture is dark. A filter looking for a bright region will therefore find an edge along the border of a padded picture that was not there before the padding was added.

Stride is how far the window moves between positions, and raising it is a way of shrinking the answer without adding a layer. At a stride of two on the page’s picture the answer comes back three by three rather than six by six, which is a quarter as many numbers carried forward.

It is not free, and the cost is countable. At a stride of two with a three by three window, 15 of the picture’s 64 cells are inside no window position at all, so 15 cells of the picture contribute nothing to any answer the layer gives. Nothing refuses that, and nothing should, since it is a legitimate thing to want; it is simply worth knowing before choosing the setting.

Push it further and the effect grows. A three by three window at a stride of four answers with a two by two map and leaves 28 of the 64 cells outside every window, which the failure contracts of Part 7 record as one of the three things that are accepted rather than refuses.

Because all four terms are integers known in advance, a setting whose answer would be smaller than a single cell can be refused the moment the layer is built. Ask for a window of seven over a picture of four and the arithmetic gives −2, which is not an extent, and the refusal says so with the sum written out rather than with a message about an array shape.

this convolution's answer would have height
(4 − 7 + 2 · 0) // 1 + 1 = −2,
so the window does not fit over what it reads

Try that setting in the geometry table by dragging the picture side down to 4, and the rows with a window of five and of seven come back carrying that refusal instead of numbers. The alternative, which is what happens when the extents are only worked out as the data flows, is discovering the mistake several layers downstream in the middle of a training run, with a message about a matrix that will not multiply.

When the layer is being trained, something above it hands down one number for every cell of the map, saying how the loss would change if that cell answered a little higher. The layer has to turn those thirty-six numbers into three things, a slope for each of its nine weights, a slope for its bias, and one number for each cell of the picture to hand down in turn.

The bias is the easy one. It was added once at every answer position, so its slope is the arriving numbers added up. The weights are the interesting one, and the reason is the sharing. A single weight multiplied a cell at every one of the thirty-six positions, so its slope is a sum over all of them, and that sum is the sharing seen from behind.

∂loss / ∂kernel[u, v]  =  Σ over i, j of  arriving[i, j] · picture[i + u, j + v]
∂loss / ∂bias        =  Σ over i, j of  arriving[i, j]
∂loss / ∂picture[r, c] =  Σ over the windows covering (r, c) of
                          arriving[i, j] · the kernel entry that read it

Both mistakes have the same shape, which is that they run perfectly and train something slightly other than what was asked for. The first is writing the kernel slope with an assignment where a sum was wanted, which leaves the layer training on the last window it happened to visit. The second is walking the input cells and asking which window each belongs to, rather than walking the output positions and adding into the cells each one covered, which loses terms at the border and under a stride greater than one.

Both are visible on the page’s own square if a slope of one is sent to every answer cell, because then every number in the answer is a plain count and can be checked by hand. Each of the nine kernel entries comes back wanting 16, which is exactly the number of lit cells in the picture, since the entry multiplied a one at sixteen of its thirty-six positions and a zero at the other twenty. The bias comes back wanting 36, one for each position it was added at.

The blame handed down to the picture is the other half. A cell read by every one of the nine kernel positions collects the sum of the nine weights, which for this kernel is 0, so the whole interior of the picture receives nothing. The very corner cell is read by one window position only, at the kernel entry holding −1, so it collects exactly −1. Thirty-two of the sixty-four cells receive nothing at all, and they are the ones where the kernel’s columns cancel.

Hand-checking one fixture is not a test of the implementation, so the reported slopes are also compared against measured ones. Each parameter is moved a millionth up and a millionth down, the loss is remeasured both times, and the difference divided by the total movement is what the slope must be. That is done for every kernel weight, every bias and every input value, on three fixtures, and the second table in the widget above reports the worst disagreement in each block.

The three fixtures are chosen for what each can catch. A plain sweep at stride one with no padding is the base case. Padding by one is where an implementation that forgets to strip the invented border off the blame shows up. Two channels at a stride of two is where an implementation walking the input rather than the output positions loses terms. On the run behind these figures the worst disagreement anywhere across the three fixtures was 7.8 × 10⁻⁹, on the inputs of the padded fixture, against reported slopes as large as 13.9.

Written the way the definition reads, the sweep is seven nested loops, over the rows of the batch, the filters, the two output axes, the input channels and the two kernel axes, with one product added at the innermost step. That version can be checked line by line against the equation, and on the page’s eight by eight picture it adds up 324 products, nine for each of the thirty-six answer cells. Through eight filters on a picture twenty-eight on a side it adds up 48,672, one at a time.

None of those products depends on any other, so the order they are added in is ours to choose. Lay every window of the picture out as one row of a table, nine numbers long for a three by three kernel on one channel, and the thirty-six windows of our square become a table of thirty-six rows and nine columns.

Write the kernel bank as a table of nine rows with one column per filter, and the whole sweep is the first table times the second, which is a matrix multiply, the operation numerical routines have been tuned hardest for. The windows do not even have to be copied to be laid out this way, since each one is the same stretch of the picture read from a different starting cell, so the table of windows is a way of reading the picture and costs no copy.

answer[f, i, j]  =  bias[f] + Σ over c, u, v of  kernel[f, c, u, v] · picture[c, i·s + u, j·s + v]

(windows × c·k·k)  ·  (c·k·k × filters)  =  (windows × filters)

The widget runs both versions on the machine serving this page, at three sizes, beside a dense layer reading the same numbers and answering the same numbers, and it compares the two versions’ answers. They agree to 1.8 × 10⁻¹⁵ at the largest size, a few rounding steps, which is what adding the same products in a different order costs and nothing more. On the runs I made behind this page the loops took between about 115 and 330 times as long as the multiply at the largest size, and only one to two times as long on the page’s own picture, where there are too few products for the multiply’s fixed setup to be repaid.

The dense layer is kept beside them because it corrects an easy reading of the parameter count. On the page’s picture the sweep holds 10 parameters against the dense layer’s 2,340 and is still the slower of the two, between about six and sixteen times on my runs, since multiplying a row of sixty-four by a table of sixty-four by thirty-six is almost nothing and laying the windows out costs more than that.

At twenty-eight on a side through eight filters the dense layer has 4,245,280 parameters to work through against the sweep’s 48,672 products, and the order reverses, the sweep taking between about a tenth and two fifths of the dense layer’s time. That is far less than the eighty-seven-fold gap in arithmetic would suggest, because at these sizes most of the sweep’s time is fixed overhead and the widget shows it barely moving from the smallest picture to the largest.

Two convolutions can agree on every word of the description above and still answer different numbers on the same picture. The decisions that separate them are mostly invisible until that happens, so it is worth writing down what a complete implementation has to fix. The table records what this one fixes and where it declines to offer a choice at all.

Below is every edge asked of a convolution as this page loads, with what came back rather than what ought to. Nineteen are attempted, sixteen are refused, and every one of the sixteen names itself in words that say what was wrong rather than in a message from the interpreter.

Two of those refusals are there because the natural mistake would otherwise escape as a bare failure. Giving a whole number where three extents are wanted is exactly what a dense layer’s width looks like, and without a guard it would come back saying that an integer is not iterable; here it says that a convolution reads channels, height and width, and got 8, which is not a sequence of extents.

Giving a flag where a count is wanted is the other, since a flag indexes as one and would otherwise slip through as a layer with a single filter.

Three are accepted, and each of the three is a decision rather than an oversight. A window the size of the whole picture answers with a single number from one position and holds 65 parameters reading 64, which is a dense layer wearing this layer’s interface and is not wrong, only pointless. A stride of four with a window of three answers two by two and leaves 28 of the 64 cells outside every window.

A padding of four with a window of three answers fourteen by fourteen, of which 96 of the 196 positions read nothing but invented border and so answer with the bias and no evidence at all.

Two things a reader might look for are absent from the table, and both for the same reason. There is no unfitted state to guard and no feature names to match, because a layer holds its weights from the moment it is built and reads positions rather than named columns. Its whole contract is the arrangement, so almost every refusal it has is about an arrangement or about a number that cannot be a window.

The last block of the widget above offers three chains of layers to a stack. The first puts a layer that reads a row directly on top of the page’s convolution, which answers a map of one channel by six by six, and the layer above reads a row of thirty-six. Those hold the same count of numbers and are not the same thing, and the stack refuses to be built.

layer 0 answers with (1, 6, 6) and layer 1 reads (36,);
both hold 36 numbers, so it is the arrangement that disagrees
and not the width

The second half of that message is the part worth having. A check that compared counts alone would have let this join through, and a message that said only that a shape did not match would send a reader looking for a bug in the code rather than for the missing layer. The third chain is the twenty-eight by twenty-eight example joined the same wrong way, where the two sides are (8, 26, 26) and (5408,) and the message says the same thing about 5,408 numbers.

The repair is a flattening layer between them, which changes no number and only restates the arrangement, and the middle chain is that. It builds, holds three layers, reads (1, 8, 8) and answers a row of three. Every join in a stack is settled this way at the moment the stack is assembled, in integer comparisons, before any picture has been read.