A Dense Layer and the Forward Pass

Take one person from the crowd the classification pages keep returning to, written in standard units so that (1, 2) means one deviation taller than average and two deviations heavier. The neurons page hands that row to one neuron and gets one number back. Hand the same row to three neurons at once and three numbers come back, and that is the whole of what a layer adds.

Nothing about any neuron has changed. Each still holds one weight per input and a bias, forms its score, applies its activation function, and passes on the output.

…

neuron 1: 1 × 1 + 1 × 2 + 0 = 3
neuron 2: 1 × 1 + (−1) × 2 + 0 = −1
neuron 3: (−1) × 1 + 1 × 2 + 0 = 1

ReLU(3, −1, 1) = (3, 0, 1)

sigmoid(3, −1, 1) ≈ (0.9526, 0.2689, 0.7311)

Because every neuron in the layer is handed the same row, a neuron expecting a different number of inputs is not an odd member of the group, it is one that can never be fed. Give the layer a neuron holding two weights and another holding three and there is no row that satisfies both.

So the width a layer reads is a fact about the group, settled when the group is formed, and a group that cannot agree on it is refused at construction rather than discovered when a row arrives. I tried it, and the refusal reads “every neuron in a layer reads the same row, so they must agree on its width, got [2, 3]”.

a layer of m neurons over n inputs holds exactly n weights in every neuron

The neurons page already made the count of weights the input width, since a dot product with a row of another length is undefined. A layer inherits that and adds one rule on top, that all of its members hold the same count. That rule is the first link in a chain that ends with a whole network’s shape being decidable before any data exists, which Part 4 is about.

Three neurons answer with three numbers whether they read two inputs or two hundred, so the width a layer answers with is how many neurons it holds, and nothing about the row it reads bears on it. The two widths are separate choices. The first is forced by whatever feeds the layer; the second is free, and choosing it is what designing a network mostly consists of.

reads (n,)   answers (m,)        here (2,) and (3,)

Widening a layer changes what the next layer must expect and changes nothing about what this one accepts, which is why a network’s shape is a chain of separate equalities rather than one number carried the whole way through. On the playground the hidden layer reads (2,) and answers (3,), and the output layer reads (3,) and answers (1,), and the second pair is a consequence of the first only at its left end.

A dense layer has one weight for every input-output connection and one bias per output neuron. Count the two layers separately.

hidden layer = 2 inputs × 3 neurons + 3 biases = 9
output layer = 3 inputs × 1 neuron + 1 bias = 4
network total = 9 + 4 = 13

The playground reports the same total.

parameters in a layer = m · n + m

hidden  3 · 2 + 3 =  9
output  1 · 3 + 1 =  4
network              13

Thirteen numbers is small enough to write out, and the point of doing so is that nothing else exists. There is no stored answer anywhere in a layer, only these numbers and the two lines of arithmetic that Part 2 writes down. Change a cell in the playground and every node re-prints, because nothing from the last pass is kept that a new one could reuse.

one hidden layer:  (2 × 6 + 6) + (6 × 1 + 1) = 25
two hidden layers: (2 × 3 + 3) + (3 × 3 + 3) + (3 × 1 + 1) = 25

Each hidden neuron’s weights are a row of two numbers, and three such rows stacked one under another are a matrix with three rows and two columns. Nothing has been computed yet; the three neurons’ weights have only been written down together. But written that way, the three dot products the last section formed one at a time are one matrix multiply, the linear algebra primer’s, and the three biases are one vector added afterwards.

z = W x + b

W = [  1   1 ]      x = [ 1 ]      b = [ 0 ]      z = [  3 ]
    [  1  −1 ]          [ 2 ]          [ 0 ]          [ −1 ]
    [ −1   1 ]                         [ 0 ]          [  1 ]

…

z₂ = 1 × 1 + (−1) × 2 + 0 = −1

The shape of W is (neurons, inputs), and both the order and the reason for it are worth having straight, because the transposed convention is common and reads differently. A row of W is a neuron, so reading across a row gives one neuron’s weights in input order, and the number of rows is the width the layer answers with. A column of W is an input, so reading down a column gives every neuron’s weight on that one input, and the number of columns is the width the layer reads.

W is (m, n)         W x is (m, n)(n, 1) = (m, 1)

The multiply conforms exactly when the matrix’s column count equals the row’s length, which is section 2’s agreement about width said in the primer’s vocabulary. The hidden layer’s W is (3, 2) and reads a row of two; the output layer’s is (1, 3) and reads the three hidden outputs. Each neuron is kept as its own object, because the neuron is the unit that was learned about, and the matrix is assembled once when the layer is built, so a pass is one multiply per layer rather than a loop over neurons.

One person is one row. Three people are three rows stacked into a block, and the same matrix multiply handles all of them in one call, because multiplying a block by the transposed weight matrix forms every person’s dot product with every neuron at once. The block that comes out has one row per person still, and one column per neuron, so a layer keeps the people axis and replaces the other axis with its own width.

Z = X Wᵀ + b        (rows, n)(n, m) = (rows, m)

The transpose is bookkeeping and not a new operation. A matrix product pairs each row of its left factor with each column of its right factor. The people are the rows of X, and section 5 wrote the neurons as the rows of W, so W is turned on its side to put each neuron’s weights in a column. The entry in row i and column j of the product is then person i’s dot product with neuron j, the same score section 1 formed for one person and one neuron.

The bias vector is added to every row alike, because a neuron’s bias does not depend on which person it is reading. Nothing in one row of Z depends on any other row of X, so a block is not a new kind of input. It is several separate passes that happen to share one call.

person (−1, 0.5)

neuron 1: 1 × (−1) + 1 × 0.5 = −0.5          ReLU → 0
neuron 2: 1 × (−1) + (−1) × 0.5 = −1.5       ReLU → 0
neuron 3: (−1) × (−1) + 1 × 0.5 = 1.5        ReLU → 1.5

output:   1 × 0 + (−1) × 0 + 2 × 1.5 = 3

The third button sends a row three numbers wide. The request lets it through on purpose, so that the layer meets it, and the layer refuses it with “this layer reads (2,), got a block arranged (3,)”. That sentence is quoted word for word, and Part 4 says why it is phrased in terms of an arrangement rather than a count.

After the multiply, each neuron transforms only its own score, so the activation function is applied to every entry of z on its own and the result has z’s shape. That is what lets a whole layer share one activation function applied to one block in a single call, and it is why softmax is not among the four activation functions here, since softmax reads a whole row of scores to answer any one of them, which the neurons page sets out.

a = f(z), entry by entry

rectifier   (3, −1, 1) → (3, 0, 1)
sigmoid     (3, −1, 1) → (0.9526, 0.2689, 0.7311)
tangent     (3, −1, 1) → (0.9951, −0.7616, 0.7616)

Each neuron carries its own activation function, so nothing in the mathematics stops a layer from mixing them, and a layer whose first neuron is straight and whose second is a sigmoid is accepted; I checked, and it answers (3, 0.2689) at the row. A uniform layer is a fact about ordinary practice rather than a law, and it is the case a vectorised pass can exploit, since one activation function over a whole block is one call where a mixed layer needs one per column.

A layer asked about a row answers with more than the row it hands on. It keeps three blocks together, the block it read, the scores it formed, and the outputs it answered with, and it keeps them as one object so that the three cannot be separated or mispaired. The scores and the outputs share a shape, since the same entry in each describes the same neuron on the same person, and the inputs share the row count.

…

On the running row the hidden layer’s response holds inputs (1, 2), scores (3, −1, 1) and outputs (3, 0, 1), and the output layer’s holds inputs (3, 0, 1), score 5 and output 5. The response reports two facts about itself besides, how many rows went through and how many neurons answered, which are the two axes of its blocks.

The forward pass is section 5’s two lines applied once per layer, with one rule joining them. The first layer reads the person, every later layer reads what the layer beneath answered, and the last layer’s outputs are the network’s answer. Written as matrices, with a subscript for the layer, the whole of the running network is four lines.

z₁ = W₁ x + b₁        h = f₁(z₁)
z₂ = W₂ h + b₂        y = f₂(z₂)

output score = 1 × 3 + (−1) × 0 + 2 × 1 + 0 = 5
identity(5) = 5

sigmoid hidden outputs ≈ (0.9526, 0.2689, 0.7311)
final output ≈ 1 × 0.9526 − 0.2689 + 2 × 0.7311 ≈ 2.1457

One response is kept per layer and the whole list handed back, with the last layer’s outputs reachable directly so that an ordinary caller never indexes into the list at all. A deeper network is the same alternation continued, an affine map, an activation function, an affine map, an activation function, for as many layers as were stacked.

A forward pass on its own would need only the outputs. The scores are kept because training needs the slope of each activation function at the point where the activation function was evaluated, and that point is the score, not the output. Under the rectifier the output 0 of the second hidden neuron could have come from a score of −1 or −100, and its slope is zero either way, but under the sigmoid the slope at a score of −1 is one number and at −100 another, and from an output alone the score would have to be recovered by running the layer again.

The inputs are kept for the same reason, since the correction owed to a layer’s weights multiplies by what the layer read, and the forward pass is the only thing that knows it.

kept per layer:   inputs, scores, outputs
read forward:     outputs only
read backward:    all three

The backpropagation page is where those three blocks are spent, walking the same row back down this same network, and it is not repeated here. What belongs here is only the design decision that makes it a plain reversed loop rather than a reconstruction of what each layer once read. An earlier version of this design kept only the scores and outputs, and the backward walk had to rebuild the list of inputs by shifting the outputs down one and pushing the caller’s row on the front, which was rebuilding something the forward pass had known and thrown away.

A dense layer does the same arithmetic whether the network is learning or answering, and so do a convolution and a pooling window. Two layers do not. A dropout layer damages its own answers on purpose while learning, and a batch normalisation layer standardises by the batch in front of it while learning and by remembered statistics while answering.

So every pass here states its purpose, predicting or training, and hands it to every layer whether or not the layer reads it, so that a layer which starts caring later does not change the interface for the rest.

purpose ∈ {predicting, training}        the default is predicting

The default is predicting, and the direction is chosen deliberately. Forgetting to say training costs a slightly slower descent; forgetting to say predicting would make every answer the model gives a matter of luck. The playground says its passes ran for predicting, which it states out loud even though it is the default, and the dropout page is where the other value earns its keep.

The definition of a layer is a loop, ask each neuron in turn, and the matrix multiply is that loop written as one call. Both are kept, the multiply for every fit and the loop as an observed route for anyone who wants to see one neuron’s arithmetic, and a test asserts that they agree, because a fast path and a slow path with nothing between them are two implementations rather than one calculation read two ways.

They agree to within floating point and not to the last bit, since the multiply sums the products in a different order than a loop does.

A dense layer knows two things about itself before any data arrives, the width it reads and the width it answers with, and each is recorded as a tuple of extents rather than as a number. For a dense layer the tuple has one entry, (2,) or (3,), and the distinction looks pedantic. It stops being pedantic the moment a layer answers with something arranged, a convolution handing on a stack of pictures, say.

A dense layer reading a row that holds exactly as many numbers still cannot read it, because a row and a stack of pictures are different arrangements of the same count, and a check that compared counts would wave that mismatch through as an agreement.

dense        reads (2,)          answers (3,)
convolution  reads (1, 28, 28)   answers (8, 26, 26)

8 · 26 · 26 = 5408 = (5408,) as a count, and not as an arrangement

A count says how many numbers there are and a tuple says how they are arranged, and the shape guarantee page is where that difference bites, with a flatten layer as the bridge. On this page every tuple has one entry, and the refusals still print the tuples, because a refusal that printed “5408 and 5408” on the case the check exists to catch would send a reader looking for a bug in the check rather than for the missing layer.

Two layers fit together when the one above reads exactly what the one beneath answers, and that is a sentence about two shapes, so it belongs to the shape rather than to whichever loop is assembling a network. The two boxes below state their four widths and nothing else. Set them, and the verdict comes back with no row sent, in the same words a real mismatch would produce hours into a training run if nobody had checked first.

above follows beneath   ⇔   beneath.answers == above.reads

Only the seam is checked. The left box’s reads and the right box’s answers are the network’s two ends, which the data and the task decide, and a layer reading eight and answering five joins one reading five and answering eight without complaint, since the seam between them holds.

An answer nobody is obliged to read is not a guarantee. A caller can ask whether two layers join, discard the answer, and discover the disagreement inside a matrix multiply later. So the check is not left to the caller. A stack of layers refuses to exist unless every seam holds, and a stack that has been built is one whose shape is already known to be sound, so no forward pass anywhere re-establishes it.

The check is a walk down a list of integers, two comparisons for three layers, and it costs nothing that could be measured.

parameters = (2 × 3 + 3) + (3 × 4 + 4) + (4 × 1 + 1) = 9 + 16 + 5 = 30

seam 0 to 1:  answers (3,)  reads (3,)   holds
seam 1 to 2:  answers (4,)  reads (4,)   holds

with the last layer reading 5
seam 1 to 2:  answers (4,)  reads (5,)   refused

A stack that exists has a shape of its own, the first layer’s reads paired with the last layer’s answers, and the playground prints it as (2,) → (1,). Everything between those two ends is internal, already agreed, and nobody else’s business. The data supplies two facts. The first layer’s width has to match how many measurements a person carries, and the last layer’s has to match what the task wants back, one number for a regression or a yes-or-no decision, one per class otherwise.

stack reads (2,)  ←  two measurements per person
stack answers (1,)  →  one number per person

The first of those is checked by the first layer itself the moment a block arrives, which is the refusal section 7 provoked with a row of width three. Nothing downstream checks anything, because every interior seam was settled when the stack was built. And a mismatch the widths cannot see stays invisible, since a row handed over with height and weight swapped has the right width and is scored wrongly with nothing raised.

I sent (2, 1) through the running network and it answered 2 where (1, 2) answers 5. The weights carry no names and match inputs by position, so the order is the contract, and this is documented rather than defended.

Before asking what a second layer buys, ask what it buys when only affine operations occur between the two. Set both activation functions to the identity and the four lines of section 10 substitute into one, a single affine map from the person straight to the answer with matrix W₂ W₁ and bias W₂ b₁ + b₂.

y = W₂ (W₁ x + b₁) + b₂ = (W₂ W₁) x + (W₂ b₁ + b₂)

combined weights = (1 − 1 − 2, 1 + 1 + 2) = (−2, 4)
output at (1, 2) = (−2) × 1 + 4 × 2 = 6

With identity activations, both layers together perform one affine transformation from two inputs to one output. They use thirteen parameters to express something that needs only three. Adding more affine layers does not change that limitation. A nonlinear activation between layers allows the network to represent functions outside that class.

A ReLU neuron has two regimes: it outputs zero when its score is negative, and it outputs its score when the score is positive. With two inputs, the change between regimes happens along a straight line where the score is zero.

The output layer combines several such responses. The resulting score is affine within each region of the input plane, with its slope able to change where a hidden neuron switches regime. Its zero contour can therefore contain multiple straight segments. This is what a piecewise-linear decision boundary means.

y = Σⱼ vⱼ · max(0, wⱼ · x + bⱼ) + c

unit j is flat where wⱼ · x + bⱼ < 0 and tilted where it is not

A sufficiently wide network with suitable nonlinear activations can approximate continuous functions on a bounded input domain. That existence result does not tell us which widths will train successfully on this particular dataset. I compared six widths and six starting seeds, using fifteen hundred epochs at a fixed learning rate. A fitted logistic classifier provides a reference.

Training thirty-six networks…

The best one-unit run classifies twenty-one of the twenty-five people correctly. A single ReLU followed by an affine output and a threshold still gives a half-plane decision region, or a constant prediction. Its higher accuracy than this fitted logistic reference does not demonstrate a more complex boundary; the training objectives and resulting fitted boundaries can differ.

One of the six starts leaves that unit inactive for every person and reaches only 0.52 accuracy. Two units reach 0.84 from every start. Three classify everyone correctly from one start; four do so from two starts, six from four and eight from all six. In these runs, extra width improves the reliability of finding a successful fit.

Section 4 showed that one hidden layer of six neurons and two hidden layers of three have the same twenty-five parameters. The deeper network’s second layer reads features already transformed by the first. Its activation regions, viewed in the original input coordinates, need not be separated by single straight lines.

This can make depth useful for representing some functions compactly. It does not guarantee easier training or better accuracy at a fixed parameter count. On this crowd, the deeper arrangement was less reliable.

Training eighteen networks…

The wide model classifies everyone correctly from four starts, while the deeper model does so from two. In the deeper model’s worst run, two of its three second-layer neurons are inactive for the whole crowd and accuracy is 0.76.

When the deeper model succeeds, its best log loss is 0.0045, compared with 0.0388 for the wide model. Both predict the correct classes, but the deeper run assigns them higher probabilities. Two layers of six succeed from every tested start, with losses below 0.004. These outcomes describe this training experiment, not a universal advantage of width or depth.

The universal approximation results promise that a network exists. Every number in the two tables above is about whether this loop, from this start, found it, and the gap between the two is the honest content of the section. Three units suffice for this crowd, since seed 2 proves it, and five of the six starts at three units did not get there.

One of the six starts at eight units ended with a unit dead for everyone, contributing a constant, and it still called every person correctly with the seven that were left, at a loss of 0.0813 against the 0.0347 of the best start.

a network exists        ≠        this walk from this start arrives

The dead units are the rectifier’s own hazard, the neurons page’s dead side met in a layer. A unit whose score is at or below zero for every person answers zero to all of them, its slope is zero everywhere it is evaluated, and nothing on any row can tell it what to do, so it stays where it is for the rest of the run. Whether it dies depends on the start, which is why the count is reported per seed. Everything after this page that is not a layer, momentum, better initialisation, normalisation between layers, is machinery for making the walk arrive somewhere good more often, and the training page runs the loop itself.

A dense layer's implementation must make its arrangements explicit before a forward pass. The reader needs to know which coordinate every weight multiplies and where every result goes.

1. State the input and output shapes. Every neuron must have one weight per input coordinate.
2. Fix the neuron order and the matrix convention. Here the weight matrix has neurons as rows and inputs as columns.
3. Keep the batch axis separate from each example's shape. A response retains the inputs, scores, and activated outputs from the same pass.
4. Specify the activation rule. This implementation can use different activations for different neurons and does not change its behavior with pass purpose.

A layer stack checks the shape at every join when it is constructed. If a join is invalid, the error identifies the adjacent layers and their arrangements. Its overall shape then runs from the first layer's input to the last layer's output.

Every row below was probed. The playground’s request refuses some of these at the door, with its own bounds, before the layer is reached, and where that happens the row says what the layer itself does when asked directly.

Two rows deserve a second look. The overflow row is a decision rather than an omission, since a layer’s response is built once per layer per call, and scanning every block on the way out as well as on the way in would double the check for a value that finite inputs and finite weights can still produce, so it is caught one seam later, where the next layer scans what it is given.

And the swapped-columns row is the price of a unit that reads unnamed coordinates from the layer beneath it. Past the first layer there are no names to check against, so the first layer cannot have them either, and the order a caller assembles a row in is the whole of the agreement.