A Neuron

An artificial neuron starts with three fundamental components: weights, a bias, and an activation function. We will work through what each contributes before putting them together to calculate an output.

15 sections · 15 questions · 4 coding challenges

1 / 15

The arrow keys move between sections too.

Could a Machine Learn Which Signals Matter?

BOTIE

Imagine building a machine that can recognise a simple pattern of light. Sensors can tell us which parts of an image are bright, but that does not tell the machine which combination means the pattern is present. We need a way for some signals to matter more than others, and a way to change their influence when the machine gets an answer wrong.

McCulloch and Pitts explored simplified mathematical neurons in 1943: units that respond when their combined inputs meet a condition. Rosenblatt's perceptron work in the 1950s investigated learning from examples. The Mark I Perceptron made adjustable connection strengths physical. Its hardware used motor-driven potentiometers, described in the machine's 1960 operator's manual.

A potentiometer is an adjustable electrical component, like the component behind a traditional volume knob. Used as a voltage divider, it changes how much of an input voltage reaches the next part of a circuit. The signal can stay the same while the knob changes its contribution. That gives us something concrete to picture when we talk about a weight. In the Mark I, motors could change these settings during learning.

For the artificial neuron we will use here, there are three fundamental components:

Weights. Each input has its own weight. Think of a separate control for each incoming signal, rather than one volume knob for the whole neuron. In software, we multiply the input by that weight. A negative weight makes the input oppose the combined score; that signed arithmetic goes beyond a passive volume knob on its own.
A bias. This is a separate adjustable constant added to the combined score. It lets us change how readily the neuron responds without changing the relative influence of all its inputs. With a threshold activation, changing the bias moves the point at which the neuron switches on.
An activation function. This turns the combined score into the output passed onward. An early threshold unit gave an on-or-off response. Other choices let the response vary continuously or suppress negative scores. We will examine those choices after the components themselves are clear.

Learning adjusts the weights and bias so that useful signals contribute appropriately. The activation determines how their combined evidence is expressed. This is a simplified computational model inspired by nervous systems, not a complete description of a biological neuron. We can now follow one small example and see exactly what each component changes.

Sources for this history

What Is Inside an Artificial Neuron?

A model needs a small component that can combine several measurements and decide what signal to pass onward. An artificial neuron fills that role.

Before combining anything, separate its three fundamental components: weights control input contributions, a bias supplies an offset, and an activation function converts the resulting score into an output.

Start with the measurements

Height and body mass are inputs in our example. They describe the person being read. The neuron's internal settings are separate from those measurements.

Observation

Height
Body mass

Neuron

Reads both measurements

Body mass is an input; model weights are coefficients.

1 of 5

Now each symbol can name a component we understand

The next calculation writes the weighted sum, bias, and activation separately. You can trace which component caused a change in the score or output.

The full neuron playground compares activation shapes after the basic roles are established.

Explore the full example

Change one weight, the bias, or the activation at a time. Compare the weighted score with the final output so you can see which operation changed it.

Show the crowd

w₁2.00w₂−1.00b0.50

Left, the neuron’s output over the plane of standardised height and weight, the dashed line where its score is zero, and the probe you can drag. Right, the chosen activation function in indigo and its slope in amber, marked at the probe’s score.

Score z at the probe

…

Output f(z)

…

Slope f′(z)

…

Part 1. One Person Through One Neuron

1. Two measurements in standard units

A neuron here reads two numbers about a person, their height and their weight, and it reads them after the feature scaling page’s standardisation, so that each is the number of standard deviations above the crowd’s mean. Done that way the two inputs share a scale, the two weights the neuron puts on them can be compared, and the whole crowd fits in a window three deviations each way, which is the square the box above draws.

The crowd’s mean height is 151.48 centimetres with a deviation of 15.61, and its mean weight 51.48 kilograms with a deviation of 14.45.

x₁ = (height − 151.48) / 15.61
x₂ = (weight − 51.48) / 14.45

Two people the page keeps coming back to

The first person in the crowd is a child of 147 centimetres and 41 kilograms, who standardises to (−0.287, −0.725), a little short and rather light. The page’s worked person is one deviation above the mean in both measurements, at (1, 1), which is 167.1 centimetres and 65.9 kilograms, and the page calls them the tall heavy person.

A second worked person at (−1, 1) is a deviation short and a deviation heavy, and the page calls them the short heavy person. Both are cells of the lattice the box draws, so every number they produce can be read off it.

Keep in mind

The neuron never sees centimetres or kilograms. It sees two standardised numbers, and everything it learns is written in those units, so a weight of 2 on height means two units of score per standard deviation of height rather than per centimetre.

2. The weighted sum and the bias

The neuron multiplies each input by its own weight, adds the two products, and adds a third number, the bias, which shifts the total without reference to either input. The result is called the score, and the three numbers the neuron holds, two weights and a bias, are everything it will ever learn. One wording trap is worth naming once. The neuron’s weights are the multipliers it holds, and a person’s weight is the second input, and the page keeps the two apart by calling the inputs x₁ and x₂.

z = w₁·x₁ + w₂·x₂ + b

The worked neuron on the tall heavy person

The worked neuron has weights two and minus one, with bias one half. For the person at (1, 1), multiply each input by its own weight and add the bias.

z = 2·1 + (−1)·1 + 0.5 = 1.5

For the person at (−1, 1), only the first input changes.

score = 2 × (−1) + (−1) × 1 + 0.5 = −2.5

Choosing a different activation leaves these scores unchanged. The activation is applied after the weighted sum and bias.

Keep in mind

The score is a weighted sum plus a constant, which is the regression pages’ linear model with the coefficients renamed. Nothing about it is new, and its three numbers are the entire learnable content of a neuron.

3. The score across the whole plane

The score is a weighted sum, so over the plane of the two inputs it is a tilted plane, rising in one direction and falling in the opposite one. In the box above the shaded square is that plane seen from above, indigo where the score is positive and amber where it is negative, darker the further from zero. Drag a weight slider and the tilt changes. Drag the bias and the whole plane rises or falls without tilting.

The worked neuron at the four corners

With weights 2 and −1 and a bias of 0.5, the corner three deviations short and three light scores −2.5, the corner three tall and three heavy scores 3.5, the corner three tall and three light scores 9.5, and the corner three short and three heavy scores −8.5. Height counts twice as much as weight and in the opposite direction, so the extreme scores belong to the tall-light and short-heavy corners rather than the diagonal ones.

Keep in mind

Along any line of constant score the shading does not change at all. The score varies only as you move across those lines, so one neuron knows exactly one direction in the plane, and how far a person lies along it.

4. Where the score is zero

Among all those lines of constant score one matters most, the one along which the score is exactly zero, drawn dashed in the box. Every activation function on the page’s list crosses its own middle there, the sigmoid at one half, the tangent at zero, the rectifier at the point where it starts to rise. Set the score to zero and solve for the second input.

2·x₁ − x₂ + 0.5 = 0
x₂ = 2·x₁ + 0.5

On the window the box draws, that line enters at (−1.75, −3) and leaves at (1.25, 3). Double the bias to 1 and it enters at (−2, −3) and leaves at (1, 3), the same slope shifted half a unit left. Set both weights to 1 and the bias to 0 and it runs corner to corner. Set both weights to 0 and there is no line at all, because the score is then the bias everywhere, 0.5 at every cell of the lattice, and the box draws nothing dashed.

Keep in mind

A weight turns the zero line and a bias slides it. Where it lies, and which side a person falls on, is the whole of what one neuron can say about the plane, and section 21 measures what that costs on the crowd.

5. The activation function, and the two numbers a neuron answers with

After the score comes one activation function, a function of a single number written f, and the neuron’s output is the activation function applied to the score. The activation function is chosen from a short list and is not learned. Press the four buttons in the box with the tall heavy person selected and the score stays at 1.5 every time while the output changes.

output = f(z)

identity   f(1.5) = 1.5
ReLU       f(1.5) = max(0, 1.5) = 1.5
sigmoid    f(1.5) = 1 / (1 + e^(−1.5)) = 0.8176
tanh       f(1.5) = tanh(1.5) = 0.9051

Conceptually, the neuron produces one output for the next layer. This SDK also returns the intermediate score so that a backward pass can evaluate the activation derivative at the same score. That saved value is part of the implementation, not a second signal sent to the next neuron.

Weights are kept in input order. In a hidden layer, those inputs are outputs from earlier neurons, so they need not correspond directly to named measurements such as height and weight.

Keep in mind

The score comes first and the activation function second, and the neuron keeps both numbers because the backward pass will ask for the score. The activation function is a choice made when the neuron is built, and the rest of this page is about what that choice does.

Part 2. Why Nonlinear Activations Matter

6. Two affine transformations reduce to one

The reason for the activation function is what happens without one. Hand the worked neuron’s output to a second neuron that has one weight and one bias and no nonlinear activation, and the second neuron computes a weighted sum of a weighted sum. Multiply it out and it is a single weighted sum with different numbers in it. The widget below builds exactly that chain and then asks the multiple-regression page’s plane fit whether the surface that comes out is a plane.

second reads   1.5 · (2·x₁ − x₂ + 0.5) − 0.25
            =  3·x₁ − 1.5·x₂ + 0.5

Activation Function between the two

second neuron’s weight1.50second neuron’s bias−0.25

collapsed weights, bias

…

plane fit weights, intercept

…

plane fit R²

…

largest residual from the plane

…

tall heavy, first neuron score, output

…

tall heavy, chain output, collapsed output

…

The chain’s output over the plane, with the tall heavy person marked. The collapsed neuron is the second weight times the first neuron’s three numbers, plus the second bias; the plane fit is the least-squares plane through every cell.

With the identity between them

The plane fit through the chain’s surface comes back with weights 3.00 and −1.50 and an intercept of 0.50, an R² of exactly 1.0 and a largest residual of exactly 0.0, which are the collapsed neuron’s numbers to the last bit. The tall heavy person scores 1.5 in the first neuron, and 2.0 comes out of the chain and out of the collapsed neuron alike.

Keep in mind

A chain using only identity activations still performs one affine transformation, however many neurons it contains. Its parameters can be combined into one set of weights and one bias. The extra stages have not expanded what it can represent.

7. A nonlinear activation changes what the chain can represent

Now press the sigmoid button in the widget. The first neuron’s score is still 1.5, its output is now 0.8176, and the second neuron reads that instead, so the chain answers 0.9764 for the tall heavy person where the collapsed neuron still says 2.0. Over the whole window the surface is no longer flat, and the plane fit says so.

activation function between	plane fit R²	largest residual	chain output at the tall heavy person
identity	1.0000	0.0000	2.0000
ReLU	0.8067	4.4978	2.0000
sigmoid	0.8852	0.5650	0.9764
tanh	0.8117	1.2629	1.1077

The least-squares plane through the chain’s surface under each activation function, with the second neuron at a weight of 1.5 and a bias of −0.25. The rectifier’s chain still answers 2.0 at the tall heavy person because 1.5 is on its live side; the residual comes from the half of the window it flattened.

›Why a chain without a nonlinear activation collapses

Write the first affine transformation with weights W₁ and bias b₁, and the second with W₂ and b₂. Expand the brackets to see that the composition has one combined weight matrix and one combined bias.

W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂)
combined weights = W₂W₁
combined bias = W₂b₁ + b₂

Inserting a nonlinear activation changes the calculation.

output = W₂f(W₁x + b₁) + b₂

In general, this cannot be reduced to one affine transformation because the nonlinear function cannot be distributed through the matrix multiplication. Particular parameters or restricted input regions can still produce an affine result. The nonlinear activation makes additional representations possible; it does not guarantee that every fitted network uses them.

Keep in mind

A nonlinear activation can make a chain express functions that a single affine transformation cannot. ReLU, sigmoid and hyperbolic tangent provide nonlinear responses; identity does not. Part 4 compares their derivatives to explain how the choice also affects learning.

8. Applying an activation to one score

Each of the four activation functions reads one number and answers one number, so applied to a whole row of neurons it treats every neuron’s score on its own, and the answer of one neuron depends on nothing another neuron did. That is why the diagram at the top of the box is complete with the activation function drawn inside the neuron rather than beside the layer, and why a neuron can carry its own activation function.

It is also why the score has to travel with the output, since a backward pass needs the activation function’s slope at that neuron’s own score and nowhere else.

Keep in mind

Each activation object in this library provides the function and its derivative. The function produces the output; the derivative is used when calculating gradients. The four functions here act on one score at a time. Section 22 compares them with softmax, which depends on several scores together.

Check yourself

Questions on Parts 1 and 2

Question 1 of 5

The tall heavy person scores 1.5 through the worked neuron. What do the four activation buttons change?

Part 3. The Four Activation Functions

9. The identity

The identity does nothing to the score, so its output is the score and its slope is 1 everywhere. The tall heavy person comes out at 1.5 and the short heavy person at −2.5, and nothing is squashed or clipped. It is on the list so that a network whose last layer predicts a quantity, a weight in kilograms say, can end in an ordinary neuron rather than a special case, since a sigmoid there would trap every answer between zero and one.

f(z) = z            f′(z) = 1

…

Keep in mind

Identity is useful when a layer should pass its score through unchanged. It often appears at an output layer that predicts an unrestricted quantity or supplies logits to a loss. It is valid elsewhere too, but consecutive affine layers without a nonlinear operation can be combined into one.

10. The rectifier

The rectifier keeps a positive score and replaces a negative one with zero, so it is the score on one side of zero and flat on the other. The tall heavy person, scoring 1.5, comes through untouched at 1.5. The short heavy person, scoring −2.5, comes out at exactly 0, and so does everyone else on that side of the dashed line, however far from it they are. Its output has no ceiling and a floor of zero.

f(z) = max(0, z)     f′(z) = 1 for z > 0,  0 for z < 0

At exactly zero, ReLU has no single derivative: the left and right slopes differ. This implementation uses zero as its backward convention there. Zero scores can occur, for example with zero inputs or parameters, so the convention should be stated rather than assumed irrelevant.

Keep in mind

The rectifier is two straight lines meeting at zero, which is why its slope is either exactly 1 or exactly 0 and never anything between, and section 18 counts what the zero half costs on the crowd.

11. The sigmoid

The sigmoid is the logistic regression page’s squash, and it maps any score into the open interval between zero and one, crossing one half at a score of zero. The tall heavy person’s 1.5 becomes 0.8176 and the short heavy person’s −2.5 becomes 0.0759. Neither end is ever reached exactly in principle, though in floating point it is, and section 18 says what happens then.

f(z) = 1 / (1 + e^(−z))

The formula as written overflows for a large negative score, since e to the 800 is beyond what a double can hold, and the sigmoid here is computed in a form that does not. I checked it at the edge of what the box can reach. A neuron with both weights at 10 and a bias of −10, shown the corner ten deviations short and ten light, scores −210 and answers 6.28 × 10⁻⁹², a tiny number rather than a warning, and the same neuron at the opposite corner scores 190 and answers exactly 1.

Keep in mind

A sigmoid output lies between zero and one and can represent a binary probability when the model and training objective give it that interpretation. Its bounded response can also be useful inside some architectures. Section 20 examines the probability interpretation; the following sections compare its learning behaviour with tanh.

12. The hyperbolic tangent

The tangent is the sigmoid recentred, stretched to run from minus one to one and crossing zero at a score of zero, so a neuron carrying it answers a number centred on zero rather than on one half, which the next layer tends to find easier to read. The tall heavy person comes out at 0.9051 and the short heavy person at −0.9866, already close to the floor at a score of only −2.5.

tanh(z) = 2·σ(2z) − 1

The recentring checked at the tall heavy person

To recover tanh at score 1.5, evaluate sigmoid at twice that score, double the result and subtract one.

tanh(1.5) = 2 × sigmoid(3) − 1 ≈ 0.9051

The two functions are related by changes to both the input scale and the output scale.

Keep in mind

Tanh is related to sigmoid by rescaling its input and recentering and rescaling its output. Both have small derivatives far from zero. Part 4 compares how quickly those derivatives shrink.

Part 4. How the Activation Changes the Gradient

13. Why the slope decides

When a network learns, a gradient arrives at each neuron from whatever sits above it, saying how the loss would change if that neuron’s output moved. To turn that into how the loss would change if the score moved, which is what the weights need, the calculus primer’s chain rule multiplies it by the activation function’s slope at the very score the forward pass produced. Every neuron a gradient passes through on its way down multiplies it by one such factor, and the backpropagation page follows one all the way.

∂loss/∂z = ∂loss/∂output · f′(z)

Keep in mind

An activation function is chosen for its slope, since the slope is what the gradient is multiplied by. An activation function whose slope is small everywhere starves everything beneath it, whatever its output looks like.

14. Deriving the four slopes

The identity’s slope is 1 and the rectifier’s is 1 on the live side and 0 on the dead side, both read straight off the definition. The sigmoid takes three lines of the chain rule, and its slope comes out as a function of its own output, which is why a backward pass never has to recompute the exponential.

step	why
σ(z) = (1 + e^(−z))^(−1)	the sigmoid written as a power, ready for the chain rule
σ′(z) = e^(−z) · (1 + e^(−z))^(−2)	the power rule on the outside, times the derivative of the inside, which is −e^(−z), and the two minus signs cancel
σ′(z) = σ(z) · (1 − σ(z))	e^(−z) / (1 + e^(−z)) is 1 − σ(z), and the remaining factor is σ(z)
tanh′(z) = 4·σ′(2z) = 1 − tanh²(z)	differentiate 2·σ(2z) − 1 and substitute the sigmoid’s slope back in

The slopes at the two worked people

At score 1.5, sigmoid and hyperbolic tangent both have relatively small derivatives. Calculate them from the corresponding outputs.

sigmoid output ≈ 0.8176
sigmoid derivative ≈ 0.8176 × (1 − 0.8176) ≈ 0.1491

tanh output ≈ 0.9051
tanh derivative ≈ 1 − 0.9051² ≈ 0.1807

At score minus 2.5, the sigmoid derivative is about 0.0701 and the tanh derivative about 0.0266. These small multipliers reduce the gradient passed back through the activation.

Keep in mind

Both squashing activation functions have slopes that are a function of their own output, and both slopes fall toward zero as the output nears either end of its range. The rectifier’s slope is a step, and the identity’s is a constant.

15. The four slopes side by side

Laid over one another the four slopes tell the whole story of which activation function to use where. The identity and the rectifier reach 1 and stay there, the tangent reaches 1 at a score of zero and falls away on both sides, and the sigmoid reaches a quarter at its best and falls away just as fast.

The factor a gradient keeps on its way back through one neuron, for each of the four activation functions. Where a curve runs along the floor, the neurons below stop learning.

Peak slope, Identity

…

Peak slope, ReLU

…

Peak slope, Sigmoid

…

Peak slope, tanh

…

Keep in mind

The largest derivative is one for identity, one on the positive side of ReLU, one quarter for sigmoid, and one for tanh. Sigmoid reaches its maximum derivative at output one half.

maximum sigmoid derivative = 0.5 × (1 − 0.5) = 0.25

Changing weights changes the score at which the activation is evaluated. It does not change this bound on the sigmoid’s own derivative.

Part 5. Saturation And Dead Units

16. The flat ends of the sigmoid and the tangent

An activation function that squashes has to go flat somewhere, and once it has, its slope is close to zero and it passes almost nothing back. That is saturation, and the table below reads the two squashing activation functions at whole-number scores off the curve the box samples.

score z	sigmoid output	sigmoid slope	tanh output	tanh slope
0	0.5000	0.2500	0.0000	1.0000
1	0.7311	0.1966	0.7616	0.4200
2	0.8808	0.1050	0.9640	0.0707
3	0.9526	0.0452	0.9951	0.0099
4	0.9820	0.0177	0.9993	0.0013
6	0.9975	0.0025	1.0000	0.000025

Outputs and slopes at whole-number scores, read from the sampled curves. The negative scores mirror these, with the outputs reflected and the slopes the same.

By a score of 6 the sigmoid’s slope is 0.0025, a hundredth of its peak, and the tangent’s is 0.000025, which is why the tangent looks so much steeper at zero and is no better off two units out. The page uses a hundredth of the peak as its working definition of saturated, and on the neuron the logistic model fitted to the crowd, whose weights are 2.55 and 1.29, the child of 120 centimetres and 25 kilograms scores −7.38 and has a sigmoid slope of 0.000622, the adult of 183 centimetres and 83 kilograms scores 8.12 and has the crowd’s smallest slope, 0.000298, and six of the twenty-five people are under the hundredth.

Keep in mind

Saturation is about the size of the score, and the score is a weighted sum, so a neuron saturates when its weights are large or its inputs are, which is one more reason the inputs arrive standardised.

17. Ten neurons in a chain

The trouble compounds. A gradient passing back through ten sigmoid neurons in a chain is multiplied by ten slopes, each at most a quarter, so in the best case, with every one of the ten scores exactly zero, it keeps a quarter to the tenth power.

0.25¹⁰ = 9.537 × 10⁻⁷
0.1491¹⁰ = 5.4 × 10⁻⁹

Under a millionth at the single best point, and the second line, at the tall heavy person’s slope of 0.1491, is two hundred times smaller than that. The tangent peaks at 1 rather than a quarter, so a chain of tangent neurons starves four times more slowly per layer and starves all the same once the scores drift from zero. The rectifier passes the gradient through at exactly 1 wherever the score is positive, however long the chain, and that is the whole reason it took over. Nothing is broken in a starved chain and nothing raises; the walk just stalls, which the training page shows happening.

Keep in mind

Ten sigmoid slopes multiply to under a millionth at their very best, and no learning rate recovers what the chain has multiplied away, since the rate scales the product it is given rather than restoring what the squashes removed.

18. The rectifier’s dead side

The rectifier escapes saturation on one side and pays for it on the other. Wherever the score is negative its slope is exactly 0, so a person on that side of the line contributes nothing to the gradient at all, and a neuron whose score is negative for every person it is shown receives no gradient from anyone and cannot move its weights to change that. The unit is dead, and since the only thing that could move its weights is the gradient it no longer receives, it stays that way.

The fitted neuron under the rectifier

Give the neuron the logistic model fitted to the crowd a rectifier instead of its sigmoid, and 12 of the 25 people land on the dead side with an output of exactly 0 and a slope of exactly 0. The other 13 pass through at a slope of exactly 1. Nothing is in between, because the rectifier has no in between.

A sigmoid can die too, in floating point. Scale the fitted neuron’s three numbers by ten and the child of 120 centimetres scores −73.8, whose sigmoid output rounds to a number so close to zero that one minus it is exactly 1, and the mirror case on the other side rounds the output to exactly 1, where the output times one minus the output is exactly 0.

Three people in the crowd have a slope of exactly 0.0 at that scale, which is saturation having become death, and I mention it because a reader who believes the sigmoid’s slope is always positive will meet a zero and go looking for a bug that is not there.

Keep in mind

A dead rectifier unit is a neuron no gradient reaches. The usual guards are a small positive bias at the start and a learning rate that cannot fling the weights across zero in one step, and the leaky rectifier, which gives the dead side a small slope, is the repair that changes the activation function itself.

19. A census of the crowd

The widget below starts from the fitted neuron and scales its two weights and its bias together, which leaves the dashed line where it is and steepens the score everywhere else. Each person is filled in proportion to the slope at their score, so as the scale rises the people far from the line fade to rings, and the counts say how many have gone.

…

scale	sigmoid saturated	sigmoid dead	tanh saturated	tanh dead
0.25	0 of 25	0	0 of 25	0
1	6 of 25	0	6 of 25	0
4	10 of 25	0	16 of 25	6
10	17 of 25	3	23 of 25	7
20	23 of 25	5	23 of 25	16

The census at five scales of the fitted neuron. Saturated is a slope under a hundredth of the activation function’s peak, dead is a slope of exactly zero, and the dead are counted among the saturated.

Keep in mind

At a quarter of the fitted scale nobody in the crowd is flat, and at twenty times it almost everyone is. The weights decide it, which is why a network’s starting weights are chosen small and its inputs are standardised before anything else is tried.

Part 6. One Neuron Against Logistic Regression

20. A sigmoid neuron is the logistic model

Fit the logistic regression page’s model to the crowd, with standardised height and weight as its two features, and it settles on a coefficient of 2.5537 for height, 1.2944 for weight and an intercept of 0.1396, after 7527 passes of the gradient climb. Hand those three numbers to one neuron as its two weights and its bias, choose the sigmoid, and show it the same people.

…

Both routes on the whole crowd

The child of 147 centimetres and 41 kilograms scores −1.5321, and the model’s probability of adulthood is 0.177694 while the neuron’s output is 0.177694. The adult of 156 centimetres and 53 kilograms scores 1.0151 and both routes answer 0.734019. Over all twenty-five people the largest gap between the two is 1.1 × 10⁻¹⁶, which is one rounding of a double, and the button in the box at the top of the page loads the same three numbers so you can drag the probe over the crowd yourself.

Keep in mind

A neuron with a sigmoid activation function and logistic regression are the same calculation, weights, a bias and a squash, and the logistic page’s single-input model is the same neuron with one weight. Nothing about the unit is new, and the dense layers page is where the difficulty starts, by feeding one such neuron’s output into another.

21. What one neuron cannot do

The dashed line in the twin’s map is straight, because a weighted sum is zero along a straight line for every choice of weights and bias, and the crowd’s tangled middle does not sort along any straight line. The fitted neuron calls 17 of the 25 people correctly, an accuracy of 0.68, and the eight it misses are the ones ringed dark in the map, all of them inside the tangle, where a child and an adult of nearly the same height and weight sit on opposite sides of whatever line is drawn.

Section 4 said one neuron knows one direction in the plane, and this is the price.

›Why no nonlinear activation rescues it

Write the score at the four corners of exclusive-or, where the answer is yes when exactly one of two inputs is on.

z(0,0) = b              z(1,0) = w₁ + b
z(0,1) = w₂ + b         z(1,1) = w₁ + w₂ + b

z(0,1) + z(1,0) = w₁ + w₂ + 2b = z(0,0) + z(1,1)

The two yes corners and the two no corners carry the same total score, for every weight and bias there is. Exclusive-or asks for both yes corners above a threshold and both no corners below it, which would make the left-hand total exceed twice the threshold while the right-hand total falls short of it, and those two totals are the same number.

The argument never touches the activation function, only the assumption that a higher score never means a lower output, which every activation function on the list satisfies, so no cleverer squash escapes it.

Composition does escape it, which is what the dense layers page builds and the training page trains on exclusive-or itself. Two neurons each draw a line, and a third reading both can say yes between the lines and no outside them.

Keep in mind

One neuron draws one straight line, and its ceiling is a fact about the unit rather than a weakness of any activation function. The crowd’s 17 of 25 is what that ceiling costs on data with a tangled middle.

22. Why softmax is not on the list

Anyone who has met the multi-class page will look for softmax among the four buttons and not find it, and its absence is a matter of what it reads rather than of taste. Each activation function on the list reads one score and answers one output. Softmax reads a whole row of scores at once and answers a row that sums to one.

softmax(z)ᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)

Three scores, one of them moved

Three neurons scoring 2, 1 and 0.1 come out of softmax at 0.6590, 0.2424 and 0.0986. Move only the third score, from 0.1 to 5, and the outputs become 0.0466, 0.0171 and 0.9362. Nothing about the first neuron changed and its answer fell by a factor of 14.1, because every output shares one denominator and the three compete for a fixed total of one.

No function of one number can do that, and no single neuron can hold it, since a neuron has no neighbours to normalise against. Softmax belongs to an output layer, where the whole row exists, and it is kept there beside the multi-class loss rather than on the list of activation functions a neuron may own.

Keep in mind

Softmax is an activation function for a vector of scores. It is outside this playground’s scalar activation interface because one output depends on the other scores too.

Part 7. Implementation And Failure Contracts

23. What a complete implementation states

A complete neuron states how many inputs it reads, which is its number of weights and cannot be zero; the order its weights match its inputs in, since they carry no names; that it holds a bias; which activation function it applies and what that activation function’s slope is, so that the two travel together; what the slope is at the rectifier’s kink; that the sigmoid is computed in a form that cannot overflow; the range each activation function’s output lies in; and that it answers with the score and the output both, since the backward pass needs the score. It also states what a chain of them does when only affine operations occur between them, which is collapse, and why softmax needs a vector-valued interface.

24. Failure contracts

Every row below was probed. The playground’s request refuses some of these at the door, with its own bounds, before the neuron is reached, and where that happens the row says what the neuron itself does when asked directly.

the edge	the behaviour
a neuron with no weights	refused; a neuron with no inputs is a constant wearing a neuron’s name, and the request asks for exactly two weights before any neuron is built.
a neuron with one weight	accepted, and it is the logistic page’s single-input model; the playground’s request asks for two because its plane has two axes.
a non-finite weight or bias	refused in words, and it cannot be written in a request at all, since JSON has no way to spell infinity or a non-number.
a row of the wrong length	refused; the weight count is the input width, and three values against two weights is named as a length mismatch.
an empty row, or a non-finite or non-numeric value in it	refused, each by the same guard every column passes through.
integers, or booleans, as inputs	accepted and coerced to floating point, so true reads as 1 and false as 0; documented rather than defended.
one person	answered; a neuron responds to a row and has no notion of a dataset, so one row is the ordinary case.
both weights zero, a constant column	accepted; the score is the bias everywhere, 0.5 at every cell, and there is no zero line to draw.
a weight or probe beyond 10, or a lattice finer than 41 cells	refused at the door as a request larger than the page allows, with the limit named.
unfitted use	nothing to refuse; a neuron is built complete from its three numbers and has no fit of its own. Fitting is the training page’s job.
mismatched feature names	not detectable; the weights carry no names and match inputs by position, so the order is the contract. Documented rather than defended.
softmax requested as an activation function	refused; it is not on the list and cannot be, since it reads a row.
the rectifier at a score of exactly zero	slope 0 by convention, the negative side’s answer; the curve’s middle sample reports it.
a score of −210 under the sigmoid	6.28 × 10⁻⁹², with no overflow and no warning, because the sigmoid is computed in its stable form.
a score of 190 under the sigmoid	exactly 1, with a slope of exactly 0, which is where saturation has become a dead unit in floating point.
a lattice window that does not open upward	refused in words, naming which input’s low edge is not below its high edge.

The two rows that deserve a second look are the ones that are accepted. Booleans coerce quietly, so a column of yes-and-no flags becomes ones and zeros without anyone saying so, and the positional weights mean a row handed over with its height and weight swapped is scored wrongly with nothing raised. Both are the price of a unit that reads unnamed coordinates from the layer beneath it, and the dense layers page is where the shape checks that catch a wrong width live.