Loss Functions

Take the four people the PCA page measured, 180, 160, 175 and 165 centimetres, weighing 78, 58, 63 and 73 kilograms, and draw a line through their mean point with a slope of 0.6 kilograms per centimetre. The line guesses a weight for each of them, 74, 62, 71 and 65, and each guess misses. The question a loss answers is how much those four misses should cost, as one number, so that a better line is one with a smaller number and the search for it can begin.

A miss on its own has a sign, and the sign is the wrong thing to add up, since minus four and four would cancel to nothing while both guesses were wrong. So every loss here first turns a miss into something that cannot be negative, and the five differ in how they do it.

Squaring, taking the size, squaring up close and taking the size far out, and for the two losses that price a probability rather than a weight, taking the negative logarithm of the probability the model gave to what was true.

A network is scored on many people at once, and the loss has to say what the whole batch costs. Every loss on this page does the same two things with the individual prices, adds them up and divides by the number of rows. The division is what keeps a batch of forty comparable with a batch of four, since without it the loss would double whenever the batch did, and so would the slope the network steps against.

loss  =  ( price of miss 1 + price of miss 2 + … + price of miss n ) / n

original mean = (8 + 8 + 32 + 32) / 4 = 80/4 = 20
duplicated mean = (2 × 80) / 8 = 20

The box at the top of the page has one slider, the raw output of a network’s last layer for one person, and it draws two charts. The upper one is what each loss charges at every raw output, and the lower one is the slope of that charge, which is the thing a network actually uses, because the slope says which way to move the output and how hard.

The value is the number we plot as a training curve and quote when a run is over, and the slope is the number the next step is taken from, so a loss that reported only its value would give a network nothing to do with it.

Both come back from one call, since computing either alone would mean running the squash twice, and the table under the charts checks that they agree with each other. The last column nudges the raw output by a hundred-thousandth either side, measures the cost at both, and divides the difference by the gap, which is the calculus primer’s definition of a slope.

At the halfway guess the largest disagreement between the slope reported alongside the value and the slope found by nudging is 2.3 parts in a trillion, and across the whole slider it never passes 2.1 parts in ten billion.

Square the miss, halve it, and that is the price. The half is bookkeeping and nothing more; it is there so the slope comes out as the miss itself rather than twice the miss, and a library that leaves it out reports values twice as large and gradients twice as steep, which the learning rate then has to absorb. Writing p for the guess and y for the truth, the price and its slope are these.

price  =  (p − y)² / 2
slope  =  p − y

individual costs = ½ × (16, 16, 64, 64) = (8, 8, 32, 32)
mean loss = (8 + 8 + 32 + 32) / 4 = 20
output gradients = (−4, 4, 8, −8) / 4 = (−1, 1, 2, −2)
large-error share of cost = 64 / 80 = 0.8

That growth is the whole character of squared error. On the playground the bowl steepens without limit, so a raw output of 3 against a target of one half costs 3.125 and pulls by 2.5, and a raw output of −5 costs 15.125 and pulls by −5.5. Nothing caps it, and section 17 is what that does when one of the misses is not a miss at all.

Take the size of the miss and stop. The price grows in a straight line, and its slope is only ever plus one, minus one, or nothing at all, because the size of a miss changes at the same rate however large the miss already is.

price  =  |p − y|
slope  =  sign(p − y)

mean absolute loss = (4 + 4 + 8 + 8) / 4 = 6
output gradients = (−1, 1, 1, −1) / 4
                 = (−0.25, 0.25, 0.25, −0.25)

The corner at zero is the price of that. Where the guess equals the truth no slope exists, and the slope reported there is zero, which on the playground is the one point where the amber dot on the gradient chart is neither at 1 nor at −1. It also means the slope never shrinks as the guess gets close, so a walk under absolute error keeps stepping at full size right up to the answer and then past it, which section 19 measures.

Huber takes a threshold, written d here, and is squared error inside it and absolute error outside it, with the outer piece shifted so the two meet without a jump. Small misses are corrected smoothly and the corner at zero is gone, while a miss beyond the knee pulls by exactly d and no more.

|p − y| ≤ d      price  =  (p − y)² / 2          slope  =  p − y
|p − y| > d      price  =  d · (|p − y| − d/2)    slope  =  d · sign(p − y)

cost at magnitude 4 = ½ × 4² = 8
cost at magnitude 8 = 5 × (8 − 5/2) = 27.5
mean loss = (8 + 8 + 27.5 + 27.5) / 4 = 17.75
output gradients = (−4, 4, 5, −5) / 4
                 = (−1, 1, 1.25, −1.25)

The d/2 in the outer piece looks like decoration and is not. At a miss of exactly d the inner piece gives d squared over two and the outer gives d times d less d over two, the same number, and the slopes are d on both sides. Drop the d/2 and the price jumps at the knee, which leaves a step an optimiser can stop in. On the playground the knee is at 1 and the worked check in section 3 lands on it.

The three rules are easiest to compare on one batch at once. The widget below draws a line of your choosing through the people, scores every miss under all three losses, and shows each person’s own cost as a bar and their share of the batch’s pull in the table. Start with the measured four and the slope-0.6 line, then tilt the line and watch which bars grow fastest.

On the four, the two eight-kilogram misses carry 80 percent of the cost under squared error, 67 percent under absolute error and 77 percent under Huber, and their share of the pull is 67, 50 and 56 percent. Squared error is the only one of the three whose pull share keeps growing with the miss, and the bars make the difference visible before any of the arithmetic is read.

squared error    (32 + 32) / 80        = 0.80
absolute error   (8 + 8) / 24          ≈ 0.67
Huber, knee 5    (27.5 + 27.5) / 71    ≈ 0.77

squared error    (2 + 2) / (1 + 1 + 2 + 2)              ≈ 0.67
absolute error   (0.25 + 0.25) / (4 × 0.25)             = 0.50
Huber, knee 5    (1.25 + 1.25) / (1 + 1 + 1.25 + 1.25)  ≈ 0.56

When the question is whether someone is an adult rather than how much they weigh, the last layer still hands over a plain number, a score that can be anything from far below zero to far above it, and a score cannot be compared to a yes or a no. The logistic page’s answer was to push the score through the sigmoid, which turns any number into a probability between zero and one, and that squash is the first thing both classification losses do, so the last layer stays straight and the activation function happens inside the loss, for a reason section 14 measures.

p  =  σ(z)  =  1 / (1 + e^(−z))

On the playground the raw output of zero squashes to exactly one half, which is the network saying it cannot tell, and that is why the page’s halfway guess is a raw output of zero rather than a probability of zero. A raw output of 3 squashes to 0.9526 and of −3 to 0.0474.

Once the score is a probability of yes, the price is the negative logarithm of the probability the model gave to whichever answer was actually true. A model that gave the truth a probability of one pays nothing, one that gave it a half pays the logarithm of two, and one that gave it almost nothing pays without limit. Writing y for the label, one for yes and zero for no, the two cases fold into one line.

price  =  −y · log(p) − (1 − y) · log(1 − p)
slope at the raw output  =  p − y

probability = sigmoid(0) = 0.5
loss = −ln(0.5) ≈ 0.6931
score gradient = 0.5 − 1 = −0.5

Two shapes are worth noticing on the playground. On the side where the squash agrees with the label the curve is flat and cheap, and on the side where it does not the curve rises like a straight line of slope one, since the logarithm of a sigmoid far out is the raw output itself. And the pull never exceeds one in size however wrong the model is, because a probability minus a label cannot.

A sigmoid pushed far enough saturates to exactly zero or exactly one in double precision, and the logarithm of exactly zero is minus infinity, which would poison the average for a row that was merely very wrong. So the probability is clipped away from both ends before the logarithm is taken, by one machine epsilon, and the price stops growing at the negative logarithm of that, which is 36.0437.

Machine epsilon is the gap between one and the next number double precision can represent. A probability clipped to sit at least that far from zero and from one always has a finite logarithm, and the largest price the clip allows is the logarithm of that gap with its sign changed.

machine epsilon   ε ≈ 2.22 × 10⁻¹⁶
ceiling           −ln(ε) ≈ 36.0437

With three classes, child, teenager and adult, the last layer hands over three scores per person, and the squash has to read across the row rather than down a column. The softmax raises e to each score and divides by the sum, so the three come out positive and summing to one, and the price is again the negative logarithm of the probability on the true class.

The slope at every score is the probability of that class minus the truth for that class, which is one on the true class and zero on the rest.

pₖ  =  e^(zₖ) / Σⱼ e^(zⱼ)
price  =  −log(p on the true class)
slope at zₖ  =  pₖ − yₖ

Tick the box that adds a hundred to every score and nothing changes, the same probabilities, the same price, the same pulls to the last digit. The softmax only ever reads the differences between the scores, which is why the row maximum is subtracted before e is raised to anything; written literally, e to the hundred-and-three overflows nothing but e to a thousand does.

On the playground the rose and sky curves lie on top of one another, which is why the sky one is dashed. That is an identity rather than a drawing choice. With two classes, and the second class’s score held at zero, the softmax probability of the first class is e to the score over e to the score plus one, which is the sigmoid of the score.

The page’s softmax reading is built exactly that way, the raw output in the first column and a zero in the second, and the two losses agree on the value, the gradient and the probability at every raw output on the slider.

e^z / (e^z + e^0)  =  1 / (1 + e^(−z))  =  σ(z)

Now look at the gradient chart with the slider at the halfway guess. Four of the five dots are at exactly the same value, −0.5. One of the four is Huber, which sits there only because a miss of one half is inside its knee of 1, where it is squared error under another name. The other three are the fact the page exists to teach. Squared error with a plain output, binary cross-entropy with a sigmoid, and softmax cross-entropy with a softmax all have the same slope at the raw output, the prediction minus the truth over the row count, and it is the simplest expression on the page.

d loss / d raw output  =  (prediction − truth) / n

For squared error the prediction is the raw output itself, zero against a target of one half. For the other two it is the probability the squash produced, one half against a label of one. Both misses are minus one half, and the halfway guess was built so that they would be. Drag the slider to 3 and the dots part, 2.5 against −0.0474, because the prediction now means different things, a number that is two and a half too high and a probability that is nearly right.

What the three share is the form, and the numbers agree only at a raw output where a miss in weight and a miss in probability happen to be the same size.

Tick the playground’s last box and a grey dashed curve appears, a sigmoid output scored by squared error, which is the pairing a reader might reach for first, since it is a probability compared to a label by the loss that has been on every page. It is computed here through a one-neuron layer that applies the sigmoid, so the number is measured rather than argued. At the halfway guess it costs 0.125 and pulls by −0.125, a quarter of what cross-entropy pulls.

The binary and multiclass cross-entropy implementations used on this site expect raw scores, also called logits. They apply sigmoid or softmax internally, so the preceding output layer uses identity.

Other APIs accept probabilities instead. A probability-based loss must receive the appropriate normalized probabilities. Check the loss interface: adding sigmoid before a loss that already expects logits applies it twice and changes the model being optimized.

The coincidence has a mechanism. Each squash is the one whose own derivative cancels the loss’s, leaving the subtraction behind, and that is what Nelder and Wedderburn called a canonical link. Binary cross-entropy is where the cancellation is easiest to watch. With p the sigmoid of the raw output z and y the label, differentiate the price with respect to p, then multiply by the sigmoid’s own slope.

Absolute error and Huber are on the page to break the pattern on purpose. Their slopes are a sign and a capped miss, neither of them prediction minus truth, and that difference is the whole reason to use them, which Part 5 measures.

The pairing follows from the question, so it can be written as a table, with the last layer straight in every row and the activation function, where there is one, inside the loss.

Take the fifteen people of the line page’s ideal case, who lie close to one line, and suppose the ninth of them, 176 centimetres and 72.5 kilograms, was entered with the digits swapped as 27.5. Nothing about the person changed, and one number in the table did. Score all sixteen under the line that fits the clean fifteen, slope 0.7968 and intercept −67.44, and the mistyped row misses by 45.29 kilograms where nobody else misses by more than a kilogram or so.

Under squared error, the corrupted row accounts for 99.87 percent of the loss and 89 percent of the summed gradient magnitudes. Under absolute error, it receives the same output-gradient magnitude as every other row. Huber caps it at the threshold divided by the batch size.

absolute-error gradient magnitude = 1/16 = 0.0625
Huber maximum gradient magnitude = 5/16 = 0.3125

These are gradients with respect to individual predictions. Parameter gradients also include the inputs through which those predictions were made.

Now let each loss fit its own line. The widget walks one linear neuron down each loss for four hundred steps from a flat line at the mean weight, on the same people, and draws the three lines it arrives at. Tick the switch to swap in the mistyped weight and watch which lines move.

On the clean fifteen all three lines nearly coincide. Squared error lands on the closed-form least squares line to within 1.1 parts in ten trillion, Huber lands on the same line because every miss is inside its knee, and absolute error settles a little away, slope 0.7993 against 0.7968, since the median line and the mean line are not quite the same line. Each guesses about 72.79 kilograms for the person at 176 centimetres.

With the weight mistyped, the squared-error line drops to a slope of 0.7464 and an intercept of −61.55 and now guesses 69.82 kilograms at 176 centimetres, three kilograms lower than before, for a person who has not changed. The Huber line moves to 0.7908 and −66.75, guessing 72.44, a third of a kilogram lower. The absolute error line moves to 0.7983 and −67.72, guessing 72.78, one hundredth of a kilogram lower.

The mistyped row is missed by 42.3, 44.9 and 45.3 kilograms by the three lines in turn, so the two robust rules leave it almost exactly as wrong as it was and squared error closes three kilograms of the gap at everyone else’s expense.

What the word robust means, when a textbook says it, is that no single row can pull harder than a fixed amount. Absolute error caps the pull at one over the row count and Huber at the knee over the row count, and squared error caps it nowhere. The cost is not what is capped; the mistyped row is most of the cost under all three rules, and it is only under the two robust ones that being most of the cost does not make it most of the say.

The same cap has a price at the other end of the walk. The readouts show the largest slope left in the neuron after the last step, and for squared error and Huber it is 8 parts in a quadrillion, which is a walk that has stopped. For absolute error it is 0.0926 on the clean fifteen and 0.125 with the outlier, because a sign never shrinks as the guess gets close, so the walk keeps stepping by the full learning rate and settles into a twitch about the answer rather than onto it.

The clean absolute-error loss is 0.3609 after four hundred steps and still moving in the third decimal.

Cross-entropy can also be sensitive to incorrect labels. Take sixteen people: eight children below 152 centimetres and eight adults above it. A fixed rule assigns a score from height, then a sigmoid probability.

score = (height − 152) / 5
P(adult) = sigmoid(score)

The rule classifies all sixteen correctly and has mean loss about 0.0854. The two people nearest the boundary, at 148 and 156 centimetres, each cost about 0.3711. Now change the 178-centimetre adult’s label to child.

The model still assigns that person about 0.9945 probability of adulthood, but the changed label now penalizes that confidence. Calculate the loss using the full score rather than rounded probabilities.

score = (178 − 152) / 5 = 5.2
incorrect-label loss = −ln(1 − sigmoid(5.2)) ≈ 5.2055
score gradient contribution = sigmoid(5.2) / 16 ≈ 0.0622
accuracy = 15 / 16 = 0.9375

The mean loss rises to about 0.4104, with this row contributing seventy-nine percent of the total. Accuracy records one wrong label; cross-entropy additionally measures how confidently the model disagrees with it.

A classifier is judged by how many people it called correctly, and it is trained by the loss, and the two can disagree. Change the scale of the rule in the widget above from 5 to 2.5 and then to 10. The accuracy is 1.0 at all three, since the probabilities cross one half at the same height whatever the scale, and the loss is 0.0290, 0.0854 and 0.1910, a factor of six apart.

The loss rewards confidence on people already called correctly, which accuracy cannot see, and the training page’s run has stretches where the loss falls for fifty epochs while the accuracy does not move at all.

Every loss on this page fails somewhere, and the failures are measured above rather than listed from memory.

A complete implementation states what it divides by, the rows or every entry or nothing; whether squared error carries the half; the units the Huber knee is measured in and what happens exactly at the knee; which branch absolute error takes at a tie; whether the classification losses apply their own squash, so that a caller knows to leave the last layer straight; how the probabilities are clipped before the logarithm and at what floor; what shape the truth takes, a column of zeros and ones or a one-hot block; and that the value and the gradient are produced by one call, so a caller cannot get one without the other.

Each row below was tried rather than reasoned about, and the page’s own request refuses at the door the cases the arithmetic lets through.

The empty batch is the one worth dwelling on, because it is the case where the documented contract and the actual behaviour disagree. The contract says an empty block is refused, and the arithmetic divides a sum of nothing by zero rows, which numpy answers with not-a-number and a warning rather than an exception. It is documented here rather than defended, the request refuses the case at the door, and nothing behind this page was changed to serve it.