More Than Two Classes

The logistic page had one observation, one positive class, one negative class, and one estimated probability. The probability of the negative class was one minus that number, because the two outcomes were the only two there were.

P(class 0 | x) + P(class 1 | x) = 1

The crowd in the box above has three colours, children, teenagers and adults. Every person is exactly one of the three, nobody is two at once, and the three cover everyone in the dataset. For one person the model should now say something like

One unit of probability, divided among mutually exclusive classes. That is what a multiclass probability model produces, and increasing one class’s share has to come out of the others.

Now a different question about the same people. Do they enjoy sports, do they enjoy music, do they enjoy reading. A person may enjoy none, one, several or all three, and the three probabilities do not need to total anything, because each answers a separate question.

Multiclass classification chooses among alternatives, and its output is one distribution. Multilabel classification answers several independent yes-or-no questions, and its output is several separate probabilities with their own thresholds. The rest of this page is about the first problem. The second appears again in section 24, because one of the two routes to the first problem is built from the tools of the second, and that is where the two get confused.

Binary logistic regression computed one linear score. With K classes, give every class a score of its own from the same inputs.

z_k = w_k · x + b_k

z_child = w_child · x + b_child
z_teen  = w_teen · x + b_teen
z_adult = w_adult · x + b_adult

Each class has its own weights and its own intercept. All three scores are computed from the same height and weight. They are unrestricted numbers, and they are not yet probabilities. What a multiclass linear model does is give every class its own opinion of the same observation, and the rest is deciding what to do with three opinions.

With two input features each class score is a flat plane over the feature plane, and the predicted class at any point is whichever plane is highest there.

…

Two classes tie where their planes cross, and two planes cross along a straight line.

z_j = z_k
(w_j − w_k) · x + (b_j − b_k) = 0

So the boundary between any two classes is linear, and the map on the right is feature space divided by which class has the largest score. Everything in the next two Parts is about what to do with the three heights of the planes at a point, and none of it moves the boundaries the argmax already drew.

Three unrestricted scores need to become probabilities that stay between zero and one, sum to one, keep whichever score was largest as the largest probability, rise when their own score rises, and fall when another class’s score rises. Those are the requirements. Softmax is the function that meets them, and it is easier to trust once the requirements are on the table first.

For each class, take the score, exponentiate it so the result is positive, add up all the exponentiated scores, and divide each one by that shared total.

P(y = k | x) = e^(z_k) / Σⱼ e^(z_j)

Every probability is one class’s exponentiated score as a share of the total across all of them. That shared denominator is the whole design.

Class 2 had the largest score and has the largest probability. Class 3 had the smallest and has the smallest. Softmax preserves the ordering of the scores and turns their gaps into relative shares, and the fitted middle person in section 16 is the same arithmetic on less tidy numbers.

In the assembly line above, hold two scores where they are and raise the third. Its probability rises, both of the others fall, and the total stays at one. Nothing was done to the other two scores. Their probabilities fell because the shared total grew, and that is what competition means here. Every class’s probability is divided by the same denominator, so no class can gain without the others losing.

Add the same constant c to every score.

softmax(z₁ + c, z₂ + c, z₃ + c) = softmax(z₁, z₂, z₃)

The shift slider in section 6 does exactly this, and the probabilities do not move, because e^(z + c) is e^z times e^c and the e^c cancels top and bottom. Softmax reads the differences between scores. The absolute level is invisible to it, and only relative evidence matters, which is the fact section 26 turns into a design decision.

Exponentiating a score of one thousand overflows an ordinary floating-point number. A direct softmax calculation can then try to divide infinity by infinity. Subtracting the largest score before exponentiation preserves all the probabilities.

largest shifted score = largest score − largest score = 0
largest exponential = e⁰ = 1

Every other exponential is at most one. Section 9 explains why this common shift leaves softmax unchanged.

softmax(z)_k = e^(z_k − m) / Σⱼ e^(z_j − m)          m = max(z)

The other route builds nothing new. It takes the one multiclass label column and expands it into three binary columns.

Each column is a binary problem of its own. The child classifier sees one child and two others, the teenager classifier one teenager and two others, and so on. A balanced three-class crowd becomes three unbalanced binary ones, and every class’s negative group is different.

Each column gets the logistic page’s model, fitted on its own.

q_k = σ(w_k · x + b_k)

Fitted separately. Each compares one class with a different combined rest. No shared denominator connects the three outputs, and no classifier knows what the other two assigned. That independence is the whole of the route, and it is what the next section’s totals reveal.

The chart below is every person’s row total under both routes on the worked people. Softmax is a flat line at one. One-vs-rest is not a line at all.

Independent class-versus-rest estimates do not form one multiclass distribution, and dividing them by their total to make them sum to one does not make them one either. It rescales three answers to three different questions, and the result is calibrated to nothing.

One-vs-rest still has to name one class, and the usual rule is to take the largest binary output.

predicted class = argmaxₖ q_k

That produces one class even though the three outputs were fitted with no reference to each other. Two cautions come with it. The largest output need not be a fair comparison, since three separately fitted models can be calibrated differently, and a 0.4 from one classifier is not necessarily more evidence than a 0.35 from another. And the ranking can differ from softmax’s. Both routes produce one predicted class, and they arrive at the comparison differently.

The three worked people are the child at 120 cm and 25 kg, the teenager at 150 and 50, and the adult at 180 and 75. The features are standardised before either route fits, and the standardised values are worth seeing.

The middle person becomes zero in both features, so their score under every class is that class’s intercept and nothing else. Standardisation changed the numbers, not who anybody is, and it is the reason the arithmetic in the next two sections is clean enough to do by hand. The widget in section 14 shows the raw and standardised coordinates side by side for whichever person is picked.

A distribution dominated by the teenager class, with the child given more than the adult because the child intercept, pinned at zero, sits above the adult’s −0.79. Pick the teenager in section 14 and the left column is this table, to the digit.

The total is unconstrained, the largest output still belongs to the teenager classifier, and both routes call this person a teenager. The numbers do not carry the same meaning. Softmax’s 0.9731 is a share of one unit. One-vs-rest’s 0.3333 is one classifier’s answer to one question, and the next Part is why that answer is exactly a third.

The teenager-versus-rest classifier has to call the middle person yes and both ends no, and the three people lie on one line. A linear boundary splits that line into a left and a right, so wherever it goes, the middle shares a side with one of the ends.

intercept = ln((1/3) / (2/3)) = ln(1/2) ≈ −0.6931

The model cannot isolate the middle person with a single straight boundary. In this symmetric example, its optimum ignores the feature and predicts the same probability for everyone. One of the three outcomes is yes.

constant probability q = 1/3
intercept = ln(q / (1 − q))
          = ln((1/3) / (2/3))
          = ln(1/2) ≈ −0.6931

The fit reaches that probability and intercept after 49 passes.

On the crowd both routes score an accuracy of 1.000. What that establishes is that both chose the correct largest class for every training person. What it does not establish is that their probabilities are equally good, that their decision regions are the same, that their calibration is equal, that they would agree on new people, or that either optimisation converged.

…

The two maps disagree on 88 of the 676 cells, and the probe finds points where the numbers differ a great deal even when the call is the same. Equal training accuracy hides substantial differences between two classifiers, and it hides them exactly where new observations arrive, between the training points.

For softmax the loss for one observation is minus the log of the probability given to the correct class, and the whole loss is the mean of that.

lossᵢ = −ln P(yᵢ | xᵢ)          loss = −(1/n) Σᵢ ln P(yᵢ | xᵢ)

The softmax figure is three terms, one per person, because each person contributes only the probability given to their own class.

−ln 0.9876 ≈ 0.0125
−ln 0.9731 ≈ 0.0273
−ln 0.9917 ≈ 0.0083

loss = (0.0125 + 0.0273 + 0.0083) / 3 ≈ 0.016

The one-vs-rest figure is nine terms, because each of the three classifiers answers for all three people. A classifier is charged −ln q for a person it should have said yes to and −ln(1 − q) for a person it should have said no to.

Nearly all of the 0.215 is the teenager classifier. The other two fits are charged 0.0047 each, because each of them can separate its one person from the other two and does.

The two losses are different objectives and the numbers should not be compared as if one probability model were being scored twice. Softmax is fitted to the multiclass loss. One-vs-rest is fitted to three binary losses independently, and its figure is the average of those. What both show is that accuracy checks only the winning class, while log loss looks at how the probability was distributed, and on the middle person the third the teenager classifier answered is a poor probability for a right answer.

Softmax outputs are a distribution. They can still be overconfident or underconfident, and summing to one says nothing about whether a 0.97 turns out to be right 97 percent of the time. One-vs-rest outputs may each be reasonably calibrated as binary probabilities and remain incoherent as a multiclass distribution, since three classifiers can all be right that their class is unlikely. Summing to one is necessary for a multiclass probability distribution and is not sufficient for a trustworthy one. Judging calibration takes data the model never saw, and belongs to the judging page.

Softmax is the natural choice when exactly one class is correct, the classes form one shared set of alternatives, a coherent distribution is required, and the classes should be trained against each other. One species among several. One digit from zero to nine. One way of getting to work from a fixed list, which was McFadden’s problem.

One-vs-rest is useful when a strong binary classifier is already to hand, when classes can be trained separately or in parallel, when new classes may need adding without refitting the rest, or when only a ranking or a final call is needed and jointly normalised probabilities are not. Its cost depends on the solver and the base model and is not automatically lower than one softmax fit. Its scores may need calibrating before they can be compared. And its performance has to be measured rather than assumed, which is section 19.

Independent sigmoid outputs are the natural model when several labels can be correct at once, which is section 2’s second panel.

P(y_k = 1 | x) = σ(w_k · x + b_k)

Each label gets its own threshold, several can be on at once, and the outputs need not sum to one because there is no one thing they are shares of. That is a multilabel problem and one-vs-rest’s machinery fits it naturally, which is a different statement from saying one-vs-rest is the right multiclass method whenever examples might have several labels. If they might, the problem was multilabel all along.

Write out softmax for two classes and divide the top and the bottom by e^(z₀).

Only the difference between the two scores matters, so binary logistic regression needs one score and not two, and multiclass softmax is the same competition extended to more classes. Two-class softmax and binary logistic regression are one relationship in two parameterisations.

Section 9 showed that adding a constant to every score changes nothing. Turned around, that means many different parameter sets produce the same probabilities, and a fit looking for the best parameters has infinitely many equally good answers. The probabilities are identified. The raw parameters are not.

The usual repairs are to pick one reference class and fix its coefficients at zero, to constrain the class parameters to sum to zero, or to add a penalty that prefers one finite parameterisation among the equals. The fit on this page takes the first. That is why the child row of section 3’s table is all zeros, and why pinning it there loses nothing. It is a choice about naming, not about the model.

For a one-hot target y with a one in the observed class and a predicted probability vector p, the derivative of one observation’s loss with respect to each class’s score is the gap.

∂lossᵢ/∂zᵢₖ = pᵢₖ − yᵢₖ
∂loss/∂w_k = Σᵢ (pᵢₖ − yᵢₖ) xᵢ

This is the logistic page’s gradient with one more index. An observed indicator, minus the predicted probability, weighted by the inputs. Softmax fitting adjusts every class’s score by the gap between what it predicted for that class and whether the class was the observed one.

The three worked people are perfectly separated, and the logistic page’s separation problem arrives with them. The dashboard below is the softmax ascent recorded pass by pass.

…

Accuracy is perfect from pass 2. The size of the coefficients is 2.3 at pass 10, 7.1 at pass 100 and 11.4 at pass 500, and it is still growing. The loss is 0.43, then 0.08, then 0.016, still falling. The probability each person’s own class receives climbs toward one and never arrives, and the run is reported as not converged, because no finite set of coefficients reaches the maximum. Perfect multiclass classification and an unbounded likelihood problem sit together comfortably.

One-vs-rest on the same people is a mixture. The teenager classifier converged in 49 passes, at the third from section 18, because it could not separate anything. The child and adult classifiers each see one person separable from two and ran out of passes with their coefficients still growing. A one-vs-rest model can contain convergent and nonconvergent binary fits at once, which the panel on the right reports one fit at a time.

The repair is the ridge page’s, for both routes. Penalise the size of the coefficients and extreme scores carry a cost, so the ascent settles at a finite point with probabilities short of zero and one. Regularised softmax has a finite maximum on separated data, and so does each regularised binary fit.

Everything above comes down to two guarantees a model can make about a table of class scores, and the two are kept apart by the kind of answer a fit returns rather than by prose.

The stronger guarantee is what expected-cost calculations over the classes need, and sampling one class, and multiclass log loss, and reporting a probability breakdown, and entropy. The weaker one is enough for ranking the classes, taking the top one, or retrieving the top few. Thresholding depends on what the threshold is meant to mean, and does not automatically need a normalised distribution.

That distinction found a real bug. The one-vs-rest specification said in prose that its rows deliberately do not sum to one, and went on returning the type that promises they do. The prose was right and unenforced, which is the state a type exists to replace. Now a caller who needs a distribution and is handed independent scores finds out when the types disagree, and not when a downstream sum comes out at 1.33.