Judging a Classifier

Twelve people are in the box above, six children and six adults measured by height and weight, and each is labelled with the chance the fitted model gives them of being an adult. Ten of them are easy. Two are not, since there is a child at 168 centimetres and 66 kilograms, and an adult at 150 centimetres and 50 kilograms. Their measurements overlap the other class, and this fitted model gives the child a higher adult score than the adult. A single threshold on those scores cannot classify both correctly.

The model gives the tall child a chance of 0.8669 and the short adult a chance of 0.4911, and with the dial at the halfway mark it calls the first an adult and the second a child. Accuracy reports 0.8333 and stops. The colours in the box say more, that one child was called an adult and one adult was called a child, and those are different mistakes that a reader may care about very differently.

Different applications can attach different costs to these two mistakes. This height-and-weight exercise illustrates that distinction; it does not establish a reliable way to determine somebody’s age.

On the twelve the two classes are balanced, so accuracy is merely coarse. Where one class is rare it is misleading. Take thirty people of whom four are adults, two of them child-sized, and consider the laziest classifier there is, which calls everybody a child. It finds no adult at all and it is right about twenty-six people in thirty.

…

The lazy classifier scores 0.8667, which is one minus the share of adults, and it would score 0.99 on a crowd where one person in a hundred was an adult. The fitted boundary at the halfway threshold scores 0.9333, finds three of the four adults and wrongly calls one child. The same boundary at a threshold of 0.7 also scores 0.9333, finds two of the four and wrongly calls nobody.

Two classifiers with the identical accuracy, one of which finds half again as many adults as the other, and a third that finds none and is right about two people fewer.

everybody a child        26 right of 30   =  0.8667     adults found  0 of 4
the boundary at 0.5     28 right of 30   =  0.9333     adults found  3 of 4, one child wrongly called
the boundary at 0.7     28 right of 30   =  0.9333     adults found  2 of 4, no child wrongly called

What replaces the one number is a table of four. Every person is either an adult or a child, and the model either called them an adult or did not, so each person falls into exactly one of four cells according to what was true and what was said. Two of the cells are the model being right and two are the model being wrong in the two different ways.

                      is adult          is child
called adult       true positive     false positive
called child       false negative    true negative

true positives    5        false negatives   1
false positives   1        true negatives    5

The words are a convention worth fixing once. Positive means the class the question is about, adult here, and has nothing to do with good news. A false positive is a child called an adult and a false negative is an adult called a child, so the first word says whether the call was right and the second says what the call was.

Suppose the model has just called somebody an adult. How much should we believe it? That question only concerns the people it called adult, which is the top row of the table, and asks what share of that row is on the diagonal. On the twelve the model called six people adult and five of them were.

precision  =  true positives / (true positives + false positives)
           =  5 / (5 + 1)  =  0.8333

…

Precision is the rate to watch when a false alarm is what costs. A test that sends every flagged person for an expensive second examination wants the flags to be right, and it can raise its precision by flagging fewer people, which is exactly the move the next question punishes.

Now ask the other question. Of the adults in the crowd, how many did the model find? That concerns only the people who really are adults, which is the left column of the table, and again asks what share is on the diagonal. There are six adults and the model found five.

recall  =  true positives / (true positives + false negatives)
        =  5 / (5 + 1)  =  0.8333

The two rates share a numerator and differ in what sits underneath, and switching the buttons above between precision and recall shows the highlighted cells turning from the row to the column. On the twelve they come out equal, which is a convenience of this crowd rather than a rule, since there happen to be six of each class and one miss in each direction. On the rare crowd at the halfway threshold they also happen to agree, at 0.75, and at a threshold of 0.7 they part to 1.0 and 0.5.

Recall is the rate to watch when a miss is what costs. A screening test wants to find every case and will accept false alarms to do it, and it can raise its recall by calling more people, which is the move precision punishes. The two are the same table read in two directions, and which one a field quotes first says mostly which mistake that field is more afraid of.

Nothing so far has looked at the children except to count the wrongly called ones. Specificity is recall for the other class, the share of the children the model left alone, and it reads down the right column. Its complement, the share of children the model wrongly called adult, is the false positive rate, which is the horizontal axis of the curve in Part 3.

On the rare crowd the difference between the two columns is plain. The boundary at 0.5 has a specificity of 0.9615, because twenty-five of twenty-six children were left alone, and a recall of 0.75, because three of four adults were found. Both are read off the same table, and a report that quoted only the first would be describing how well the model does on the class it was not asked about.

Each of these rates is a count over a count, and a count can be zero. Turn the dial in the box above to 0.99, higher than any chance in the crowd, and the model calls nobody an adult. Its precision is then zero over zero. That is not a precision of zero, since a model that made no positive claims has not made a wrong one, and the difference is kept by refusing to return a number rather than by returning 0.

precision at 0.99   =   0 / (0 + 0)     undefined, nobody was called adult
recall on a fold with no adults   =   0 / (0 + 0)     undefined, there was nobody to find
F1 with precision 0 and recall 0   =   2 · 0 · 0 / (0 + 0)     undefined

Recall has the same gap when there are no adults to find, which no threshold can cause on a fixed crowd and which a held-out fold of a small crowd causes easily, as Part 7 shows. Accuracy has a denominator whenever there is anybody at all, and specificity whenever there is a child. A tool that quietly substitutes zero for any of these turns a fact about the data into a poor score for the model, and every rate on this page comes back as undefined where its denominator is empty.

The model in the box does not say adult or child. It says 0.4911, and somebody has to decide what to do with that. The logistic regression page introduced the threshold as the number a chance is compared against, and this page treats it as a dial, because the boundary and every chance were computed once and the dial only moves the line between the two calls. Nothing refits when it turns, and the four cells change only because people cross the line.

…

Drag the marker down to 0.3. The short adult’s 0.4911 now clears the bar, so all six adults are found and recall is 1.0, while the tall child’s 0.8669 still clears it too, so seven people are called adult and six of them are, for a precision of 0.8571. Drag it up to 0.9 and only the three adults above 0.9 are called, so precision is a perfect 1.0 and recall halves to 0.5, with the adults at 0.7376, 0.7684 and 0.4911 now called children.

The model did not change between the two readings, and neither did any chance; the line the calls are read against did.

Raising the threshold can only turn a call of adult into a call of child, never the reverse, so the true positives can only fall and the false negatives can only rise. Recall is the true positives over the number of adults, and the number of adults does not depend on the threshold at all, so recall is a falling count over a constant and cannot rise. That is why the indigo curve above only ever steps downward.

threshold up   ⇒   TP can only fall,  FN can only rise
recall  =  TP / (TP + FN)     where TP + FN is the number of adults, fixed

so recall never rises as the threshold rises

Precision has no such guarantee, because both its numerator and its denominator move. Reading the amber curve at five thresholds on the twelve gives 0.75 at 0.1, 0.8571 at 0.2, 0.8333 at 0.5, 0.75 at 0.8 and 1.0 at 0.9. It rose, fell, fell and rose, and each move is one person crossing the line.

Between 0.2 and 0.49 the seven people above the line include the tall child and every adult; at 0.5 the short adult drops out, taking one true positive from a numerator of six and one person from a denominator of seven, and the ratio falls from six sevenths to five sixths.

Put the two rates against each other rather than against the threshold, and the threshold becomes a position along the curve. Start with it above the highest chance, where nobody is called, and lower it through each person’s chance in turn. Every step adds one person to those called adult. If they were an adult, recall moves right; if they were a child, precision drops. Nothing else can happen, so the curve is a staircase, and it is drawn as one.

…

On the twelve there are twelve distinct chances and so twelve corners. The first three steps add the three most confident adults and precision holds at 1.0 while recall climbs to 0.5; the fourth adds the tall child and precision drops to 0.75 with recall unmoved; three more adults take recall to 1.0 with precision recovering to 0.8571; and the last five steps add the remaining children, lowering precision to 0.5 while recall has nowhere left to go.

The area under the steps is the average precision, 0.9151 here, and the rightmost precision is always the share of the crowd that is adult, since calling everybody adult finds every adult and is right about a prevalence’s worth of them.

average precision  =  Σ  (recall gained at the corner) × (precision at the corner)
                   =  (1/6)·1 + (1/6)·1 + (1/6)·1 + 0·0.75 + (1/6)·0.8 + (1/6)·0.8333 + (1/6)·0.8571 + 0 + …
                   =  0.9151

The other picture swaps precision for the false positive rate and puts recall up the vertical axis. It starts at the origin, where nobody is called and neither rate has anything in its numerator, and ends at the top right, where everybody is called. Lowering the threshold through an adult moves it up by one sixth, and through a child moves it right by one sixth, so a model whose chances rank every adult above every child climbs the whole left edge before crossing the top.

…

The area under the curve has a meaning that can be counted rather than integrated. Pair each adult with each child, thirty-six pairs on the twelve, and ask in how many the adult’s chance is the higher. The tall child at 0.8669 outranks three adults, the short one at 0.4911 and two more at 0.7376 and 0.7684, and every other pair is in the right order, so thirty-three of thirty-six are, and the area is 0.9167. A coin would draw the diagonal and score 0.5.

area under the ROC  =  share of (adult, child) pairs with the adult ranked higher, a tie counting a half
                    =  33 / 36  =  0.9167 on the twelve
                    =  102 / 104  =  0.9808 on the rare crowd

The two curves are built from the same sweep and disagree about what a good model looks like when one class is rare, because one of them reads a column of the table and the other reads a row. Recall and the false positive rate are each a share of one class, so the ROC does not know how many of each class there are, and its diagonal is a coin’s curve on any crowd.

Precision is a share of the people called adult, who are drawn from both classes, so the precision-recall curve knows the prevalence intimately and ends at it.

On the rare crowd the model’s average precision is 0.9167 against a floor of 0.1333, where on the twelve it is 0.9151 against a floor of 0.5, so the rare crowd’s figure is the larger achievement though the two numbers are within 0.002 of each other. The ROC area reads 0.9808 and 0.9167 against the same floor of 0.5 on both.

When the positive class is rare and the false alarms are what will be looked at, the precision-recall picture is the one that shows what it will cost, since a false positive rate of 0.04 on the rare crowd is one child, and on a crowd of ten thousand children it would be four hundred people to be examined for every few adults found.

Sometimes two rates have to become one, to rank candidate models or to put a figure in a report. The obvious way is to average them, and it is wrong in a specific way. At a threshold of 0.9 the twelve give a precision of 1.0 and a recall of 0.5, and the arithmetic mean of those is 0.75, which is a respectable-looking score for a model that missed half the adults. The harmonic mean is 0.6667, and it is lower because the harmonic mean is pulled towards whichever rate is worse.

F1  =  2 · precision · recall / (precision + recall)

at 0.9 on the twelve:   2 · 1.0 · 0.5 / (1.0 + 0.5)  =  0.6667     the arithmetic mean would say 0.75
at 0.5 on the twelve:   2 · 0.8333 · 0.8333 / 1.6667  =  0.8333    the two agree, so any mean of them is 0.8333

F1 weighs the two rates equally, and nothing says they should be. The general form carries a number beta that says how many times more recall matters than precision, and the usual pair of alternatives are a half, which leans towards precision, and two, which leans towards recall. At a half the recall is weighted a quarter as much in the denominator; at two it is weighted four times as much.

F_β  =  (1 + β²) · precision · recall / (β² · precision + recall)

at 0.9 on the twelve, precision 1.0 and recall 0.5:
   F½   =  1.25 · 0.5 / (0.25 + 0.5)  =  0.8333
   F1   =  2 · 0.5 / (1 + 0.5)        =  0.6667
   F2   =  5 · 0.5 / (4 + 0.5)        =  0.5556

…

The three curves agree wherever precision and recall agree, which on the twelve is the whole stretch from 0.5 to 0.73 where both are 0.8333, and part company wherever the two rates do. At 0.9, where precision is perfect and recall is half, the precision-leaning score says 0.8333 and the recall-leaning one says 0.5556, and the reader who is choosing between them is choosing which mistake to forgive.

Once there is one number, the sweep can be searched for the threshold that maximises it, and the dashed lines in the chart above mark where each F-score peaks. On the twelve all three peak at 0.18, which is the lowest threshold in the long flat stretch from 0.18 to 0.49 where every adult is found and only the tall child is wrongly called, and the sweep reports the lowest tied threshold rather than choosing among equals. On the rare crowd the three part.

On the rare crowd the recall-leaning score sends the threshold down to 0.14, where all four adults are found and two children are wrongly called, and the other two hold it at 0.51, just above the one child the model gives a chance of 0.5072, where three adults are found and nobody is wrongly called. Those are two defensible deployments of one model, and beta is what chose between them.

With children, teenagers and adults there are three ways to be right and six ways to be wrong, and the table grows to three rows by three columns, the true class down the rows and the call across the columns. The diagonal is the right answers and every other cell names a particular confusion, this class taken for that one. The crowd below is the multiclass page’s thirteen with three people added who confuse the classes, a teenager built like an adult, an adult built like a teenager, and a tall child.

With everybody judged and the width stated, fourteen of the sixteen are on the diagonal, for an accuracy of 0.875. The two off it are the tall teenager, called an adult, and the short adult, called a teenager, and the tall child was placed correctly. Those two cells say something a two-class table cannot, that the model confuses teenagers with adults in both directions and never confuses either with a child.

Precision and recall survive the move to three classes, and they stop being single numbers. Each class gets its own pair, read off its own row and column of the table exactly as in Part 2, with that class standing as the positive one and the other two together as the negative.

teenager   precision  5 / 6  =  0.8333     recall  5 / 6  =  0.8333     F1  0.8333
adult      precision  4 / 5  =  0.8000     recall  4 / 5  =  0.8000     F1  0.8000
child      precision  5 / 5  =  1.0000     recall  5 / 5  =  1.0000     F1  1.0000

A model can be excellent on the common class and useless on the rare one, and that is exactly the fact a single number hides. The per-class figures are what a report should show when the classes are of different sizes or different importance, and they are one row of the table each.

Three precisions can be collapsed to one in two ways, and they answer different questions. Averaging the three per-class figures, which is called the macro average, gives every class an equal vote however many people it holds. Pooling the counts first and dividing once, the micro average, gives every person an equal vote and lets the common classes dominate.

macro precision  =  (1.0000 + 0.8333 + 0.8000) / 3  =  0.8778
micro precision  =  (5 + 5 + 4) / (5 + 6 + 5)      =  14 / 16  =  0.8750

On this crowd the two are close, 0.8778 against 0.8750, because the classes are nearly the same size. The micro figure is worth a second look, because where every person gets exactly one call the pooled numerator is the diagonal and the pooled denominator is everybody, so micro precision, micro recall and accuracy are the same number, 0.875 here, reported under three names, and the widget shows all three so the reader can see that they agree.

A three-class table needs to know it has three classes, and there are two ways it can find out. It can be told, or it can read the true classes of the rows it was handed and count. The second sounds harmless and is where a held-out fold goes wrong, because a fold of a small crowd can easily hold no adults, and a table that counts its classes from the rows then has no column for adults at all.

Switch the widget in section 16 to judge the crowd without its adults and untick the stated width.

The quiet case is the one the refusal above narrowly avoids. I measured a version of this crowd without the short adult, where the model places the tall teenager correctly, so that no judged row was called adult, and the inferred table came back two by two with no adult column and every figure defined, a table that is silently the wrong shape.

Stating the width turns that into a three-by-three table with an empty row, whose recall is undefined and says so. So the width is stated up front and the rows are held to it, which is the rule the fold endpoint of Part 7 uses on every fold.

Everything so far has judged the calls, and a call throws away the chance it came from. The model said 0.7376 about one adult and 0.9546 about another, and if those numbers mean what they say then, among many people given a chance near 0.75, about three in four should turn out to be adults. Whether they do is a separate question from whether the calls were right, and it is answered by binning the chances and counting.

…

On the twelve the bin from 0.8 to 1.0 holds four people at a mean chance of 0.9230, and three of the four are adults, a share of 0.75 against a promise of 0.92, since the tall child is one of the four. The bin from 0.6 to 0.8 holds two people at a mean of 0.7530 and both are adults.

The bin from 0.4 to 0.6 holds one person, the short adult, and the bin from 0.2 to 0.4 holds nobody. Twelve people cannot fill five bins, so most of what the diagram says here is that twelve is too few, which is a fair thing for it to say.

Over everybody the model balances, with a mean chance of 0.5000 against an observed adult rate of 0.5, which a logistic fit with an intercept does on its fitting rows because the fit itself demands it. The balance in the large and the gaps in the bins are both true at once, and only the bins can see that the chances above 0.8 are too high.

The diagram can be summarised in two numbers, and they measure different things. The expected calibration error takes each bin’s gap between its mean chance and its observed share, weights it by how many people the bin holds, and averages. The Brier score skips the bins and takes every person’s squared gap between their chance and their outcome, 1 for an adult and 0 for a child.

expected calibration error  =  Σ_bins  (people in bin / everybody) · | mean chance − observed share |
Brier score                 =  (1 / n) Σ_people  (chance − outcome)²

The area under the ROC and the reliability diagram can disagree completely about one model, and the reason is in what each reads. The area counts pairs and asks only which of two chances is the higher. Squash every chance towards 0.5, or stretch every chance towards the ends, and as long as the order of the people is untouched the area is untouched, at 0.9167 on the twelve, while the diagram moves every dot.

This is why the two are reported separately and why a model with a high area can still be untrustworthy when its 0.7 is read as seven in ten. The repair for that is a second, small fit that maps the model’s chances onto observed frequencies on held-out rows, which is not built here and which this page therefore only names. What it does provide is the diagram and the two scores, which say whether such a repair is needed.

The twelve in the box are the twelve the boundary was fitted to, and so are the thirty and the sixteen. Every rate, both curves, every peak and every bin above is therefore a training score, and the held-out evaluation page is about why that flatters and how dealing the crowd into folds and judging each on the rows it was not fitted to repairs it. That argument is not repeated here. What this page adds is what happens when the thing being folded is a table rather than a single squared error.

Two things change. The standardising of heights and weights that the fit needs has to be learned inside each fold, so that a held-out person never lends their mean to the fit that judges them, and the widget below does that. And a fold of two or three people can hold no adults at all, which is the undefined recall of section 7 arriving in practice.

With the twelve dealt into folds and the boundary refitted inside each, there are two ways to combine the folds into one verdict. Average each fold’s accuracy, so each fold counts equally, or add the folds’ tables into one table and divide once, so each person counts equally. The widget deals the same twelve four ways and reports both.

The pooled column reads 0.8333 on every row, because every person is held out in exactly one fold whatever the deal, so adding the folds’ tables always rebuilds one table of the same twelve people. The averaged column does not. Dealt seven ways it says 0.8571, because twelve people do not divide into seven equal folds and averaging gives a fold of one the same vote as a fold of two, and dealt five ways with the classes balanced it says 0.8.

On the plain five-fold deal the two happen to agree to the last digit, which is arithmetic luck and not a rule.

What settles the question is the fold with no adults, which the plain five-fold deal produces. It holds two people, the refitted boundary wrongly calls one of them an adult, and its recall is zero found over zero present. Averaged, that fold has to contribute something and whatever is chosen is a convention. Pooled, it adds nothing to either side of the ratio, which is right, because a fold with no adults in it has no evidence about finding adults.

So a classifier’s folds are pooled, and a spread across folds is reported only for accuracy, which is defined on any fold with anybody in it and reads 0.5 here on every deal.

pooled     ( Σ TPₖ ) / ( Σ (TPₖ + FNₖ) )      a fold with no adults adds 0 to both
averaged   (1/k) Σ  TPₖ / (TPₖ + FNₖ)        a fold with no adults has no term to add

A complete implementation of these judgements states which class is positive and what labels it accepts, whether the threshold comparison is at or above, what it does with a denominator of zero for each rate, whether its curves are read at every distinct score or at a fixed grid, whether the area under the precision-recall curve is a step sum or a trapezoid and whether a conventional first point is added, how ties in the scores are counted in the ROC area, which beta an F-score uses, whether a multiclass average is macro or micro and what it does when a class’s rate is undefined, whether the class width is stated or inferred, how many bins a reliability diagram uses and whether they are equal in width or in count, and whether folds are pooled or averaged.

The choices on this page are these. Adult is positive and the labels are 0 and 1; a person is called adult at or above the threshold; an empty denominator gives an undefined rate rather than zero; the sweep reads every hundredth and both curves read every distinct chance; the average precision is a step sum with no conventional first point; a tied pair counts a half; the three betas are a half, one and two; both multiclass averages are reported and the macro one is undefined if any class is; the width can be stated and is inferred otherwise; five equal-width bins; and folds are pooled, with a spread for accuracy alone.

Every row below was run, through this page where it can reach the case and directly on the model where it cannot.

Two of those are documented rather than defended. A fit that never converges is the logistic regression page’s complete-separation case, and this page reports the passes run and the flag rather than refusing, because the chances it produces are still a ranking even where their values have run away. And the reliability reading needs bins, and a different binning gives a different calibration error from the same chances, so that number is reported with its binning and the Brier score beside it.