Held-Out Evaluation

Think of a student preparing for an exam from a booklet of practice questions. Score them on those same practice questions and you cannot tell two very different students apart, the one who understood the subject and the one who memorised the answer key, since both score perfectly. The only way to separate them is a question they have not seen. Everything on this page is that idea applied to fits, and the fit in question is a curve through the fifteen noisy measurements of the thrown ball.

The playground above has four of the fifteen hidden from the fit and ringed in amber, and the curve is fitted to the eleven indigo measurements alone. At degree 2 the two readouts nearly agree, 0.995 on the training share and 0.989 on the held-out one, since a parabola is what a thrown ball actually does.

At degree 9 the training share reads 0.999 and the held-out share reads about −5480. Nothing about the procedure changed between those two lines except how many polynomial terms the model could use, and the second number is what a curve that has memorised its eleven points does when it is asked about the other four.

There is a reason the training readout in the playground never falls as the degree rises, and it needs no calculus. Every curve of degree d is also a curve of degree d + 1, namely the one whose top coefficient is zero, so the collection of curves the degree d + 1 fit may choose from contains the whole collection the degree d fit chose from. Least squares picks the best curve in its collection, and a larger collection cannot have a worse best.

best RSS at degree d + 1  ≤  best RSS at degree d
so   training R² at degree d + 1  ≥  training R² at degree d

On the fixed split the training curve runs 0.222, 0.995, 0.996, 0.996, 0.996, 0.996, 0.997, 0.997, 0.999 from degree 1 to degree 9, never once falling. The held-out curve is free to do anything, and it rises to 0.989 at degree 2, wanders between 0.966 and 0.993 through degree 5, then reads 0.693, −1.396, −104 and −5479.

Read the argument for what it does not say. Nothing in it mentions the hidden rows, so the guarantee binds only the data the fit was optimised on, and a score that is guaranteed never to fall as the model grows more capable cannot say when to stop.

Both readouts are the regression page’s R squared, computed on whichever share they name. The residual sum is the curve’s squared misses on that share, and the yardstick it is measured against is the squared misses of a flat line at that same share’s own mean. So the held-out score does not ask how the curve compares with a flat line through the training mean; it asks how the curve compares with the best flat line for the four hidden rows themselves, and it can lose that comparison.

R² on a share  =  1 − RSS on the share / TSS on the share
TSS on the share  =  Σ (hᵢ − mean of h over that share)²

A held-out score below zero therefore means the curve predicts the hidden measurements worse than the average of those four would have, and the degree 9 curve does not lose mildly. Its squared misses on the four hidden rows are about 5480 times the spread those rows have about their own mean, because between two of its eleven training points it swings through heights no ball ever reached.

The measurements arrive in time order, from the throw at zero seconds to the landing at four, and cutting an ordered list without shuffling would hold out the whole landing. So the recipe shuffles first. The fifteen positions are permuted by a generator with a stated seed, a stated share of them is taken as the held-out rows, and the rest train. Seeding the shuffle is what lets this page quote the numbers it quotes, since the same seed deals the same rows every time.

The split here rounds the wanted share to a row count and clamps it so that neither side is ever empty, which is why a half of fifteen rows comes out as eight held out and seven kept rather than the other way round, since seven and a half rounds to the even neighbour. It also takes the held-out block from the front of the shuffled order, where the library’s own splitter takes it from the end, so the two hide different rows under one seed, which the practice section at the end makes visible.

The held-out rows must touch nothing about the fit, and that rule reaches further than the coefficients. The polynomial page manufactures its columns, the time squared and cubed and so on, from the one measured column, and those columns are rebuilt from the eleven training rows only, so the four hidden rows are transformed afterwards by a recipe that never saw them. For a polynomial the recipe learns nothing from the rows, only the names, so the point is invisible here, and Part 6 measures what happens when the recipe does learn something.

One fit, two verdicts. The training share answers how well the curve matched its examples; the held-out share answers whether what it learned travels, and the difference between them is a signal of possible overfitting, also affected by sampling and distribution differences. On the fixed deal the gap is 0.006 at degree 2, 0.030 at degree 3, 0.010 at degree 5 and about 5480 at degree 9, and by that reading the memorisation begins in earnest at degree 6, where the held-out score first drops to 0.693 while the training score sits at 0.996.

Notice one thing the table shows that the playground’s story does not. The best held-out score on this deal is at degree 4, at 0.993, and degree 2 is 0.004 behind it. A reader who chose the degree by this one held-out score would pick a quartic for a thrown ball, and Part 3 is about whether that 0.004 means anything.

Four rows is a small jury and eleven rows is a small class, and the share sets both at once. Hold out fewer rows and the fit is better fed while the verdict rests on two or three measurements; hold out more and the verdict steadies while the fit starves. The table refits the degree 2 curve under thirty seeds at each share, so each row is a whole family of deals rather than one.

…

At a tenth of the rows the jury is two measurements, and across thirty deals the lowest verdict is −8.34 with a spread of 9.33, since two rows have almost no spread of their own for a ratio to explain. At three tenths the spread is 0.043, at four tenths 0.032, and at a half 0.031, after which holding out more begins to cost the fit and the spread turns back up to 0.044 at six tenths.

The mean training score moves from 0.9957 to 0.9964 over the whole table, because a parabola is easy to fit on six rows or on thirteen.

The number we actually want is how well a curve fitted this way would do on measurements of the ball we have not taken yet, and no finite sample can give that number exactly. The held-out score is an estimate of it, computed from four rows the fit did not see, and an estimate from four rows is a guess with a large margin. It is an honest guess where the training score is not, since the four rows were not consulted by the fit, and it is still a guess.

held-out R²  ≈  R² the fit would earn on measurements never taken
                 estimated from the few rows held back

The way to find out how much a single held-out score is worth is to deal again. The strip below holds out four of the fifteen under thirty different seeds, refits the curve on each training share, and scores each held-out share, so every amber dot is a verdict the playground could have shown instead of the one it does.

At degree 2 the thirty verdicts run from 0.955 to 0.998, a spread of 0.043 around a mean of 0.988, while the mean training score across the same thirty fits is 0.996. So the 0.004 by which degree 4 beat degree 2 on the fixed deal is a tenth of the distance the verdict moves when nothing changes except which four rows were hidden, and it settles nothing.

Turn the dial to degree 5 and the lowest verdict is 0.355 against a highest of 0.998, a spread of 0.643, which is the same curve family being called excellent by one deal and poor by another.

The two summaries in the strip behave very differently as the seed changes, and the reason is which rows each one reads. The training score is computed on eleven rows and they are the same rows the coefficients were chosen to please, so it barely moves. The held-out score is computed on four rows the curve had no say over, and which four they are decides how hard the question is.

A held-out share that happens to hold the landing, where the ball is moving fastest and the noise costs most, asks a harder question than one that holds four measurements from the top of the arc.

A single split spends four rows on judging and never lets them train, and spends eleven on training and never lets them judge. Folding lets every row do both. Shuffle once, cut the fifteen into five folds of three, and then five times over hold one fold out, fit on the other four, and score the hidden one. Every measurement is judged exactly once by a fit that never saw it, and every fit sees twelve of the fifteen rather than eleven.

Under the page’s seed the five folds hold measurements 1, 10 and 12, then 0, 2 and 7, then 8, 9 and 13, then 3, 4 and 6, then 5, 11 and 14, and at degree 2 they score 0.992, 0.992, 0.998, 0.966 and 0.996. The fourth fold holds measurements 3, 4 and 6, three heights between 13 and 19.4 metres near the top of the arc, and it is the one that scores lowest, for the reason section 10 gave.

Fifteen rows do not always divide evenly, and when they do not the extra rows go to the earlier folds, so ten folds of fifteen are five folds of two and five of one.

The summary of five fold scores on this page is their mean, and read across every degree it is the validation curve, the chart on this site that every dial is set by. The candidate at each degree is the whole pipeline, the column manufacturing and the regression together, refitted inside every fold so no fold ever scores a column built from rows it is about to judge.

mean across folds  =  (1/k) Σ R²ₖ
spread across folds  =  max R²ₖ − min R²ₖ

At degree 2 the mean is 0.989 and at degree 3 it is 0.989 as well, 0.0001 behind, so the folds agree with the throw that a parabola is enough. The fixed deal’s preference for degree 4 has gone, since across the folds degree 4 reads 0.986. At degree 8 the mean is −1.44, and at degree 9 it is 0.71, which is not a recovery. Section 13 explains why the mean at those degrees is not a number to read on its own.

The second number reported is the spread, the best fold’s score less the worst’s. At degree 2 it is 0.032, so the five verdicts nearly agree and the mean stands on firm ground. At degree 8 the five folds read 0.979, −10.29, 0.828, 0.916 and 0.369, a spread of 11.27, and the mean of −1.44 above that band is an average of verdicts that have nothing in common. The honest reading is that a degree 8 curve has become a gamble on which three rows it is denied.

Two things the spread is not. It is not a standard error, since it is a range over five numbers and grows with the number of folds for that reason alone, so it prices the trust in one mean and should not be used as a threshold for choosing between two. And the five verdicts are not independent, because any two of the five fits share three of their four training folds, so the folds agreeing with each other is weaker evidence than five separate experiments agreeing would be.

There is a second way to summarise the folds. Instead of scoring each fold’s three predictions on their own and averaging the five scores, collect all fifteen held-out predictions, each made by a fit that never saw its row, and score them once against all fifteen measurements. Every row was held out exactly once, so that is one prediction per row of the whole dataset, judged against the whole dataset’s spread.

pooled R²  =  1 − Σ over folds RSSₖ / TSS of all fifteen measurements

At degree 2 the pooled score is 0.994 where the mean across folds is 0.989, and the pooled squared error is 0.248 per measurement. They differ because each fold’s own score measures its misses against the spread of only three rows, and the fourth fold’s three rows from the top of the arc have very little spread, so the same misses cost that fold 0.034 of R squared and cost the pooled figure almost nothing. At degree 8 the two part company harder, −1.44 against −2.76.

Two folds, three, five, ten or fifteen. More folds means each fit sees more of the data and the verdict is built from more fits, and it means more fits to pay for. On fifteen measurements the table runs the degree 2 curve through every choice under one seed.

…

From two folds to five the mean across folds moves from 0.991 to 0.989 and the pooled score from 0.991 to 0.994, which is the difference between fits trained on seven or eight rows and fits trained on twelve. At ten folds five of the folds hold a single measurement, and at fifteen every fold does, which is leave-one-out.

A fold of one row has no spread about its own mean, so its R squared is a ratio with zero underneath, and it is refused by name rather than reported as a number. The mean across folds therefore stops existing at ten folds, while the pooled score carries on, 0.993 at ten and 0.993 at fifteen, because pooling measures the misses against the spread of all fifteen rows and a single held-out row contributes only its miss.

One more debt closes here. The bagging page’s out-of-bag score is this same idea collected free of charge. Each member of a bagged committee is fitted on a resample that misses about a third of the rows, so every row can be judged by the members whose resample happened to miss it, with no folds dealt and no extra fits made. The difference from folding is that the number of judges a row gets is a matter of luck rather than exactly one, and with few members some rows get none at all, which that page counts rather than hides.

The throw has a number to predict at every row, so any three rows make a fold that can be scored. A classifier’s folds are different, because a fold is only a test of the boundary if it holds people from both sides of it. The twelve overlapping people from judging a classifier, six children and six adults, dealt plain into five folds under the page’s seed, show what a shuffle-and-cut can do with two classes.

Under the plain deal the five folds hold 1, 2, 2, 0 and 1 adults, so the fourth fold holds two children and no adult and the third holds two adults and no child, so the count of folds missing a class is two. A tree two questions deep is refitted inside each fold. On the third fold it gets neither adult right, because the two it was denied, at 150 and 159 centimetres, are exactly the two adults inside the children’s range, and the tree trained without them puts both on the children’s side.

On the fourth fold the question of recall cannot be asked at all, since recall is the share of adults found and the fold has no adults to find.

The repair is to deal each class separately. Split the twelve into a pile of children and a pile of adults, shuffle each pile on its own, and deal each pile round the five folds in turn, so randomness still decides which adult lands in fold two and no longer decides how many do. Switch the deck above to the stratified deal and the five folds hold 1, 2, 1, 1 and 1 adults, no fold is missing a class, and recall exists on every fold.

plain      shuffle all twelve, cut into five blocks
stratified deal the six children round the folds, then the six adults

Stratifying distributes what exists and cannot manufacture what does not. Relabel the crowd so that only one adult remains and deal it stratified, and four of the five folds hold no adult whatever the seed, because one person can only be in one fold. The count of folds missing a class is reported rather than assumed to be zero once the deal is stratified, and on this page that count is 0 for the twelve with six adults and 4 for the twelve with one.

With the folds scored there are two ways to combine five accuracies into one. Average the five, so each fold counts equally, or add the five folds’ confusion tables into one table and divide once, so each person counts equally. On the plain deal the averaged accuracy is 0.4667 and the pooled accuracy is 0.5, and on the stratified deal they are 0.8 and 0.8333.

Twelve people do not divide into five equal folds, so averaging gives a fold of two the same vote as a fold of three, and the two disagree by exactly that reweighting.

pooled     ( Σ rightₖ ) / ( Σ heldₖ )        =  6 / 12  on the plain deal
averaged   (1/5) Σ  rightₖ / heldₖ           =  0.4667  on the plain deal

The reweighting is a nuisance, and the fold with no adults is what settles the question. Recall on that fold is zero found over zero present, and averaging has to do something with it, either drop the fold and average the other four, which gives 0.5 here, or invent a value. Pooling adds zero to the top of the ratio and zero to the bottom and needs no convention, and it reads 2 of 6 adults found, 0.3333, which is the number the twelve people actually earned.

A classifier here is scored by pooling for exactly this reason, with a spread reported only for accuracy, which is the one rate that is defined on any fold with people in it.

The regression result reports a spread across folds beside its mean, and the classification result reports one too, for accuracy alone. Accuracy is a count of people got right over a count of people held out, and every fold with anyone in it has both counts, so the five accuracies always exist and their range can be read. Recall and precision cannot be spread the same way, since on the plain deal one fold has no recall to contribute and the range over four numbers would be pretending to be a range over five.

On the plain deal the spread is 1.0, since one fold got everyone right and another got nobody right, and a pooled accuracy of 0.5 sitting over a spread of 1.0 is the same warning the degree 8 curve gave in section 13. Stratified, the spread is 0.5 and the pooled figure 0.8333, and on twelve people a spread of 0.5 is still one person in a fold of two.

Section 5 said the held-out rows must touch nothing about the fit, and a preprocessing step that learns a mean and a spread from the rows is part of the fit. Standardise the columns once over all fifteen measurements before dealing the folds, and every training fold has been scaled by a mean that the held-out rows helped to compute. The table measures what that costs on the throw, for the degree 2 terms scaled either before the folds are dealt or refitted inside each fold, under thirty seeds.

For a ridge fit at a penalty of 0.01 the mean score is 0.9676 scaled beforehand and 0.9669 scaled inside, a gap of +0.0007 that flattered 15 of the 30 seeds and went the other way on the other 15, with the largest single-seed gap at 0.019. That is the same shape the pipelines page found on its larger measurement, a leak within noise of zero with about half the seeds flattered, because a mean and a spread do not consult the answers and so cannot lean the fit toward them.

Raise the penalty to 0.1 and the gap is +0.014 with 9 of 30 flattered; lower it to 0.001 and it is +0.00007. Measured on the throw, then, the leak is a few thousandths at most, with the seeds divided about evenly on its sign.

The second row of the table is the control, and it is exact. A least-squares fit with an intercept is unchanged by shifting any column or scaling it, since the intercept absorbs the shift and the coefficient absorbs the scale and the fitted heights come out identical. So it does not matter where the standardiser learned its mean, and the two arrangements agree to the last decimal, a mean gap of 7.5 × 10⁻¹⁶ and a largest single-seed gap of 2.9 × 10⁻¹⁴, which is floating-point rounding and nothing else.

There is a leak the folds cannot close, and this page has already committed it. Section 12 read the validation curve and chose degree 2, and the score it chose by, 0.989, is now the best of nine candidates rather than the score of one, and the best of several noisy estimates is higher than any of them deserves. Nine candidates on a smooth curve make the effect small here, and the grid search page measures it on a case built to show it, twenty-five candidates on a target of pure noise, where the winner reported a score no candidate could honestly earn.

A held-out evaluation states its deal in full. The share held out and how it is rounded to a row count, whether the rows are shuffled and by what seed, which block of the shuffled order is held out, how many folds and how uneven sizes are handled, whether the folds are stratified and on what column, what is refitted inside every fold, which metric is read on a fold and what happens when that metric is undefined there, whether the folds are averaged or pooled, and what the spread beside the summary is a range of.

Every one of those is a number a reader can check only if it is written down.

What follows is what the code behind this page actually does at every edge a reader might hand it, each row run rather than remembered. Most are refused by name, a few are refused at the door before any fitting begins, two are accepted with a fact reported beside the answer, and one is documented rather than defended.

Every row of that table was run for this page. The one behaviour documented rather than defended is the rounding of the held-out share, which follows the round-half-to-even rule the language uses, so a reader who expects a half of fifteen to hold out seven should read the count the response reports rather than assume it.