The Standard Score

The orchard this page works through is twenty-four apple trees. Each one was measured twice before the harvest, its trunk girth written down in millimetres and the water it was given over the season written down in metres, and then the fruit it bore was weighed in kilograms. Both units are ones somebody could reasonably have chosen, and neither is wrong; what matters is that they are a thousand apart, so the trunk numbers run in the hundreds while the water numbers run below one.

The two columns are very nearly unrelated to each other, their correlation being −0.083, and they are not at all equally useful. Water very nearly decides the fruit; the trunk barely moves it. That combination is what makes the orchard worth measuring on, since the column recorded in the larger numbers is also the column that carries almost nothing.

Take one tree and ask which three trees in the orchard are most like it, then predict its fruit as the average of what those three bore. The likeness has to be a number, and the usual one adds up the gap in each column after squaring it, so a difference of seven millimetres of trunk and a difference of a third of a metre of water are added together with nothing standing between them.

distance² = (trunk gap in mm)² + (water gap in m)²

…

The reason is arithmetic rather than bad luck, and it can be written in one line. Rewriting a column in a unit a thousand times smaller multiplies every gap in that column by a thousand, and a squared distance sees the square of that, so the column’s say in the total is multiplied by a million. What decides the balance between two columns is therefore the ratio of their spreads, squared.

girth spread / water spread = 113.652 / 0.172 = 659.21
the same ratio squared = 434,551

One tree is an anecdote, so the same question is put to all twenty-four. Each tree in turn is held out, the fit is rebuilt on the other twenty-three, and the twenty-four predictions are then scored together against the twenty-four true weights. The score is the usual one, where 1 means every prediction was exact and 0 means the fit did no better than answering the orchard’s average weight to every tree, so a negative score means it did worse than that.

The repair is one line, applied to each column on its own. Work out the column’s average and its spread, take the average away from every value, and divide what is left by the spread. Those two numbers are the whole of what a column teaches, and the value that comes out is called the standard score of the value that went in.

z = (value − average) / spread

The spread here is the standard deviation, which is the square root of the average squared distance from the average. The playground at the top of the page is that line drawn twice. The upper line holds trunk girths in millimetres and the lower line holds the same trunks in standard scores, both on rulers that stay put, so a dot that falls off either end is drawn faint at the end it fell off. Drag a trunk about and watch how differently the two lines answer.

Press the nursery row button on the playground. Five saplings were measured at 210, 210, 260, 160 and 310 millimetres, and the whole arithmetic comes out in whole numbers as far as the square root. The average first.

210 + 210 + 260 + 160 + 310 = 1150,   1150 / 5 = 230

400 + 400 + 900 + 4900 + 6400 = 13,000
13,000 / 5 = 2600,   √2600 = 50.990195

−20 / 50.990195 = −0.392232
−20 / 50.990195 = −0.392232
 30 / 50.990195 =  0.588348
−70 / 50.990195 = −1.372813
 80 / 50.990195 =  1.568929

Two things are true of the transformed column whatever went into it. Its average is exactly zero and its spread is exactly one. Both fall out of the definition in a line each, which is worth doing because the second one is where a convention hides.

zᵢ = (xᵢ − m) / s
average(z) = (1/n) Σ (xᵢ − m) / s = (1/s)(m − m) = 0
variance(z) = (1/n) Σ (xᵢ − m)² / s² = s² / s² = 1

The nursery row shows both. Its five standard scores average to 0 and their variance is 1, printed by the playground as an average of 0.000 and a spread of 1.000.

The claim that the standard score removes the unit can be checked rather than believed. Write every trunk in centimetres instead of millimetres, which divides each value by ten, and the average and the spread are divided by ten as well, so the quotient is untouched. Add a thousand millimetres to every trunk and the average moves by exactly a thousand while the spread does not move at all, so again the quotient is untouched.

(a·x + b − average(a·x + b)) / spread(a·x + b) = a(x − average(x)) / (a · spread(x))

A standard score of 2 says that the value is two spreads above what is typical for its column. That is a sentence about the column as much as about the value, and it is the reason the number is worth having. A trunk of 620 millimetres tells you nothing on its own unless you keep orchards; a trunk 2.04 spreads above the average trunk in this orchard tells you at once that it is among the widest here and not absurdly so.

(620 − 387.625) / 113.652 = 2.0446

Millimetres of trunk and metres of water cannot be compared. There is no question that 620 and 0.95 both answer, so asking which of the two readings is the more unusual has no meaning at all while they are in their own units. In standard scores the same question has an answer, because both numbers are now counts of the same kind of thing, namely how far a value is from typical measured in that column’s own spreads.

…

The same change reaches the summary of how two columns move together. Once both are in standard scores their covariance is their correlation, since a correlation is exactly a covariance with each column’s spread divided out; measured on the orchard the two numbers agree to sixteen decimal places, both coming to −0.0826.

The same comparability reaches the fitted weights of a line. A line fitted on the recorded columns puts a weight on trunk girth measured in kilograms per millimetre and a weight on water measured in kilograms per metre, and those two weights cannot be ranked against each other any more than the raw readings could. Fitted on standard scores, both weights are in kilograms per spread of their own column, and the ranking is then a real one.

…

Since a standard score is a count of spreads, it is tempting to reach for the familiar figures, that about two thirds of a column falls within one spread and about nineteen twentieths within two. Those figures belong to the bell curve and not to the standard score, and the orchard shows the difference without being unusual in any way.

There is one statement about standard scores that holds for every column whatever its shape. At most one part in k squared of a column can lie further than k spreads from the average, so at least three quarters of any column at all is within two spreads and at least eight ninths is within three. Both columns clear that comfortably, which they must.

share beyond k spreads ≤ 1 / k²

The split runs along one line, and the line is whether the fit ever adds a quantity from one column to a quantity from another. A squared distance does exactly that, so every fit built on distances is at the mercy of the units, and Part 1 measured how badly. The same sum is inside k-means, whose centres are chosen to minimise it, and inside principal component analysis, whose first direction follows whichever column happens to have the largest variance in whatever unit it was recorded in.

…

A penalised line adds the squared size of every weight to what it is trying to make small, so the weights compete against one number. A column recorded in a large unit needs a small weight to do its work, and a small weight is cheap under the penalty, so the penalty falls unevenly for a reason that has nothing to do with the trees.

A decision tree compares values only within a column. Every question it asks is whether a value falls below a threshold, and the apple trees that go left when asked about 0.665 metres go left when asked about −0.082 spreads, because the threshold was chosen from the same column that was rescaled and moves along with it.

…

The more surprising member of the untouched group is the plain least-squares line, which does add the columns together and still does not care. The reason is that it has a free weight per column and nothing stopping it from choosing whatever weight the unit calls for. Multiply a column by a hundred and the fitted weight is divided by a hundred, the fitted values are identical, and the score does not move.

Standardizing gives every column an equal say, and an equal say is not always the say a column deserves. The orchard makes that concrete, since its trunk column carries almost nothing about the fruit and standardizing hands it half of every distance.

So the honest ordering on this orchard puts the standard score second. It repaired a fit that was worse than a constant and it did not reach what simply knowing which column mattered would have reached. What it bought is that it needed no such knowledge, which is the trade being made whenever a column whose worth is unknown is standardized rather than dropped.

A standard score is not a property of a value. It is a property of a value and a column together, and which column is a real choice when some rows are held back to test on. The rule is that the average and the spread are worked out from the training rows alone and then applied unchanged to everything afterwards, whether that is a held-out tree, a tree planted next season, or a training tree put through again.

held-out score = (held-out value − training average) / training spread

The tempting alternative is to standardize the held-out rows with their own average and their own spread, which looks like the same operation and is a different one. Take the four wettest trees out of the orchard, fit on the twenty that remain, and then read the four both ways.

…

There is a milder version of the same mistake, and it is the common one. Standardize the whole orchard first, then split it into folds and cross-validate. Nothing is refitted on the held-out rows, so the disaster of the last step does not happen, and yet the two numbers were computed with the held-out rows in the sum, so each fold’s columns carry a trace of the tree that fold was about to be tested on.

…

Two of those zeros are exact for a reason worth keeping. A tree and a least-squares line are untouched by any affine rescaling of a column, as Part 4 measured, so which rows the average and the spread were read from cannot possibly reach them. The zeros for the nearest-neighbour fits are not guaranteed by anything; they say that dropping one tree of twenty-four did not move the two numbers enough to change any ranking of distances.

Subtracting an average and dividing by a spread moves a column and rescales it and does nothing else whatever. Every value keeps its place in the order, every gap keeps its size relative to every other gap, and the picture of the column is the same picture with different numbers written under it.

…

Every value in the column is 300. Its average is 300, every distance from that average is exactly zero, and so the spread is exactly zero. The standard score of any of those values is therefore zero divided by zero, which has no value at all. The same holds for a column of one value, since one value is always its own average and so its one distance from it is zero.

Because the quotient does not exist, an implementation has a genuine choice to make rather than a bug to avoid, and the three answers in use cost different things. Refusing tells the caller at once that a column they thought was informative is not, and stops a fit that was about to be built on nothing.

Substituting a spread of one and passing the column through turns it into a constant column of zeros, which most fits ignore and a penalised fit charges nothing for, so the run completes and nobody learns anything about the data. Dropping the column silently changes how many columns come out of the transformation, which breaks anything downstream that matched columns by position.

The boundary between defined and undefined is a single point and the arithmetic sits right up against it. Nine readings each one machine epsilon further along than the last have a spread of about 2.6 × 10⁻¹⁶, which is positive, so every one of the nine has a standard score and they run evenly from −1.55 to 1.55.

…

There is nothing in the numbers that says whether those nine readings differ because the quantity really varies at that scale or because a sum somewhere lost its last bits. Answering the first way spreads nine indistinguishable values across the full width of the ruler and hands a fit nine confident positions built out of rounding. Answering the second way, which some implementations do by treating any spread below a few machine epsilons as zero, leaves a column whose values genuinely differ at that scale looking constant.

Both numbers the transformation uses are sums over every value in the column, so both can be moved a long way by a single value. That is awkward in a particular way here, since the reason for reaching for a standard score is often that one column contains something extreme, and the extreme value is the one deciding what typical means and what a spread is.

…

Notice what did and did not happen. The transformation is still exactly correct, the scores still average zero and still have a spread of one, and the mistyped value is still marked as the odd one, which is genuinely useful. What is lost is every distinction among the twenty-three real trees, which a fit reading these scores can no longer see.

The last thing the transformation does not say is the one its name most invites. A standard score is defined for any column whose spread is positive, and the definition never mentions a bell curve, a symmetry or a shape. Standardizing a column that is nothing like a bell gives a column that is nothing like a bell, with an average of zero and a spread of one.

Two habits follow from the confusion and both are worth breaking. Reading a score of 3 as one in a thousand assumes a bell; on the fifteen readings above, a score of 3 is exceeded by one value in fifteen. And treating a fit as ready for a method that assumes normality because its columns have been standardized skips the step that method actually needed, which is a transformation of the shape rather than of the location and the scale.