Feature Scaling

The crowd is eleven people, five children and six adults, with heights between 118 and 183 centimetres and weights between 24 and 83 kilograms. A model that asks how far apart two people are adds up the gap in each column, squared, and the height gap is counted in centimetres while the weight gap is counted in kilograms, as if a centimetre and a kilogram were the same size. Nothing in the arithmetic knows they are not.

distance² = (height gap)² + (weight gap)²

centimetres: distance² = 3² + 4² = 25
             height share = 9 / 25 = 0.36

millimetres: distance² = 30² + 4² = 916
             height share = 900 / 916 ≈ 0.98

metres:      distance² = 0.03² + 4² = 16.0009
             height share = 0.0009 / 16.0009 ≈ 0.00006

Now let the unit decide a real question. A person of 150 cm and 60 kg is put to the crowd and their three nearest neighbours vote on whether they are a child or an adult. The widget takes that vote four times, with the heights written in millimetres, in centimetres and in metres, and then with both columns standardized so that each is measured in its own spreads. Drag the green query about to find the people the four readings disagree on.

A model fitted by walking downhill feels the unit differently. Predicting weight from height by gradient descent means stepping down a bowl with one direction for the line’s level and one for its slope, and how steeply the bowl curves along the slope direction is set by the mean of the squared centred heights, a number that is 521.8 in centimetres, ten thousand times smaller in metres, and one hundred times larger in millimetres.

The level direction always curves at one. One step size has to serve both directions, so it is set by the steeper one and crawls along the shallower.

curvature along the slope = mean of (height − mean height)²
condition number = steeper curvature / shallower curvature

…

Every correction on this page is one line of arithmetic, applied to each column on its own. Pick a number to call the column’s centre and a number to call its spread, take the centre away from every value, and divide what is left by the spread. What a method learns from a column is those two numbers and nothing else, and what it does to a value is this.

scaled = (value − centre) / spread

The playground at the top of the page is that line drawn five times. The top row holds one column of raw values, and each row beneath it is the same column after one choice of centre and spread, printed at the left. The amber dot is the same value in every row, so the rows are five rulers laid against one number, and the readouts at the right are that number read on five rulers.

What differs between the five methods is only where the centre and the spread are read from, and the whole family fits in a table.

standardize        centre = mean              spread = standard deviation
min-max            centre = smallest value    spread = largest − smallest
max-abs            centre = 0                 spread = largest magnitude
robust             centre = median            spread = third quartile − first quartile
root mean square   centre = 0                 spread = √(mean of the squares)

Each row is an answer to a different worry. Standardizing leaves a column with mean zero and spread one, so a coefficient fitted on it says how much the target moves per standard deviation of that column, which can be compared across columns. Min-max lands everything in the unit interval, which is what an image pipeline or a bounded activation wants.

Max-abs divides without centring, so a zero stays a zero, a sparse column stays sparse and the sign survives. Robust reads the median and the quartiles, which never look at the ends of the column. Root mean square divides by magnitude about zero and keeps the level, on the view that a column’s level is information.

Press the five readings button, five values of 9, 19, 24, 29 and 39, chosen so that every centre and spread comes out as a whole number. The mean first.

9 + 19 + 24 + 29 + 39 = 120,   120 / 5 = 24

225 + 25 + 0 + 25 + 225 = 500,   500 / 5 = 100,   √100 = 10
81 + 361 + 576 + 841 + 1521 = 3380,   3380 / 5 = 676,   √676 = 26

min-max: centre = 9; spread = 39 − 9 = 30
max-abs: centre = 0; spread = 39
robust: centre = 24; spread = 29 − 19 = 10

(39 − 24) / 10 = 1.5
(39 − 9) / 30 = 1.0
39 / 39 = 1.0
(39 − 24) / 10 = 1.5
39 / 26 = 1.5

The readouts at the right say 1.50, 1.00, 1.00, 1.50 and 1.50. Standardize and robust agree here by coincidence rather than by rule, since the sample is symmetric, so its median is its mean, and its two quartiles happen to be exactly one standard deviation apart. The outlier button breaks that coincidence at once, and Part 3 says how.

What a method promises is whatever its map fixes, and standardizing promises a mean of exactly zero and a variance of exactly one. Three lines show why. Write m for the mean and s for the population standard deviation, so that s² is the mean of the squared deviations, and put every value through the map.

zᵢ = (xᵢ − m) / s
mean(z) = (1/n) Σ (xᵢ − m) / s = (1/s) · (m − m) = 0
var(z)  = (1/n) Σ (xᵢ − m)² / s² = s² / s² = 1

The same move reads the promise off every other row of the table. Min-max sends the smallest training value to 0 and the largest to 1 and fixes nothing in between. Max-abs sends the value of largest magnitude to 1 or −1 and leaves zero where it was. Robust sends the two quartiles to points exactly one apart, at whatever position the median leaves them, so the middle half of the column always spans one unit.

Root mean square makes the mean of the squared scaled values one, since it divided by the square root of exactly that mean.

Press the outlier button on the playground and the 39 becomes 99. The mean jumps from 24 to 36, the min-max spread from 30 to 90 and the max-abs spread from 39 to 99, so under those two methods the four ordinary values are squashed into the first third of the ruler, the 29 landing at 0.22 under min-max and at 0.29 under max-abs. The standardize row shifts as well, since both the mean and the standard deviation feel the new value.

The robust row still prints a centre of 24 and a spread of 10. The median and the quartiles are positions in the sorted list, and replacing the largest value with a larger one changes no position, so the 99 lands at 7.5, far off the end of that ruler, while the other four are precisely where the five readings left them.

(99 − 24) / 10 = 7.5

One method is honest about the outlier by moving everything. Another is honest about it by moving nothing except the outlier. Which honesty you want depends on whether the wild value is a measurement or a mistake, and a scaler cannot know which, so the choice is yours and the page can only show what each choice costs.

A quarter of the way along a sorted list is rarely a whole position, so a rule is needed for what the first quartile of eleven values means. The rule followed here, and by most others, reads the quartiles at a quarter, a half and three quarters of the last index, interpolating between two neighbours when the position is not whole.

Five values have a last index of four, so the cuts fall at 1, 2 and 3, exactly the second, third and fourth values. Eleven values have a last index of ten, so they fall at 2.5, 5 and 7.5.

…

lower quartile = (122 + 145) / 2 = 133.5
median = 156
upper quartile = (162 + 178) / 2 = 170
interquartile range = 170 − 133.5 = 36.5

The outlier the crowd is likely to meet is a mistake of units, one person’s height entered as 1830 because the form said millimetres. The widget puts the crowd’s eleven heights through the five rulers as measured, then with the tallest person typed as 1830, then with a middle person, the 147, typed as 1470. The amber dot is the 156 cm person, the median, whom each method ought to leave alone.

The last row of the playground looks like standardizing with a different spread, and on a column centred at zero that is exactly what it is, since the root mean square of a zero-mean column is its standard deviation. The crowd’s heights are nowhere near zero. Their root mean square is 153.5, almost the mean of 151.8, because when every value is about 150 the magnitude about zero is about 150.

root mean square = √(mean of height²) = 153.5
standard deviation = √(mean of (height − 151.8)²) = 22.8

A scaler is fitted like any model, and what it learns is its centre and spread, read from the rows it was shown. Those two numbers are then reused, unchanged, on every row that comes later, whether that row is a held-out person, a new person next year, or the same training row put through again. The widget below lets you hold people out of the crowd and watch what the remaining rows teach the scaler and where the held-out rows then land.

The tempting alternative is to standardize the held-out rows with their own mean and deviation, and the widget’s second number line does exactly that. The three tallest people, standardized against each other, come out at −0.16, 1.30 and −1.14, which announces that one of them is shorter than average, and the only average they were measured against was their own. The scaler has been refitted on rows the model never saw, which is the leak, and the numbers it produces are on a different scale from the one anything downstream learned.

predicted weight ≈ 43.3 + 14.0 × standardized height

A scaler learns one centre and spread per named column, and a transform matches the columns it is handed by name and never by position, so features may arrive in any order. Two rules follow, and they differ from the rule a model’s predict follows. A scaler fitted on height and weight will transform height alone, since scaling one column of a held-out set is a legitimate thing to want, and measured, a one-column request came back with one column.

It refuses a column called age that it never saw, because a column the fit never saw has no centre and no spread, and inventing one would answer a question nobody asked.

Because the map is a subtraction and a division, undoing it is a multiplication and an addition, and every scaler here has an inverse that reads an answer back in the units the data came in.

value = scaled × spread + centre

…

On the crowd’s heights the round trip is exact to the last bit for standardizing, min-max and robust, and off by 14 and 28 quadrillionths of a centimetre for max-abs and root mean square, which is one or two units in the last place of a number near 150 and is what dividing and then multiplying by the same number costs. The same arithmetic translates a coefficient. A line fitted on standardized height has a slope in kilograms per standard deviation, and dividing by the deviation gives kilograms per centimetre.

slope in cm = slope in z / spread
intercept in cm = intercept in z − slope in cm × centre

slope in cm = standardized slope / height standard deviation
            ≈ 20.33 / 22.84 ≈ 0.8901 kg/cm
intercept in cm = mean weight − slope in cm × mean height
                ≈ 53.36 − 0.8901 × 151.82 ≈ −81.77 kg

Part 1 measured it on a nearest-neighbour vote, and the same sum of squared column gaps is inside k-means, whose centres are chosen by it, and inside principal component analysis, whose first direction follows whichever column has the largest variance in whatever unit it was recorded in. Every one of these reads a column’s unit as its importance, because the unit is a factor on every gap in that column and the square of a factor on every squared gap.

Step 3 measured it on gradient descent. Two passes standardized, 374 in metres, 8415 in centimetres, and an unfinished walk in millimetres, all to the same line, and the difference was the shape of the bowl. A network trained by the same descent has this problem at every layer, which is why the normalisation layers exist, standardizing inside the network what feature scaling standardizes at its door.

Ridge adds the squared size of every coefficient to what it minimises, and the size of a coefficient depends on the unit its column is in. A column recorded in metres needs a coefficient one hundred times larger than the same column in centimetres to do the same work, so its squared penalty is ten thousand times as large, and the fit reports that accident of bookkeeping as a fact about the data.

…

A decision tree never notices. Every split asks whether a value is above a threshold, and the rows that go left when asked about 151.5 centimetres go left when asked about 1515 millimetres, because any rescaling that keeps the order moves the threshold along with the values.

…

Plain least squares is equivariant to the unit, and the word means that the answer changes in exactly the way the unit did. Measure height in metres instead of centimetres and the slope is multiplied by one hundred, the intercept, the fitted weights, the residuals and the R² are untouched, and the line is the same line.

…

A complete implementation states how the centre and the spread are read, including the population or sample divisor for a standard deviation and the interpolation rule for a quartile; that the fit reads the training rows only and the transform reuses those numbers; how columns are matched, which is by name here, and whether a subset is accepted; what a zero spread does, which is refuse rather than substitute a one; that an inverse exists and what its round trip costs; and what happens to a value outside the fitted range, which is that it is scaled by the same map and lands wherever that puts it.

Every row below was tried rather than reasoned about, and the widget under the table tries the same list again on every load, so the table cannot go stale without the widget saying so.

…

The table exposes four separate numerical decisions. We should understand what each one allows before treating every accepted column as suitable for modeling.

1. A very small positive spread is accepted. Scaling can therefore magnify tiny differences, including differences caused by rounding or measurement noise.
2. A constant nonzero column can pass through max-absolute or RMS scaling because the divisor is nonzero. It remains constant afterward and still provides no variation.
3. Robust scaling is refused when the interquartile range is zero. A column can contain some differing observations while its middle half still has no measurable spread.
4. A new value outside the fitted range is transformed without clipping. It can therefore appear outside the training output range, preserving evidence that it differs from the fitted examples.