Centring on the Mean
Sometimes it is more useful to know how far a measurement is from the average than how far it is from zero. Subtracting the mean gives us that new reference point.
15 sections · 15 questions · 4 coding challenges
The arrow keys move between sections too.
Where Should We Put the Origin?
BOTIE
In his 1901 work on fitting lines and planes to points, Karl Pearson used the centre of the observations to organise the geometry. This was part of a wider statistical practice of expressing measurements as departures from their mean. The important historical problem was how to describe the shape of a collection of observations without confusing it with their location.
Subtracting the mean moves the origin to the group's centre. A positive value now means above that centre, and a negative value means below it. The distances between people do not change. We have changed the reference point, which makes later calculations about spread and shared variation easier to interpret.
Sources for this history
Move Zero to Somewhere Useful
Suppose we want to describe how someone differs from a measured group. Raw height is measured from zero, far from everyone in the group. The average person may be a more useful reference.
Centring changes the reference point without changing who is taller or how far apart two people are.
Begin with the original ruler
These heights use the ruler's original zero. That zero is not the centre of this particular group. Each position in the list identifies the same person throughout.
Centimetres
- 160
- 165
- 170
- 175
- 180
Subtraction has a geometric purpose
Subtracting the training mean places the origin at the average training observation. Scaling is a separate operation that changes the unit rather than just the starting point.
The detailed examples show which calculations depend on that origin and which remain unchanged.
Explore the full example
Compare the original coordinates with the centred ones. Watch which distances stay fixed when the origin moves to the mean.
…
Part 1. Moving the Zero to the Average Person
1. Where zero sits in the crowd
The crowd is eleven people, five children and six adults, with heights between 118 and 183 centimetres and weights between 24 and 83 kilograms. Every one of those numbers is measured from zero, and a height of zero and a weight of zero describe nobody in the crowd or anywhere else, so the point every number is measured from is 118 centimetres below the shortest person in it.
Centring moves that point to the average person, by taking each column’s mean away from every value in the column and dividing what is left by nothing.
centred value = value − mean of its columnThe crowd, centred
The eleven heights add to 1670 centimetres and the eleven weights to 587 kilograms. Divide each total by the number of people to find the centre of that column.
mean height = 1670 / 11 ≈ 151.82 cm
mean weight = 587 / 11 ≈ 53.36 kgNow subtract the corresponding mean from each measurement. The first child is 147 centimetres tall and weighs 41 kilograms. The tallest adult is 183 centimetres tall and weighs 83 kilograms.
first child: (147 − 1670/11, 41 − 587/11) ≈ (−4.82, −12.36)
tallest adult: (183 − 1670/11, 83 − 587/11) ≈ (31.18, 29.64)Negative values mean below the column average; positive values mean above it. The units remain centimetres and kilograms.
| height | weight | centred height | centred weight |
|---|---|---|---|
| 147 | 41 | −4.82 | −12.36 |
| 156 | 53 | 4.18 | −0.36 |
| 145 | 57 | −6.82 | 3.64 |
| 159 | 57 | 7.18 | 3.64 |
| 162 | 61 | 10.18 | 7.64 |
| 120 | 25 | −31.82 | −28.36 |
| 122 | 28 | −29.82 | −25.36 |
| 118 | 24 | −33.82 | −29.36 |
| 180 | 80 | 28.18 | 26.64 |
| 183 | 83 | 31.18 | 29.64 |
| 178 | 78 | 26.18 | 24.64 |
The centred columns are still in centimetres and kilograms. The scaler learned one number per column and divided every centred value by exactly one.
Keep in mind
2. What the subtraction keeps
Taking the same number away from every height moves the whole column along its ruler and changes nothing about how the heights sit relative to one another. The 145 cm child and the 147 cm child were two centimetres apart before and are two apart after, and so is every other pair; the largest change in the gap between any two people’s heights came out at exactly zero.
The spread survives as well, because spread is measured about the mean and the mean moved with the column, so the variance of the heights is 521.79 square centimetres before centring and after it.
(xᵢ − m) − (xⱼ − m) = xᵢ − xⱼ
variance of (x − m) = variance of xWhat centring removes is the level, the part of every value that the whole crowd shares. The standard score is this step followed by a division by the spread, and doing the two in that order agrees with standardising the recorded columns to within 2.5 × 10⁻¹⁶, a unit or two in the last place of numbers near one, since standardising was going to take the mean away first in any case.
Keep in mind
3. The promise, and the last few bits
A centred column averages exactly zero, and the reason is one line of algebra. The mean is the value that a column’s excesses and shortfalls balance around, so the deviations from it always add to nothing, and a column of deviations therefore has a mean of nothing too.
Σ (xᵢ − m) = Σ xᵢ − n m = n m − n m = 0Zero, in floating point
The centred heights average 5.2 × 10⁻¹⁵ and the centred weights −3.2 × 10⁻¹⁵. The mean height of 151.8181… repeats forever and cannot be written exactly in binary, so each of the eleven subtractions is off in its last bit and the leftovers do not quite cancel. That residue is about one part in 10¹⁶ of the numbers involved, and it is why a test of a centred column compares its mean with zero to a tolerance.
Keep in mind
Part 2. A Direction Measured From the Wrong Place
4. The longest direction from zero points at the crowd
A principal component search asks which direction the people spread along most. It measures, along each candidate direction, how far every person lies from zero, squares those distances and adds them up, and that total is a measure of spread only when zero is the average person. Hand the same search the recorded columns, where zero is a person of no height and no weight, and the direction it returns is the one along which the people lie furthest from that point, which is the direction in which the whole crowd sits.
…
On the crowd, measured from zero
The longest direction from zero runs at 20.09 degrees above the height axis. The arrow from zero to the average person runs at 19.37 degrees, and the cosine between those two is 0.99992, so the search found where the crowd is. The first principal component of the centred columns runs at 42.21 degrees, the direction in which taller people are also heavier, and its cosine with the uncentred answer is 0.926.
On the measured four, whose average person is 170 cm and 68 kg, the uncentred direction is 21.85 degrees against an arrow to the mean at 21.80, while the centred component lies at exactly 45.
Keep in mind
5. Length splits into location and spread
The reason is an identity that holds for any column of numbers. The mean squared distance of the values from zero is their variance plus the square of their mean, so a column far from zero carries most of its squared length as location, and a search for the direction of largest squared length is mostly a search for the location.
mean of x² = variance of x + (mean of x)²
26,849.4 = 952.9 + 25,896.4The crowd’s squared length, split
The average squared distance from zero contains two contributions: variation between people and the position of the average person. For the eleven people, only about 3.5 percent comes from variation. The remaining 96.5 percent comes from where the whole crowd sits relative to zero.
mean squared distance from zero
= sum of column variances + squared length of column means
≈ 952.9 + 25,896.4
≈ 26,849.4 (figures rounded independently)The measured four make the same split with whole numbers.
squared length of the mean = 170² + 68² = 33,524
mean squared distance from zero = 125 + 33,524 = 33,649Keep in mind
6. Move the zero and the answer moves with it
If the uncentred direction were a fact about the people, it would not care where their heights were measured from. Record every height as the distance above a mark on the wall, which changes no gap between any two people, and the uncentred answer turns, while the centred component stays exactly where it was. Drag the height slider on the playground to watch it happen.
| heights measured from | longest direction from zero | its share | first component, centred |
|---|---|---|---|
| 0 cm | 20.09° | 0.9946 | 42.21° |
| 50 cm | 28.59° | 0.9954 | 42.21° |
| 100 cm | 45.32° | 0.9980 | 42.21° |
| 118 cm, the shortest person | 54.79° | 0.9871 | 42.21° |
| 150 cm | 78.91° | 0.8909 | 42.21° |
The mark turns the uncentred answer through almost sixty degrees. At 100 cm it lands about three degrees from the centred one, which is the case where an uncentred analysis looks right by luck.
Keep in mind
Check yourself
Questions on Parts 1 and 2
Question 1 of 5
What does a column keep when it is centred?
More than one of these may hold. Submit when you have them all.
Part 3. Lines Forced Through Zero
7. A line with no intercept
A line predicting weight from height has two numbers, a slope and an intercept, and the intercept is the weight it predicts at a height of zero. Drop the intercept and the line is forced through zero height and zero weight, which on the crowd is 118 centimetres from the shortest person, so the line has to swing to reach it.
with an intercept weight = b₀ + b₁ × height
through zero weight = b₁ × height…
Weight on height, both ways
With an intercept the slope is 0.890 kg per centimetre and the line explains 0.9588 of the variation in weight, on the usual scale where one is perfect and zero is no better than giving everyone the mean weight. Forced through zero, the slope falls to 0.363 and the share explained to 0.6155, because a line from the origin has to split the difference between the children and the adults.
Centre both columns first and the line through zero has a slope of 0.890 again and explains 0.9588 again. Move the zero to 100 cm on the playground and the forced slope becomes 1.007 with 0.9390 explained, better because the point it is forced through is nearer the people.
Keep in mind
8. Centring is what an intercept does
The centred line through zero and the line with an intercept agreed on the slope, and that agreement is an identity. The least-squares intercept always sends the line through the average person, so fitting an intercept is the same as centring both columns and fitting through zero, and the intercept is what appears when the centring is undone.
b₀ = mean weight − b₁ × mean height
−81.77 = 53.36 − 0.8901 × 151.82What the intercept means, twice
On the recorded heights the intercept is −81.77 kg, the weight the line gives a person of no height, an extrapolation 118 centimetres past the shortest child that describes nobody. On centred heights the intercept is 53.36 kg, the weight the line gives the average person, which is the mean weight exactly. The slope is 0.8901 in both, so the two are one line with the zero of its height axis in two places. On the measured four the same pair is a slope of 0.6 with intercepts of −34 and 68.
Keep in mind
9. A penalty that reaches the intercept
A ridge line adds the squared size of its slope to what it minimises, and leaves the intercept out of that penalty on purpose, since shrinking the intercept would pull the line towards a weight of zero at a height of zero, the very point step 7 forced it through. Because the intercept is exempt, a ridge line with an intercept is exactly a ridge line through zero on centred columns, at any penalty.
| penalty | slope, intercept exempt | slope, centred and through zero | slope, recorded and through zero | explained, recorded and through zero |
|---|---|---|---|---|
| 1 | 0.88992 | 0.88992 | 0.36342 | 0.6155 |
| 100 | 0.87484 | 0.87484 | 0.36328 | 0.6155 |
The first two slopes agree to within 2 × 10⁻¹⁵ at both penalties. The third is an intercept penalised without limit, held at zero whatever it costs.
The ridge fitted here always exempts its intercept, so a penalty of some middling size on the intercept cannot be shown with it. What can be shown is the far end, an intercept held at zero, which is the line of step 7 with a penalty added, and its slope barely moves as the penalty rises from 1 to 100 because a slope of 0.36 is cheap to hold. The damage there was done by the missing intercept, and the penalty adds almost nothing to it.
Keep in mind
Part 4. How Centring Changes a Walk
10. Two columns pointing almost the same way
A least-squares fit with an intercept treats the intercept as the weight on a column of ones, one for every person, fitted beside the height column. On the recorded heights those two columns point almost the same way, since every height is somewhere near 150 times one, and the cosine between them is 0.9889. Centring makes the height column add to zero, which is exactly the condition for it to be at right angles to the column of ones, and the cosine becomes 2.3 × 10⁻¹⁶.
cosine(ones, height) = Σ heightᵢ / (√n × √(Σ heightᵢ²))
Σ (heightᵢ − m) = 0, so cosine(ones, centred height) = 0Two columns that point the same way make the bowl a gradient walk descends into long and narrow. Along its two natural directions its curvatures are 0.0221 and 23,571.5 on the recorded heights, a condition number of 1,064,838, and 1 and 521.8 on centred heights, a condition number of 521.8.
›Why this works
The curvature of the mean squared error is twice the mean of the outer products of the design’s rows. For a column of ones beside a height column that is twice the matrix below, with m the mean height and v its variance, and its determinant is v whatever m is.
[ 1 m ]
[ m m² + v ]Write the two curvatures as c₁ and c₂, the mean height as m, and the height variance as v. Their sum and product explain why a large mean creates a narrow valley.
c₁c₂ = v
c₁ + c₂ = 1 + m² + vAs the mean grows, one curvature grows roughly with its square while the other shrinks. On this crowd the sum is 23,571.6 and the product is 521.8. Centring sets the mean to zero, leaving a diagonal matrix with curvatures one and v.
Keep in mind
11. The walk, pass by pass
Here is what the two bowls do to a walk. Each walk steps at half the rate that would make it diverge, the rule the feature scaling page gave every walk it compared, and each is capped at twenty thousand passes. The chart plots how much of the way to the least-squares line the slope and the level have come.
…
Twenty thousand passes against 8415
Centering separates the mean level from variation in height, making the two parameter directions easier to optimize independently in this example.
For centered heights, the first update reaches a slope of 0.89008. The learning rate was chosen for the steep direction, so that component of the quadratic error is corrected immediately. The level changes more slowly and reaches 53.36 when the run stops at pass 8415.
For the original heights, the steep direction is close to the through-zero line. The first slope is 0.3634, but later updates make slow progress along the remaining shallow direction. After twenty thousand passes, the slope is 0.3732 and the level is −1.52, compared with the fitted solution's 0.8901 and −81.77.
The run reaches its update limit before converging. Centering has not changed the regression task; it has changed the parameter geometry encountered by this particular update procedure.
Keep in mind
12. A column and its own square
A curve through the crowd uses height and height squared as two columns, and on the recorded heights those two are nearly the same column, since between 118 and 183 cm a square rises almost in a straight line with the height. Their correlation is 0.9980. Centre the heights before squaring and the square measures distance from the average person in either direction, which is a different shape from the height, and the correlation falls to −0.229.
| heights | correlation with the square | condition number | weight on height | weight on the square | explained |
|---|---|---|---|---|---|
| recorded | 0.9980 | 1,245,454 | 0.0652 | 0.0027573 | 0.96213 |
| centred | −0.229 | 1073 | 0.9024 | 0.0027573 | 0.96213 |
The curve is the same curve both times, with the same weight on the square and the same share explained. The weight on height changed meaning, from the slope of the curve at a height of zero to its slope at the average height.
Keep in mind
Check yourself
Questions on Parts 3 and 4
Question 1 of 5
Forced through zero on the recorded columns, the slope falls from 0.890 to 0.363 and the share explained from 0.9588 to 0.6155. What happens if both columns are centred first?
Part 5. Angles From the Average Person
13. Every person points the same way from zero
The cosine between two people treats each of them as an arrow from zero and asks how nearly the two arrows point the same way, whatever their lengths. Measured from zero height and zero weight, every arrow in the crowd points up and to the right at a shallow angle, the children’s as much as the adults’, so every pair looks alike; the smallest cosine between any two people is 0.9748, between the 118 cm child and the 183 cm adult.
cosine(a, b) = (a · b) / (‖a‖ ‖b‖)…
The smallest child and the tallest adult
From zero, the child at (118, 24) and the adult at (183, 83) point only 12.9 degrees apart. Both arrows have positive height and weight coordinates. Their weight-to-height ratios are:
child: 24 / 118 ≈ 0.203
adult: 83 / 183 ≈ 0.454From the average person, their coordinates instead become approximately (−33.82, −29.36) and (31.18, 29.64). Those arrows point almost opposite ways: their cosine is −0.9990 and their angle is 177.4 degrees. The most opposite pair after centring is the 120 centimetre child and the 178 centimetre adult, with cosine −0.9996.
Keep in mind
14. Correlation is a cosine after centring
The same move made on the columns rather than the people gives a familiar number. The cosine between the height column and the weight column, each taken as an arrow of eleven numbers, is 0.97439 on the recorded values. Centre both and it is 0.97919, and the correlation between height and weight is 0.97919 as well, because a correlation is defined as exactly that cosine.
correlation(x, y) = Σ (xᵢ − x̄)(yᵢ − ȳ) / √(Σ (xᵢ − x̄)² × Σ (yᵢ − ȳ)²)
= cosine(x − x̄, y − ȳ)Keep in mind
15. Nearness by angle, before and after
Now put the two angles to work in a vote. Each person in turn is held out, their three nearest among the other ten vote on whether they are a child or an adult, and the votes are counted right or wrong. The widget in step 13 shows the tally under its two panels. By angle from zero 10 of the 11 votes are right, by angle from the average person all 11 are, and by straight-line distance 9 of 11 whether the columns were centred or not.
| three nearest chosen by | recorded columns | centred columns |
|---|---|---|
| straight-line distance | 9 of 11 | 9 of 11 |
| angle | 10 of 11 | 11 of 11 |
The straight-line vote cannot tell the two sets of columns apart. The angle vote gave one person a different answer.
Keep in mind
Part 6. The Methods That Never Notice
16. Any gap between two people
A straight-line distance is built from the gap in each column, and a gap is one value minus another, so the mean both values lost cancels out of it. Every method built only on those gaps sees the same crowd before and after centring, and the table below fits each of them twice to show it.
(aᵢ − m) − (bᵢ − m) = aᵢ − bᵢ…
Before and after, measured
The 110 distances between two different people agree to within 5.1 × 10⁻¹⁴ centimetres, the three-person vote by distance scores 9 of 11 both ways, and k-means asked for two groups finds the same two groups with the same total squared distance to their centres, 2952.607, both times.
Keep in mind
17. A threshold, a group and a slope
A decision tree asks whether a height is below a threshold. Centring shifts the threshold by the same amount as every height. The first split on this crowd uses 151.5 centimetres.
centred threshold = 151.5 − 1670/11 ≈ −0.318 cmThe eleven predictions stay the same. A least-squares fit with an intercept also preserves its predictions: the intercept absorbs the shift, as step 8 showed. Its slopes agree between the two coordinate systems to within floating-point rounding.
Keep in mind
Part 7. Where Centring Stops Being Defined
18. Nothing to average, and one value
The mean of a column is its sum divided by its count, and for a column with nothing in it that is zero over zero. There is then no centre to subtract, and whoever is asked for one faces the same three choices a spread of zero forces on the standard score, which are to refuse, to invent a number, or to hand back something that is not a number. The cases around it are better behaved than they look, because centring divides by nothing.
| the column | what the mathematics says |
|---|---|
| an empty column | the mean is 0 / 0, which has no value, so there is no centre and no centred column. Refusing is the one answer that invents nothing; returning zero would put a centre on a column that has none. |
| one value, 170 | defined. The mean is 170 and the centred column is a single 0, which is correct and says nothing about how people differ, since one person is not a crowd. |
| a constant column, 7, 7, 7 | defined, and it centres to 0, 0, 0 exactly. The standard score of the same column is 0 / 0, because it goes on to divide by a spread of zero; centring divides by nothing, so the case that stopped the standard score is ordinary here. |
| a missing or infinite value | the sum is not a number, or is infinite, and so is the mean, and every value in the column is then centred by it, so one bad entry reaches every other value in the column through the mean they share. |
| a column of counts, mostly 0 | defined, and every zero becomes minus the mean, so a column that was mostly zeros has none left. Where zero meant none, that meaning is gone and a sparse column is dense; the scalers that divide without centring exist for this case. |
Keep in mind
19. A mean learned on other rows
When some people are held out to test on, the mean has to come from the training rows alone and then be taken away from the held-out rows unchanged. That is perfectly well defined, and the answer it gives looks wrong at first sight, since the held-out rows no longer average zero.
held-out centred value = held-out value − training meanThe three tallest held out
Without the three tallest adults the other eight have a mean height of 141.125 cm. The held-out heights of 180, 183 and 178 centre on that to 38.875, 41.875 and 36.875, which average 39.21, far from zero and correctly so, since they are three people taller than anyone the mean was learned from. Centred on their own mean of 180.33 they become −0.33, 2.67 and −2.33, which says that one of the three tallest people in the crowd is below average, and the only average in sight is their own.
Keep in mind
20. A mean that is not a place
A mean is always a number, and it is a place only when the column’s numbers are positions along a line. Times of day are positions around a circle. Two bedtimes, eleven at night and one in the morning, written as 23 and 1 hours after midnight, have an arithmetic mean of 12, which is noon, the one time of day neither of them is near, and measured from noon each is 11 hours away.
…
Bedtimes on a clock face
The circular mean draws each time as a point on the rim of a clock face, averages the points, and reads the time off the direction of the average. For 23:00 and 01:00 it is midnight, with each time an hour either side, and the averaged point lies 0.966 of the way out to the rim, which says the two agree closely.
A week of bedtimes between half past ten and one in the morning has an arithmetic mean of 13.36, early afternoon, and a circular mean of 23.64, a little after twenty to midnight. Six in the morning and six in the evening have an arithmetic mean of noon and no circular mean at all, since their points average to the centre of the face, 6 × 10⁻¹⁷ from it, and a point at the centre has no direction.
A column of codes has the same trouble more quietly. Numbering four blood groups 1 to 4 gives the column a mean, and taking it away gives every person a number, but no position on that line is a blood group and the gap from group 1 to group 3 is twice the gap from 1 to 2 only because of the order the numbers were handed out in.
A column of zeros and ones is the exception worth knowing, since its mean is the share of ones, 6 adults of 11 in the crowd or 0.545, and a centred label measures how much more or less adult than the crowd’s average a person is, which is a real quantity.
Keep in mind
Check yourself
Questions on Parts 5 to 7
Question 1 of 5
Measured from zero, the smallest cosine between any two of the eleven people is 0.9748. Why is every pair so alike?
Try it with Botie
Build with Botie
Write and run Python