Simple linear regression

Five people, each measured twice. Everything on this page is built from them.

height (cm):  160   165   170   175   180
weight (kg):   58    66    68    74    74

Each person contributes one input and one observed outcome, and together they make one point on the plot above. The input is written x and the outcome y, so person one is the pair (160, 58).

Which measurement plays which role is a decision rather than a property of the data. Here height predicts weight. Swapping them, predicting height from weight, is a different problem with a different answer, even though the scatter of dots looks exactly the same either way. The picture is symmetric and the question is not.

A line on this plot is a rule for guessing. Pick a height along the bottom, go straight up until you meet the line, and read across to the weight it predicts. Do that for 165 and the line answers; do it for 172 and it answers just as readily, even though nobody in the data is 172 tall.

Written down, that rule is

ŷ = βx + α

The hat on ŷ is doing real work. It separates what the line predicts from what was actually measured, which is written plain as y, and the gap between those two is the whole subject of Part 2.

Two numbers position a line, and they do different jobs.

The slope says how much the prediction changes for each unit of input. A slope of 0.8 means one more centimetre of height raises the predicted weight by 0.8 kg. Its units are the outcome’s over the input’s, kilograms per centimetre, which is worth remembering because it makes the number readable.

The intercept says where the line sits when the input is zero, and its job is mostly to slide the line up and down until it runs through the data. Change the slope in the box above and the line pivots. Change the intercept and it shifts without tilting.

Take the shortest person, 160 cm and 58 kg, and a line that predicts 60 kg for them. It is 2 kg too high, and that miss has a name.

residual = measured − predicted
e = y − ŷ = 58 − 60 = −2

The sign says which way the line missed. A positive residual means the point sits above the line and the prediction was too low. A negative residual means the point sits below and the prediction was too high. Zero means the line went through the point.

One residual per person, so a line produces five of them here. For the line that predicts 60, 64, 68, 72 and 76,

Each row is one grey stalk on the plot. The middle person sits exactly on the line and has no stalk at all.

The obvious way to score a whole line is to add up its residuals, and it does not work.

(−2) + (+2) + 0 + (+2) + (−2) = 0

A total of zero, from a line that is wrong about four of the five people. A line that predicted every weight exactly would also total zero, and no summary that cannot tell those two apart is worth having.

The problem is the signs. Missing by 2 above and 2 below is reported as not missing at all, so what is needed is a way of counting a miss as a miss regardless of direction.

Squaring each residual does that. A negative squared is positive, so nothing cancels, and it has a second effect worth seeing clearly.

Doubling a miss quadruples its contribution. One residual of 4 costs 16, the same as four separate residuals of 2, so squaring says a line would rather be a little wrong about many people than very wrong about one.

Add the squares and a whole line collapses to one number, the residual sum of squares.

RSS = Σ (yᵢ − ŷᵢ)²

The box below draws it. Each square has a residual for its side, so its area is that residual squared, and RSS is the total shaded area.

Once we have chosen residual sum of squares as the criterion, we can compare candidate lines by one number. A smaller value means smaller total squared errors on these observations.

The widget at the top fits the line automatically. Load The measured five to reproduce the example, then drag a point and watch the fitted line and its residuals change. Each new dataset has its own best-fitting coefficients.

Best-fitting means exactly that and nothing more. The line whose RSS is the lowest of all lines.

There are two controls, so the score depends on two numbers at once. Plot RSS against both and the result is a surface, a landscape whose height at any point is how badly that slope and intercept fit.

The surface is a bowl. Every direction away from its floor costs more error, the floor is a single place rather than a ridge, and the best-fitting line is whatever slope and intercept sit at the bottom of it.

Which turns the search into a question calculus already knows how to answer, and Part 6 answers it.

Here is the answer ahead of its derivation, because it is worth being able to read before it is worth being able to prove.

β = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²

α = ȳ − β·x̄

The bars mean averages, so x̄ is the mean height. Read the slope in three pieces. The top asks whether people above average in height tend also to be above average in weight, since it multiplies the two deviations together for each person and adds. The bottom asks how much height varies on its own. Their ratio is how much of the joint movement to credit to each centimetre.

The intercept is then whatever value puts the line through the point (x̄, ȳ), the centre of the data, which every least-squares line passes through.

The whole fit is one table and two divisions. Means first, then deviations, then the two columns that get summed.

x̄ = 850 / 5 = 170        ȳ = 340 / 5 = 68

β = 200 / 250 = 0.8
α = 68 − 0.8 × 170 = 68 − 136 = −68

ŷ = 0.8x − 68

The slope is 0.8 kilograms per centimetre. Within the range these people cover, from 160 to 180 cm, one extra centimetre of height goes with 0.8 kg more predicted weight.

“Goes with” is deliberate. The line says these two measurements move together in this data, and it says nothing about what would happen to somebody’s weight if their height changed.

The intercept is −68 kg, which is the predicted weight of a person zero centimetres tall. That is not a claim about anything. Zero is far outside the range of anybody measured, and the intercept’s job here is to position the line rather than to mean something on its own.

Run every height through the fitted line and the rest follows.

RSS = 4 + 4 + 0 + 4 + 4 = 16

Every dataset has a least-squares line. A cloud of points with no pattern in it has one, and the arithmetic produces it just as confidently as it produced ours.

So “this is the best line” and “this line is worth using” are separate claims, and RSS answers only the first. Press the no-pattern button in the box at the top and the fit still arrives. What is missing there is not optimisation, it is a relationship to find.

Answering the second claim needs something to compare against.

The simplest possible model ignores height entirely and guesses the same weight for everybody. If it has to be one number, the mean weight of 68 kg is the best single number available, since the mean is precisely the constant with the smallest squared error.

Drawn on the plot that is a flat horizontal line at 68. It uses none of the information in the input, so any line worth fitting ought to beat it.

Score that baseline the same way any line is scored, by the sum of its squared misses.

weight − mean:   −10    −2     0     6     6
squared:         100     4     0    36    36

TSS = 176

That total has its own name, the total sum of squares, and two readings. It is the squared error of the mean-only baseline, and it is a measure of how much the weights vary at all. Both readings are the same number, which is what makes it the right thing to compare against, since it is how much variation was there for a line to account for.

Now the two numbers sit side by side. The baseline scored 176 and the fitted line scored 16, so the line removed 160 of the 176.

R² = 1 − RSS / TSS
   = 1 − 16 / 176
   = 1 − 0.0909
   = 0.909

Read the ratio in two steps. RSS over TSS is the fraction of the baseline’s error still left after fitting, about 9%. One minus that is the fraction the line accounted for, about 91%.

Stated carefully, the number says this. The fitted linear relationship with height accounts for about 91% of the variation in weight among these five measurements.

Three things it does not say, and each is a mistake worth not making. It does not say height causes weight, since the same 0.909 would appear if a third thing drove both. It does not promise accuracy outside the range of 160 to 180 cm, where no measurement has tested whether the relationship continues. And it does not say a line was the right shape to fit, only that this line beat the flat one.

Part 3 turned the search into finding the bottom of a bowl, and the bottom of a bowl is where the surface is flat in every direction. There are two directions here, one per control, so there are two derivatives to set to zero.

Written with both controls visible, the score is

RSS(α, β) = Σ (yᵢ − βxᵢ − α)²

Differentiate with respect to the intercept first.

The third line is worth a second look, because it says the residuals add to zero. That was not imposed on the model; it fell out of asking for the flattest point in the intercept direction, which is why section 13 could rely on it.

Now the other direction.

Σ x̄(yᵢ − ȳ) = x̄ · Σ(yᵢ − ȳ) = x̄ · 0 = 0

Setting derivatives to zero finds a flat place, and flat places include the tops of hills. One more argument is needed to know which this is.

RSS is a sum of squares of expressions that are straight lines in α and β. Squaring a straight line gives an upward-curving parabola, and adding upward-curving things gives something upward-curving, so the surface is a bowl everywhere rather than only near the answer. A bowl has no hilltops, so the one flat place is the lowest one.

The bowl is genuinely bowl-shaped rather than a trough only when the heights actually vary, which is the condition Part 7 is about. The surface in the box above is the picture of this argument.

Suppose all five people are exactly 170 cm tall, with the weights unchanged. Every height deviation is then zero.

xᵢ − x̄ = 0  for every person
Σ(xᵢ − x̄)² = 0

The slope formula divides by that, so it has no answer. It is worth being precise about what has actually failed, because “there is no line” is not quite right.

A prediction at 170 cm is still perfectly available, and it is the mean weight of 68 kg, which is the best constant guess as always. What cannot be determined is how the prediction should change as height changes, because height never changed. Infinitely many slope and intercept pairs pass through (170, 68), every one of them fits the data equally well, and nothing in the data prefers any of them.

Drag one dot in the box at the top well away from the others and watch the line lean toward it. That pull is the squaring from section 7 doing exactly what it was chosen to do, since a residual of 10 contributes a hundred where a residual of 2 contributes four.

Whether that is the right behaviour depends on what the unusual point is, and the data cannot say. It might be a genuine rare observation the model should accommodate, a measurement or entry error, someone from a different population than the rest, or a sign that a variable nobody recorded is doing the work.

Those call for different responses, and only the second is a case for removing the point. Deleting whatever sits far from the line is a way of guaranteeing a model that fits, which is not the same as one that is right.

The people measured here run from 160 to 180 cm. A prediction inside that range interpolates, and the data has something to say about it. A prediction well outside it extrapolates, and the data has not been asked.

The line does not know the difference. It continues in both directions forever, which is how it produced an intercept of −68 kg at a height of zero, and that number is what the arithmetic says rather than what anyone should believe.

So the strange intercept is not an oddity to explain away. It is the clearest available demonstration of what a fitted line does outside the range that produced it.

Two last boundaries, both about what the fit is evidence for.

A regression quantifies an association. Height and weight move together in this data and the slope says by how much, and none of that establishes that one produces the other. Other things affect both, and the model has no way to distinguish a direct influence from a shared cause. The slope is not a prediction about what would happen if somebody grew taller.

And the R² of 0.909 describes how well the line fits the five people used to build it. That is not the same question as how well it predicts a sixth person, and a model can score well on the first while doing poorly on the second.