Polynomial Features

We have twenty-two people measured three ways, height in centimetres, waist girth in centimetres, and weight in kilograms. The smallest is a child of 104 centimetres weighing 16.1 kilograms and the largest an adult of 191 centimetres weighing 81.3, and everyone in between is arranged along that span. Predicting weight from height is the obvious thing to want, and a straight line is the obvious way to want it.

There is a reason to expect trouble before any fitting happens. A person is roughly a solid of some height and some thickness, and the weight of a solid goes as a length times a girth squared, so weight against height alone should follow something nearer a cube than a line over a range this wide. Set the playground above to degree 1 and the straight line through these people scores 0.9620, which looks like an excellent fit, and it is the number I want to argue with first.

The thing worth looking at is what the fit leaves over. Each person’s miss is their measured weight less the weight the fit gives them, and if the fit has the shape right those misses should be scattered without order, some people above and some below with no relation to how tall they are.

Under the straight line they are not scattered. The two youngest people are above the line by 8.30 and 4.05 kilograms, then the six people from 118 to 149 centimetres are all below it, and the tall end climbs back above. That is a bow, and a bow in the misses is the fit telling us that the line is too straight for these people rather than that these people are noisy.

Add the squared column and the longest stretch on one side falls from six people to five, and the child the line put at 7.80 lands at 14.02. Add the cube as well and the longest stretch is three and the child lands at 16.67 against a measured 16.1.

Here is the sentence the whole page turns on. A linear model is a weighted sum of its columns, and the thing it is linear in is the weights rather than the measurements. Nothing anywhere in the method says that a column has to be something a tape measure produced.

prediction = b₀ + b₁ c₁ + b₂ c₂ + … + b_k c_k

Read that with c₁ standing for height and c₂ standing for height multiplied by itself, and the prediction is a quadratic in height while remaining an ordinary weighted sum of two columns. So we may compute the squared column ourselves, before anything is fitted, hand it over as though it were a measurement, and get a curve out of a method that only ever draws flat things.

weight = b₀ + b₁ × height + b₂ × height²

So we compute one column, height times itself, put it beside height, and fit exactly as before. Nothing in the fitting changes. The same weighted sum is formed, the same squared misses are added up, the same system of equations is solved for the weights, and the only difference is that there are now three numbers to find rather than two.

It is worth seeing the two fits side by side in the playground at the top. At degree 1 the line cuts through the young people and comes out under the tall ones; at degree 3 it follows the same sweep the people do. The 191 centimetre adult is still missed by 8.44 kilograms at degree three, and that is because he really is light for his height, which is a fact about him and not a shortcoming of the curve.

One detail decides whether the expansion is reproducible. Which columns exist, and in what order, has to be settled once, on the people the fit learns from, and then applied unchanged to anybody who arrives later. Otherwise a new person with a slightly different set of measurements would produce a slightly different set of columns, and the weights learned earlier would be multiplying the wrong things.

Nothing is learned from the values here, only from the names, which makes this the lightest of the preparation steps here. Even so it is a step that has to be carried with the model, and the pipelines page is where the consequences of forgetting to carry it are measured.

With one measurement the expansion is just its powers. With two it is more than that, because two columns can be multiplied by each other as well as by themselves, and a term is any product of the two whose exponents add up to no more than the degree. At degree two over height and girth there are five such products.

height,  girth,  height²,  height × girth,  girth²

The table below is exactly what the fit is handed, four people and every column built from them, so each entry can be checked against the two numbers that produced it.

The powers and the products are usually taught together as though they were the same idea, and they answer quite different questions. A power lets a column’s own effect vary nonlinearly. A product lets one column’s effect depend on another column, and no amount of squaring or cubing will give a fit that ability, since each power still contributes its own fixed amount to the answer.

To see it we need people whose two measurements do not move together, so here are twelve arranged deliberately, four girths at each of three heights. Since every girth appears at every height, knowing one measurement tells us nothing at all about the other.

The totals say the same thing more bluntly. On those twelve people the pure powers leave 28.07 as the total of the squared misses and the powers with the products leave 1.83, so one extra column removes 93 percent of what was left, which is the largest reduction in the misses anywhere on this page. The column responsible holds nothing but each person’s height multiplied by their girth.

weight = b₀ + b₁ height + b₂ girth

weight = b₀ + b₁ height + (b₂ + b₃ height) girth

That was a group built so the product would have something to say. Press the other button on the same widget and the twenty-two people come back, and there the two measurements agree with each other at 0.9886, since taller people in this crowd are also thicker. The product column earns almost nothing there. The pure powers leave 98.89 and the powers with the products leave 91.03, a gain of eight percent for a column that was worth a factor of fifteen a moment ago.

The reason is worth stating plainly, because it is easy to read the first measurement as a general claim about products. When girth is nearly a fixed multiple of height, the product of the two is nearly a multiple of height squared, and the squared column is already in the expansion. The product has nothing of its own left to say. So whether the cross terms are worth their columns depends on the data rather than on the method, and the quantity that decides it is how much the two columns already agree.

Everything so far has been what the expansion buys. What it costs is columns, and the count rises faster than most people expect, because it counts the ways of choosing a degree’s worth of factors from the available measurements with repeats allowed.

columns = C(p + d, d) − 1

In that, p is how many measurements were taken and d is the degree, and the one that is subtracted is the term where every exponent is zero, which is the constant and belongs to the fit’s intercept rather than to any column. Each cell below is a count read off an expansion that was actually built.

A count of columns turns into a hard limit as soon as it is put beside a count of people. A fit sets one number per column plus the intercept, and it cannot set more numbers than it has people to set them from, so the growth table is also a table of how many people each expansion demands.

This is the practical reason to keep the degree low that has nothing to do with overfitting. Long before the fit becomes untrustworthy it becomes impossible, and on a wide table it becomes impossible at degree two, since ten measurements at degree two already ask for 66 numbers and most tables of ten measurements do not hold 66 rows to spare.

Suppose there are people enough. Then the degree is a dial that turns freely, and turning it up always improves the fit we can measure on the people we fitted, for a reason that has nothing to do with the data. Every column the higher degree adds is a new one, so the lower degree’s answer is still available to the higher one by setting the new weight to zero. Whatever the fit finds is therefore at least that good.

squared misses at degree d + 1  ≤  squared misses at degree d

So the score on the fitted people cannot fall as the degree rises, whatever the shape of the data, and a number that only ever improves cannot tell us when to stop.

The way to see what the degree is really doing is to keep people back. Seven of the twenty-two are held out of every fit on this page, leaving fifteen to fit on, and both scores are drawn against the degree below. The height column is put on a common scale before it is raised, for reasons Part 6 measures.

The two lines run together and then separate. The score on the fitted people climbs from 0.9620 to 0.9998, quietly and without interruption. The score on the seven held back rises to 0.9762 at degree four, drifts down to 0.9105 by degree nine, jumps back to 0.9785 at ten, and then leaves the picture, reading 0.3641 at degree eleven, −36.35 at twelve and −24,967 at fourteen. A score below zero means the fit is worse than answering everybody with the average weight.

It helps to look at what the high-degree fit is actually doing, which the playground at the top of the page will draw. Take the degree slider past ten and the curve still passes close to every solid dot, since those are the people it was fitted to, and between them it climbs and dives off the top and bottom of the frame. The hollow rings are the people it was not shown, and they sit where the curve is not.

Nothing about that is a failure of arithmetic. A curve with fifteen numbers to set and fifteen points to pass through has exactly one way to do it, and the shape it takes between the points is not constrained by anything at all. Runge’s 1901 paper is about precisely that freedom, and it grows worse towards the ends of the range rather than better.

The wild swings come from very large weights, some strongly positive and some strongly negative, cancelling almost exactly at the people and not at all between them. So the repair is to make large weights expensive, which is what a penalised fit does, and it is the reason the two ideas are usually taught in the same breath.

minimise  Σ (miss)²  +  penalty × Σ b²

Turn the penalty slider on the sweep above, or on the playground, and the third line is the same degrees with the weights shrunk. The effect at the top end is not subtle.

A penalty of one thousandth takes degree fourteen from −24,967.20 to 0.9631, which is a working model built out of a fit that was ruinous a moment earlier. It does not quite reach the 0.9762 that degree four managed without any penalty at all, and the same penalty applied at degree four makes things slightly worse rather than better, 0.9675 against 0.9762. So the shrinkage is insurance. Where the degree was already sensible it costs 0.0087 of the second score, and where it was not it is worth about twenty-five thousand.

It would be dishonest to leave the impression that a curved relationship demands an expansion. A fit made of straight cuts on the raw height column can approximate a curved relationship too, by answering one weight for everybody below some height, another for the band above that, and so on, with no squared column anywhere. It approximates the curve with piecewise constant predictions, and the question is which of the two does better on these people.

On these twenty-two the expansion wins, and by a real margin. Its best setting is degree four at 0.9762 on the held-back seven, and the stepped fit’s best is a depth of three at 0.9502. That is what a smooth relationship should look like, since the steps pay for their flat tops on a curve that never stops rising, and each step spends people on deciding where to cut rather than on the shape.

The other column of the comparison goes the other way. The worst the stepped fit does across its depths is 0.6438, at a single cut, and the worst the expansion does across its degrees is −24,967. Both methods have one dial and both can have it set badly, and only one of them can be set badly enough to produce an answer that is thousands of times worse than saying nothing.

The steps also need no decision about scale, which the last Part measures, and they read a column that has not been touched at all.

Everything the method needs is supplied by the data except one number, and that one number is the whole of it. The degree is not estimated, not implied by the measurements, and not settled by any quantity computed from them. It is chosen before the fit and the fit then answers the question it was asked.

We could hope that the score on held-back people picks it out, which is the best available procedure and is still not a principle. That score is itself a measurement made on a handful of people and carries their noise, so it can prefer one degree to a neighbour for reasons that have nothing to do with either.

On this page’s seven held-back people it reads 0.9785 at degree ten, the highest reading anywhere in the sweep, and 0.3641 at degree eleven, and no property of the relationship between height and weight changes between those two settings.

The searching for a setting page is about that whole difficulty and measures how much the winner’s own score flatters it. What belongs here is only that the degree is exactly the kind of dial that page exists for, and that it has no setting a derivation could supply.

The expansion builds every product up to the degree, and it has no way of knowing what any of the columns are. Multiply a height in centimetres by a girth in centimetres and the result is an area, which is a quantity a person could reason about. Multiply a height by a code that happens to say which of four clinics a person attends, written as 1, 2, 3 and 4, and the result is arithmetic that means nothing whatever.

Both columns are built, both are handed to the fit, and both get a weight.

The table gathers four such cases. Two of them are failures of the mathematics rather than of taste, and they fail in opposite ways, one loudly and one without a sound.

The last difficulty is the one that connects back to the scaling pages, and it is much sharper here than it is there. A height of 191 centimetres cubed is 6,967,871, while the height itself is 191, so the two columns handed to the same fit differ in size by a factor of 36,481. Write the same heights in metres and the cube is 6.97 against 1.91, a factor of 3.6481. The people did not change.

Plain least squares does not feel it, and the widget shows why it is tempting to conclude there is no problem. A change of unit multiplies a column by a constant, a coefficient can divide by that same constant, and the fit lands in exactly the same place, 0.984072 in both readings. The moment a penalty is switched on it feels it severely, because the penalty is a statement about the size of the coefficients and the coefficients have just changed size by four orders of magnitude.

At a penalty of 100 the centimetre fit scores 0.977374 and the metre fit 0.703648, from the same people and the same instruction.

The stepped fit from section 15 reads 0.990159 in centimetres and 0.990159 in metres, identical to the last digit, because it only ever asks which side of a threshold a value falls on and a change of unit moves the value and the threshold together. That is a second advantage of the stepped fit, alongside the floor section 15 measured.

The limits are worth gathering in one place. Each is a fact about what the expansion and the fit behind it can mean rather than a matter of taste, and they have the same shape as one another, a quantity one of the two steps needs and the data cannot supply.