Ridge & Lasso

The box above is the polynomial page’s problem, the same noisy throw fitted at degree 9, far more flexibility than a ball deserves. Set the penalty slider to its lowest and the fit answers only to the data. It scores 0.997 on the fifteen readings, and on the polynomial page the same curve scored −5479 on four readings held back from it. The training score and the held-out score are not measuring the same thing, and the whole of this page follows from the gap between them.

Look at the bars under the curve rather than the curve. Each is one of the nine coefficients. They are what the fit actually chose, and everything the curve does between the readings is built out of them.

Bars alone do not show how the terms interact, so pick one moment in time and take the prediction there apart. Each term contributes its coefficient times its column, positives stack upward and negatives stack downward, and the prediction is where the stack ends.

A moderate prediction produced by large opposing pieces is sensitive to the smallest change in any of them. Nudge the t⁶ coefficient by a hundredth of a percent and the prediction moves by more than four metres. That sensitivity is the problem regularisation is for, and it is worth seeing before any penalty is mentioned.

Coefficient size matters because of what it does when the data changes. Below, the same degree-9 model is fitted three times, on the noisy throw, on the same readings with one of them moved a metre and a half, and on a fresh set of readings of the same throw with its own noise.

Small change in the data, large change in the coefficients, large change in the curve between the readings. That is what an unstable model is, and it is a stronger case for restraint than the curve looking implausible, because it says the fit will not survive contact with the next sample.

There are two things to want from a fit. It should match the observed data, and it should not depend on extreme coefficients to do it. Section 3 offers exactly that choice. Sample A with no penalty scores 0.997 with a largest coefficient of 85,263. At λ = 1 it scores 0.846 with a largest coefficient of 7.64.

So the question is not whether the second model fits worse. It does. The question is how much training error you would accept in exchange for a model that does not fall apart when one reading moves, and that is a trade rather than a correction. Regularisation makes the trade deliberately.

Write the trade down as one number to minimise, made of two parts.

total objective = data error + coefficient cost
total objective = RSS + λ · penalty

RSS is the residual sum of squares from every regression page so far, and asks how well the model fits the readings. The penalty asks how costly the coefficients are. λ sets the relative price of the two. Below, the objective is drawn at every λ on the path in its two halves, grey for RSS and colour for λ times the penalty, and the fit at each λ is the set of coefficients that made the whole bar shortest.

Regularization minimizes a combined objective. At penalty one, this fit accepts a larger residual sum of squares in exchange for smaller coefficients.

combined objective ≈ residual cost + penalty cost
                   ≈ 101 + 79 = 180

The unpenalized fit’s residual cost was only about 1.7, but its coefficient penalty would make its combined objective larger. Compare complete objectives when judging which solution the regularized fit should choose.

λ does not set a coefficient. It changes the trade that decides them all, which is why sliding it moves every bar in the box at the top of the page at once. The path above is drawn on a log scale because the useful values run from a thousandth to ten, and a linear slider would spend most of its length past the point where anything interesting happens.

The regression page’s slope is 0.8 kilograms per centimetre. Expressing height in metres makes the slope eighty kilograms per metre. The height contribution to the prediction stays the same.

in centimetres: 0.8 × 170 = 136
in metres:      80 × 1.70 = 136

coefficient ratio = 80 / 0.8 = 100
squared-penalty ratio = 100² = 10,000

The predictions are unchanged, but the penalty is not. This is why the units of a feature matter to regularization.

A large coefficient can mean a feature matters a great deal, or it can mean the feature was measured in small units. Penalising raw coefficients charges for units rather than for complexity, and the choice of units ends up choosing the model.

The repair is the statistics primer’s z-score, applied to every feature column before the fit.

zⱼ = (xⱼ − x̄ⱼ) / sⱼ

Subtracting the mean centres the column. Dividing by its standard deviation puts every column on a common spread, so a coefficient of 2 means the same thing whichever column it sits on, and the penalty can compare them. Every fit on this page does this. The bars in the box at the top are coefficients on standardized columns, which is why t⁹, whose raw values reach 262,144, can be drawn beside t.

Centring also settles the intercept. On centred features the intercept is the prediction at the average of every input, which for the throw is the mean height, 12.76 metres. That number sets the overall level of the predictions rather than how they respond to any feature, and penalising it would pull every prediction toward zero rather than toward the data. So the coefficients are charged and the intercept is not, on both of this page’s models.

Ridge charges the sum of the squared coefficients.

RSS + λ · Σ βⱼ²

Before combining it with anything, look at what β² costs for one coefficient. Zero costs nothing, 1 costs 1, 2 costs 4, 4 costs 16. Moving away from zero becomes increasingly expensive, a positive and a negative coefficient of the same size cost the same, and the curve is smooth everywhere, including at zero.

The slope readout is the part to notice now and use later. The slope of β² is 2β, which at zero is zero. Near zero, ridge barely pushes at all, and that is why its coefficients approach zero without arriving. The other panel is Part 5.

For one feature the ridge answer can be worked out completely, and the conditions are worth stating because the formula depends on them. One feature. Feature and target both centred, so the intercept is handled separately and drops out. And the objective written as RSS, not a mean and not halved.

Σ (yᵢ − βxᵢ)² + λβ²

Ordinary least squares on centred data gives the slope as a ratio of two sums, and ridge changes exactly one thing about it.

β = Σxᵢyᵢ / Σxᵢ²          ordinary
β = Σxᵢyᵢ / (Σxᵢ² + λ)    ridge

For the five-person dataset, the centred cross-product is 200 and the centred sum of height squares is 250.

ridge slope = 200 / (250 + λ)

Increasing the penalty increases the denominator and shrinks this slope toward zero.

The whole objective is drawn below over the slope, RSS as a parabola with its floor at 0.8, the penalty as a second parabola with its floor at zero, and their sum. Slide λ and the sum’s lowest point slides toward zero.

The indigo curve on the shrinkage path below is that formula fitted at every penalty from 0 to 500. The amber curve is Part 5.

With several features the coefficients are fitted together, and the one-feature picture needs care to generalise. What ridge does is discourage large overall coefficient magnitude. It generally keeps every feature with a nonzero coefficient, and it steadies the fit when features are correlated, which is the case Hoerl built it for. What it does not do is shrink every coefficient at the same rate, or shrink the noisiest first, or hold an opinion about which features deserve a say.

The path below is every one of the nine polynomial coefficients against log λ, and they cross each other, change sign and shrink at nine different speeds.

The right-hand panel is the point of the exercise. The training score only ever falls as λ rises, since every penalty is a constraint the free fit did not have. The held-out score, on four readings the fit never saw, climbs from ruin at no penalty to 0.993 near λ = 0.0001 and only then begins to fall. Restraint buys prediction, up to a point, and Part 7 is about finding the point.

Why correlated features are steadied is a geometric fact and deserves its own picture. Below, two columns track each other, the second close to twice the first, and the residual sum of squares is drawn over every pair of coefficients.

…

Notice also where ridge went. Not toward the origin along the line from the least-squares point, but across, to a pair where both coefficients carry some of the effect. Two large opposing coefficients cost a great deal in β², and two moderate shared ones cost less while predicting nearly the same, so ridge picks a smaller and steadier pair from among the many that fit similarly. That is the whole of what “spreading credit across correlated inputs” means, and it is a consequence of the circle rather than a belief about features.

Lasso charges the sum of the absolute values instead.

RSS + λ · Σ |βⱼ|

For one coefficient the cost is 0, 1, 2, 4 for β of 0, 1, 2, 4, a straight line either side of zero. Go back to section 9’s widget and drag β across both panels. Ridge is a bowl, smooth at the bottom. Lasso is a V with a corner at zero, and its slope is +1 on one side and −1 on the other, the same size everywhere and undefined exactly at the corner.

So lasso pulls toward zero with constant force, however small the coefficient already is, where ridge’s pull fades as the coefficient does. The corner is what makes the difference in the next two sections.

Same conditions as section 10, one centred feature, RSS without a half. Write the two sums as S for short.

Sₓᵧ = Σxᵢyᵢ = 200        Sₓₓ = Σxᵢ² = 250

For a positive slope the lasso answer subtracts from the numerator rather than adding to the denominator.

β = (Sₓᵧ − λ/2) / Sₓₓ     while λ/2 < Sₓᵧ
β = 0                     once λ/2 ≥ Sₓᵧ

In this one-feature example, lasso subtracts half the penalty from the positive data term before dividing by 250. Once the subtraction exhausts that term, the solution is zero.

positive slope = max(0, 200 − λ/2) / 250
zero threshold: λ/2 = 200
                λ = 400

This exact zero is the feature of lasso that the ridge solution does not share in this example.

The half comes from the objective being written as RSS with no factor in front, so that the derivative of the squared term carries a 2 the penalty term does not. Write the objective with a half, or as a mean, and the threshold moves. The mechanism does not.

The amber path in section 10’s shrinkage chart is this formula fitted at every λ, and it walks straight into zero at 400 and stays. Ridge’s indigo path passes 0.31 there and keeps gliding.

Return to section 10’s two panels, which draw the same RSS parabola under both penalties, and slide λ upward. The ridge minimum moves toward zero continuously and sits just beside it however far λ goes. The lasso minimum arrives at zero at λ = 400 and stays there for every λ after.

Lasso can remove a coefficient because a penalty with a corner can overpower a weak data contribution at a finite λ. A penalty that is smooth at zero never can.

On the nine polynomial terms the lasso paths below do what the one-feature arithmetic predicts. Coefficients arrive at the zero line and stay, the count of active terms falls, and the fitted curve keeps its shape on fewer and fewer terms.

Two things on that chart are easy to over-read. The active count is not monotone. It falls from 9 to 6 by λ = 0.056, to 4 by 0.178 and to 2 by 3.2, then rises to 3 at 5.6 before falling back to 2 at 10, because a term that was switched off came back when a correlated neighbour was shrunk and its share of the work needed picking up. And the first term to reach zero is not the least important term in any objective sense. It is the term whose contribution the others could most cheaply absorb at that λ, on this sample.

A model that ends with most coefficients at exactly zero is smaller and easier to inspect, and in a setting with hundreds of candidate features that can be worth more than the predictions. It is a sparse predictive representation, and it is not a list of which features matter.

Everything above used the same data, the same standardization, the same degree-9 columns, the same held-out split and the same λ scale, so the two can be set beside each other fairly.

Neither is the better method. Ridge is the one to reach for when many features each carry a little signal and the problem is instability. Lasso is the one to reach for when the model has to be small enough to read, or when most of the features are believed to be irrelevant. Both are available at once, in the elastic net, which is a share of each penalty.

The clearest account of why one produces zeros and the other does not is section 11’s valley, with the toggle set to lasso. For two coefficients the RSS contours are ellipses. Ridge’s penalty is a circle, and the solution is where the expanding circle first touches the best available contour. Lasso’s penalty is a diamond, |β₁| + |β₂| constant, and its corners lie on the axes, exactly where one coefficient is zero.

An ellipse meeting a circle touches it at a point with no special relationship to the axes, so both coefficients come out nonzero. An ellipse meeting a diamond very often touches it at a corner first, because corners stick out, and a corner is a solution with one coefficient at exactly zero. Slide the λ control with lasso selected and watch the green dot ride the diamond’s edge until it snaps onto the vertical axis at λ ≈ 3, where the first coefficient is zero from then on.

Lasso produces zeros because the shape of its penalty has corners on the coefficient axes, and ridge does not because a circle has none.

The dashboard below is every instrument on one λ. Pick a penalty and read the objective, the coefficients, the fitted curve and the two scores together.

The pattern to see is in the scores panel. Training R² only falls as λ grows. Held-out R² rises first, because the unpenalised fit was memorising the eleven training readings, and then falls, because too much restraint stops the curve from following the arc at all. Under ridge the turn is near λ = 0.0001 on this split, and under lasso near 0.1. Those numbers belong to this data, this standardization and this split, and to nothing else.

The penalty is chosen by performance on data that did not determine the coefficients. Nothing on the training side of the chart can choose it, since that side always prefers λ = 0.

The old rule on this page said the right penalty is past the wiggle and short of the flatline, and the box at the top still shows what that means. Keep it as a picture of the trade and not as a procedure. A curve that looks moderate is not evidence that it will predict well, and two λ values that produce curves you could not tell apart by eye can differ by a tenth in held-out score.

The procedure is the one section 18 previews. Fit several candidate values of λ on the training rows. Score each on held-out rows, or better, let every row take a turn as the judge, which is cross-validation. Choose by that score, and where it matters, confirm the choice once on data that took no part in choosing it. The held-out evaluation and grid search pages do this properly, including the part where the winning score flatters itself.

Both reduce overfitting and they make different choices. Choosing a lower degree removes whole columns from the model. Ridge keeps every column and shrinks its coefficient. Lasso keeps the columns and may zero some, and with raw polynomial powers it can keep t⁵ while dropping t⁴, so what survives is not a tidy polynomial of some lower degree.

Two things in that table are worth a look. Degree 2 is already the right shape for a ball, and a small penalty still helped it, from 0.989 to 0.996. And λ = 1 is a heavy penalty on eleven standardized rows whatever the degree, which is section 6’s caution in numbers.

In the matrix language of the polynomial page, the ridge objective and its solution follow in four lines.

With one centred feature XᵀX is the single number Σx², and the last line is section 10’s formula. With the intercept in play it is either kept out of β and fitted separately on centred data, as this page does, or the identity matrix gets a zero in the intercept’s position so that it alone goes uncharged. And as on the polynomial page, the system is solved directly rather than by forming an inverse.

J(β) = ‖y − Xβ‖² + λ‖β‖₁

The same first step fails on the second term. |β| has no ordinary derivative at zero, so there is no gradient to set to zero and no linear system to solve, and lasso has no normal-equations solution with many features. It does have one with a single standardized feature, which is section 13, and in a few special designs, but nothing that plays the role of (XᵀX + λI)⁻¹.

What it has instead is the corner. At zero the absolute value admits every slope between −λ and +λ, a subgradient rather than a gradient, and zero is optimal whenever the data’s pull on that coefficient lies inside the range. The solver used here is coordinate descent. It sweeps the coefficients one at a time, and for each it holds the others fixed, which reduces the problem to section 13’s one-feature case on the current residual, applies the soft threshold, and moves on.

Repeated until nothing moves, that arrives at the lasso solution, one soft threshold at a time.

After standardization and shrinkage a coefficient is not the number it was on the earlier pages. It is biased toward zero on purpose. Its size depends on the preprocessing and on λ, and it was chosen partly for the stability of the predictions rather than to describe the feature. A ridge coefficient of 0.3 on standardized t⁴ is a statement about what this fit decided at this λ, and not a measurement of how height depends on the fourth power of time.