Pipelines

Take the grid search page’s twelve people, measured by height in centimetres and weight in kilograms, and hold three of them out, the people at 153, 165 and 177 cm. A model that predicts weight from standardized height is fitted on the other nine, and now the first held-out person arrives to be scored. Their height has to be standardized before the model can read it, and the question is what centre and what spread to subtract and divide by.

Their own are no use. One height has no spread at all, so asking to standardize a single person is refused, because the column is constant.

The numbers in the boxes come from the nine training people. Their heights centre at 167.11 cm with a spread of 10.38 cm, and the held-out person at 153 cm becomes −1.360 under those two numbers, at 165 cm −0.203, and at 177 cm 0.953. Press the second button and hold out the three tallest instead.

The centre falls to 162.11 and the spread to 7.67, and the three held-out heights standardize to 1.942, 2.334 and 2.725, more than two spreads above anyone the fit saw, which is the truth about them and is what the scaled numbers should say.

So the centre and spread are learned from the training people and reused on everyone who comes later, which makes them a fitted parameter of the model in every sense that matters. They were read off the training rows, they change when the training rows change, and a prediction cannot be made without them. The middle three and the tallest three give two different standardizers from the same twelve people, because the nine left to train on differ, and each standardizer is the right one for the model fitted beside it.

z = (height − 167.11) / 10.38        the middle three held out
z = (height − 162.11) / 7.67         the three tallest held out

training mean = 1504 / 9 ≈ 167.11 cm
score at 153 cm ≈ (153 − 167.11) / 10.38 ≈ −1.360
score at 177 cm ≈ (177 − 167.11) / 10.38 ≈ 0.953

Written by hand, that is two objects to keep in step, a standardizer and a model, with a rule that the standardizer is fitted exactly where the model is fitted and applied exactly where the model predicts. Every place the model is used is a place the rule can be broken. A fold loop written by hand has to refit the standardizer on each fold’s training rows; a search over settings has to do it for every candidate at every fold; a serving endpoint has to apply the one standardizer that the deployed model was fitted with and never another.

None of that is hard, and each is the kind of rule that is right when the loop is first written and wrong after the third change to it.

The alternative is to make the two objects one. A chain holds its steps and its model together and exposes the same fit and predict as any single model on this site, so wherever a model is fitted the steps are fitted, and wherever a model predicts the fitted steps are applied, because there is nowhere else the fitting or the applying could happen.

The chain on this page has two steps and a model. The first step expands height to height and its square, the second standardizes both columns, and the model is a ridge regression at a penalty of one. Fit runs the steps in that order on the training rows and then fits the model on what the last step produced. Each step is named, the names are unique, and the order the steps are given in is the order they run in.

fit      expanded = expansion fitted on the training rows, applied to them
         scaled   = standardizer fitted on expanded, applied to expanded
         model fitted on (scaled, training weights)
predict  expanded = the fitted expansion, applied to the new rows
         scaled   = the fitted standardizer, applied to expanded
         model.predict(scaled)

A chain with no steps at all is allowed, and it is then the model and nothing more, which is measured rather than assumed. On the nine training people a chain with an empty step list and a bare ridge at the same penalty predict the three held-out weights to a largest difference of exactly zero. Allowing the empty chain is what lets it be the default wrapper rather than a special case a caller has to decide about.

The standardizer in this chain does not learn the centre and spread of height. It learns the centre and spread of whatever the expansion handed it, which is two columns, height and height squared, and the boxes in section 1 show both. On the nine training people the square centres at 28033.78 with a spread of 3467.39, numbers the standardizer could not have known from the raw height, because the column did not exist until the expansion made it.

standardizer, fitted after the expansion, learned
  height    centre 167.11     spread 10.38
  height²   centre 28033.78   spread 3467.39

That is the rule, and at predict time nothing else would work, since the standardizer will be handed the expansion’s output and so has to have been fitted on the expansion’s output; a standardizer fitted on the raw height would meet a column called height squared that it had never learned a scaling for, and refuse it.

Predict is the shorter half. It hands the new rows to the fitted expansion, hands what comes back to the fitted standardizer, and hands what comes back from that to the fitted model. Nothing is fitted anywhere along the way. Measured on the chain in section 1, the standardizer’s centre for height is 167.1111 before a predict on the three held-out people and 167.1111 after it, the same number to the last digit, because predict has no code path that could change it.

That sentence is the whole of the leak protection at serving time. There is no second call site where a transformer could be fitted on the rows being scored, since the only place a transformer is ever fitted is inside fit, and fit is called on the training rows.

The held-out evaluation page’s fold loop calls fit on a fold’s training rows and evaluate on its held-out rows, once per fold, on whatever model it is handed. Hand it the chain and it does exactly that, and since the chain’s fit fits the steps, the standardizer is refitted on each fold’s training rows for no reason other than that fit was called. The loop knows nothing about what the chain contains and needs to know nothing.

Now the mistake the object exists to prevent, written down and then measured. Standardize the columns once over all the people, and only then deal the folds. Every training fold has now been scaled by a centre that its held-out rows helped to compute. With five folds each training fold holds four fifths of the rows, so the centre over everyone is a weighted average of the centre the fold should have used and the centre of the rows it is about to be judged on.

centre over all rows = (4/5) · centre of the training fold + (1/5) · centre of the held-out fold

Whether that matters, and how much, cannot be settled by looking at the formula, so the playground at the top of the page measures it. Twelve hand-typed people are too few to fold thirty times over, so the measurement draws people by seed, thirty, sixty or two hundred of them, measured by height in centimetres, age in years and how far they walk in a day in kilometres, with a weight that is a plain line through the three plus a little noise.

The three columns are on three different scales, so a k-nearest neighbours regressor that skipped the standardizer would let age decide who is near, and the standardizer genuinely has work to do.

Press the button for the first thirty seeds and read the thirty-people column. Fitted on every person and then folded, the standardizer arrangement scores a mean held-out R² of 0.4813; refitted inside each fold it scores 0.4830. The shortcut did not flatter the score at all here, it cost 0.0017, and fourteen of the thirty seeds went the other way. At sixty people the gap is +0.0004 with fifteen of thirty flattered, and at two hundred it is +0.0001 with eighteen of thirty, a coin toss that shrinks as the people multiply.

The strip is the number that matters. A step that leaked something worth having would flatter most seeds, and this one flatters fourteen of thirty and costs the other sixteen, with the dots straddling zero and the largest single-seed gap at thirty people, 0.1019, larger than the mean gap by a factor of sixty. Press the button for seeds 30 to 59 and the mean is −0.0104 with fourteen of thirty flattered again. I have not found a family of seeds on which the standardizer flatters more than about half of them.

Two reasons, and the model is the first. Nearest neighbours reads only distances, and a distance between two standardized people subtracts the same centre from both, so the centre cancels entirely and only the spread survives.

d(a, b)² = Σⱼ ((aⱼ − bⱼ) / sⱼ)²        the centre mⱼ appears in neither term

What leaks, then, is the ratio of each column’s spread over everyone to its spread over the training fold, a number close to one that reweights the three columns by a fraction shrinking like one over the number of people. That is the second reason. Nothing about that ratio knows the weight, so it cannot lean the fit toward the held-out answers any more than away from them, and a leak that is blind to the target can move the score in either direction, which is why fourteen of thirty seeds were flattered and sixteen were not.

The lower panel of the playground fits something deliberately not built here, a step that keeps whichever of twenty noise columns correlates best with a target that is itself noise. No column deserves any score above zero, and a neighbours fit on any one of them should do worse than answering the mean weight every time.

Both arrangements do exactly that. Choosing the column inside each fold scores −0.8867 at thirty people, and choosing it on every person first scores −0.5979, which looks less bad by a margin that has survived every family of seeds I have run.

Feature selection used the target values from the very rows later presented as held out. Among twenty noise features, the strongest chance correlation can look substantial on a small sample.

The resulting score inflation is positive in twenty-seven of thirty seeds in this experiment. It shrinks as the sample grows, but remains visible. A seemingly useful score can therefore arise from leaking answers into feature selection even when the input features contain no predictive information.

It would be easy to read Part 3 as saying the chain matters only for a step that reads the target, and that reading is half right. No step on this page reads the target, and for the steps that are here the leak is within noise of zero. The argument for the object was never the size of the penalty for getting the fold wrong.

It is that the question stops being the caller’s to remember, at every fold, every candidate and every serving call, and that once the steps and the model are one object a setting inside a step becomes something a search can vary. Parts 4 and 5 are about those two things.

Section 7 handed one chain object to four folds in turn and claimed each fold got its own fit. For that to be true the steps and the model a caller builds the chain from must never themselves be fitted, or the second fold would start from whatever the first fold learned. So fit copies them. The copies are fitted and stored, and the originals, which are the configuration, stay exactly as they were built. The ledger reads that claim off a chain fitted on the nine training people.

…

The configured model and both configured steps report themselves unfitted after the fit; the fitted copies report themselves fitted; and the copies are different objects from the configuration, so a change to one cannot reach the other. Refitting the same object on a different nine and comparing it with a chain built fresh for those nine gives predictions that agree to a largest difference of exactly zero, which is what starting from scratch means.

The same check, run inside the fold loop. At each of the four folds the shared object is refitted on that fold’s nine and scored on its three, and a chain built fresh for the fold is fitted and scored beside it. If anything from an earlier fold survived on the shared object the two would differ somewhere; the figure over each bar is the largest difference between their predictions on that fold.

Zero at every fold. Afterwards the shared object is fitted, to the fourth fold it saw, and its configuration is still unfitted, which is the same pair of facts the playground reports about the chain it handed to a hundred and fifty fits per column. Five folds would give five independent fits, and thirty seeds of five folds give a hundred and fifty, for the same reason.

A fit can fail halfway. The expansion may succeed and the standardizer refuse, or both steps may succeed and the model refuse, and a chain that had already stored its new fitted steps by then would be left holding fitted steps from one fit beside a fitted model from another. So the chain computes every part into a result first and stores nothing until the model has fitted too.

The ledger’s seventh and eighth rows break a refit on purpose, handing the fitted chain a constant height column, which the standardizer refuses because a constant column has no spread.

refit on a constant column   refused, the column has zero variance
still fitted afterwards      yes
predictions moved by         0.0

The refusal comes back by name, the chain is still fitted to the nine people it had before, and its predictions on the three held-out people have moved by exactly nothing. A refit that fails leaves the previous fit whole rather than half replaced.

The grid search page varied a neighbour count, which is a field of the neighbours model, and a ridge penalty, which is a field of the ridge model. The degree of the expansion in this chain is neither. It is a field of the expansion step, and a search over the ridge model alone has no way to reach it.

Asked to vary a degree on the ridge model, the search refuses at construction and names the two fields the model does have, its intercept switch and its penalty, and that refusal is quoted under the lattice below.

Wrapped in a chain, the whole configuration is one model whose two fields are its steps and its model, and both can be varied. That is the capability the object buys, and it is stronger than the leak argument, since without it the question of which degree to use could not be put to a search at all.

Each search candidate combines a complete preprocessing sequence with a model configuration. This example crosses four polynomial degrees with four ridge penalties, then evaluates each combination on four shared folds.

candidates = 4 degrees × 4 penalties = 16
cross-validation fits = 16 × 4 folds = 64

Every candidate uses the same seeded folds, so their scores compare configurations on the same partitions.

steps  ∈ { expand to 1 then standardize, expand to 2 then standardize, expand to 3 ..., expand to 4 ... }
model  ∈ { ridge at 0.01, ridge at 0.1, ridge at 1, ridge at 10 }
candidates = 4 × 4 = 16        fits = 16 × 4 folds = 64

On the twelve people the search keeps degree 1 at a penalty of 0.1, with a mean held-out R² of 0.9796, and degree 1 at 0.01 finishes within a few millionths of it. Degree 2 at 0.1 scores 0.9790, degree 2 at 1 scores 0.9758, and every candidate at a penalty of 10 is far behind, degree 1 at 0.6157 and degree 4 at 0.9081.

The spread across the sixteen is 0.3639, which says the settings mattered, though what mattered was the penalty; the degrees at any one small penalty finish within 0.015 of one another, and the expansion earns nothing on twelve people who lie very nearly on a line.

Some pipeline APIs address nested settings with a string combining the step name and parameter name. Such an API can validate those names; silently ignoring a misspelling is an implementation error, not an inherent property of strings.

This SDK builds candidates from configured objects. Its constructors reject unknown fields before the search starts, so an invalid candidate cannot quietly become another copy of the default model.

a range over the chain's field "stepz"     refused at construction, the chain has "model" and "steps"
a range over the ridge model's "degree"    refused at construction, the model has "fit_intercept" and "penalty"

Two steps can run in two orders, and the chain treats the order as part of the model rather than as a detail, so it is worth asking whether the model can tell. Expand height and then standardize both columns, and the ridge is handed two columns each with centre zero and spread one. Standardize height and then square it, and the ridge is handed the standardized height beside its square, whose centre is one and whose spread is something else. The chart runs both orders at six penalties on the same four folds.

At a penalty of zero the two orders score 0.9579 and 0.9579 and predict the three held-out weights to a largest difference of 1.2e−13, which is rounding. At every positive penalty they part, and the chain’s own order scores higher at each one, 0.9764 against 0.9581 at 0.01, 0.9790 against 0.9597 at 0.1, 0.9758 against 0.9523 at 1, and 0.8105 against 0.5539 at 10, with the held-out predictions differing by 0.735 kg at a penalty of one and 2.334 kg at ten.

The two column sets span the same curves. Height and its square, or standardized height and its square, either pair together with an intercept can draw any parabola in height, so a fit that only asks which parabola is closest to the nine training people finds the same one either way. That is what the penalty of zero shows, and a plain least-squares model in place of the ridge gives the same 1.2e−13.

A ridge penalty is a charge on the squared coefficients of whatever columns it was handed, and handing it different columns changes what it is charging for.

expand then standardize, penalty 1    height 4.821   height² 4.853        both columns centre 0, spread 1
standardize then expand, penalty 1    height 9.191   height² 0.241        the square has centre 1, spread 0.932

On heights between 150 and 183 cm the standardized height and the standardized square are very nearly the same column, and the ridge splits the slope between them, 4.821 and 4.853, which is the grouping the ridge and lasso page describes. Standardized first and then squared, the square is centred at one and carries little that the height does not, and the ridge puts 9.191 on the height and 0.241 on the square. Both orders fit the same nine people with the same family of curves, and the one that hands the penalty every column at the same spread scored higher at every penalty tried.

A complete implementation states what a step is and how steps are named and ordered, that each step is fitted on the output of the step before it, that predict applies the fitted steps and fits nothing, that the configured steps and model are copied before every fit and never fitted themselves, that a fit commits nothing until every part has succeeded, how a search varies a setting inside a step and what a wrong name does, whether an empty step list is allowed, which kind of model a chain of each task will accept, and what it does when a row arrives with a column missing, a column it never saw, or its columns in another order.

Every row below was put to the chain, on the twelve people where a row needs people, and the widget under the table asks the same list again on each load so the two cannot drift apart.

…

Two rows in the table are honest gaps rather than contracts. A step is accepted by name and object at construction and nothing checks that the object can transform, so a ridge model handed in as a step is accepted and fails inside fit with a bare attribute error about a missing method, which is not one of the named refusals.

And an expansion met by a column it never saw ignores the column rather than refusing it, so a chain whose first step is an expansion will quietly predict for a row carrying an extra measurement, where a chain whose first step is a standardizer will refuse the same row by name. Neither is defended here.

The first would be a type check at construction; the second is a difference between two steps’ own rules that the chain inherits rather than makes.