Which Feature Mattered

Ask a fitted model which of its columns mattered and it can reasonably answer three different questions. Which columns did it ask about most often? Which columns removed the most impurity when it asked? And which columns would it miss if they were taken away, in the sense that its predictions would get worse? Those sound like one question and they are not, because a column can be asked about at every other node and remove almost nothing, or be asked about once at the root and decide everything, or be leaned on so heavily that scrambling it wrecks the model while another column that removed more impurity in the growing turns out to be replaceable.

…

The four rows of bars are the four readings on that forest, and the column with the most bars is not the column with the most credit. Switching the widgets further down to the lone tree shows the same crowd read from a model that memorised it.

Two datasets carry the page. The first is the tangled crowd of twenty-five people that the bagging and forests pages fit, children and adults by height and weight, on which a tree grows eight splits to reach pure leaves, and that is where the split reading is assembled by hand. The second is the larger drawn crowd from the box above, which has two columns added to it that a careful reader should distrust.

One is a raffle ticket number, drawn without reference to anything and different for every one of the two hundred people. The other is height read a second time with a little measurement error, correlated with the first reading at 0.996. Weight is correlated with height at 0.898 and the ticket at −0.131, which is what chance looks like on two hundred rows.

Beside those sits the puzzle in the box at the top of the page, built to defeat the trees page’s search. Two columns are coin flips, and the class is 1 exactly when the two flips disagree. Ask about either coin alone and both sides of the split come back half and half, no better than the node was before, so no single question helps.

A third column is pure noise, drawn without any reference to the class, and with three hundred distinct values it hands the search a threshold between every adjacent pair, and among hundreds of tries one always lines up with a few rows by chance. It is the ticket column’s trick in a setting where the real columns cannot defend themselves.

Whichever question is asked, the answer comes back the same way, as one non-negative number per column that the model was fitted on, scaled so that the numbers sum to one. Every column is included, a column that earned nothing at zero, because a zero is a finding rather than an absence. The scaling is what lets a reader say that a column accounts for a fifth of the explanation without first working out the total, and it is also the sharpest limit on what the numbers mean.

They are shares of this fitted model’s explanation, so two columns carrying one signal divide a share between them and each looks half as important as either would alone, which is a fact about the fit and not about the columns.

share(feature) = raw score(feature) / Σ raw score over every feature

The raw score is the quantity each measure produces, impurity removed for the first and accuracy lost for the second. A model whose raw scores all come to zero has no shares to report, and rather than divide zero by zero the reading is refused, in words the last Part quotes.

Start with the trees page’s gain. A node holds some people with some mixture of children and adults, and that mixture has an impurity. Asking a question sends the people to two children, each with its own impurity, and the gain is the parent’s impurity minus the two children’s impurities weighted by how many people went each way. It is the number the split search maximised when it chose the question, and the tree recorded it at the time.

gain = I(parent) − (n_left / n)·I(left) − (n_right / n)·I(right)

So the credit a question earns for its column is its gain times the number of people that reached the node. A question asked of every person decides where every person goes; a question asked of three people near a leaf decides where three go, and an unweighted sum of gains would call those equal. The tangled crowd has the clearest case of the difference I have found.

credit(node) = rows at node × gain at node

fourth split credit ≈ 3 × 0.4444 ≈ 1.333
root split credit ≈ 25 × 0.1881 ≈ 4.702

Now walk the tangled crowd’s tree and keep a ledger. Each split adds its credit to its column’s running total, and at any moment the running shares are the running totals divided by their sum. The slider adds the eight splits in the order the decision-tree page steps through them, breadth first from the root.

…

After the root, height holds all the credit. The second split asks about weight in a node of nine people.

second split credit ≈ 9 × 0.0494 ≈ 0.444

Weight’s share is now about 0.086. The fourth split lifts it to 0.232, the closest the two columns come. Later splits favour height, including two questions worth 1.5 apiece. The final ledger is:

height credit ≈ 9.702
weight credit ≈ 2.778
total credit ≈ 9.702 + 2.778 = 12.480

Divide each feature’s accumulated credit by the total credit to obtain its reported share.

height share ≈ 9.702 / 12.480 ≈ 0.777
weight share ≈ 2.778 / 12.480 ≈ 0.223

A feature that wins no split receives a zero. The implementation keeps the full list of columns so that an unused feature appears explicitly. On the parity puzzle, the tree never asks about the first coin, and its reported share is zero.

share(feature) = Σ over its nodes of rows × gain  /  Σ over every node of rows × gain

A single tree’s ledger is as unstable as the tree. Change a few people and the root changes, and a column carrying three quarters of the explanation can fall to nothing. A forest holds twenty-five ledgers, one per member, each grown on its own resample under its own feature draws, and the forest’s reading is the average of the members’ shares. Shares rather than raw totals, because a deeper member removed more impurity in absolute terms without that meaning its columns mattered more.

The forest’s figure is close to even because the feature lottery made it so. A member denied height at a node spends that node on weight and the ledger credits weight for it, so the forest’s reading describes how these twenty-five trees used their columns, which the lottery decided as much as the crowd did. The bagged committee, with no lottery, leans on height nearly as hard as the lone tree.

Nothing in Part 2 touched the data again. Every gain and every row count was recorded when the tree was grown, so the reading is one walk of the tree, and for a forest one walk of each member. That is why it is the reading every tree library hands out by default, and why it is worth knowing exactly what it is a reading of before it is handed on.

The search compares the best question each column can ask. A coin offers one threshold, so it gets one chance to show a gain. A column with three hundred distinct values offers a threshold between every adjacent pair, so it gets hundreds of chances, and the search keeps the best of them. On the parity puzzle that is decisive, since the coins cannot win a single question, and the lone tree roots on the noise column and asks about it at three of its four nodes, for a share of 0.807 on a column that carries nothing.

On the drawn crowd the same trick meets real columns, and I expected the ticket to win a few splits deep in the lone tree. It won none. Height and weight bring the tree to pure leaves in sixteen questions and the ticket is never the best option at any of them, so the bias needs the search to be short of better options before it shows. The forest arranges exactly that, because offering one column per split means a third of its nodes see the ticket alone.

…

Offered the ticket by itself at a node, a member must ask about it, and with two hundred distinct values some threshold always lines up with a few people by chance. The forest asks about the ticket at 196 of its 559 splits, more than it asks about height, and the ledger credits it with 0.107 of the impurity removed. Switch to the lone tree and the ticket drops to zero of sixteen.

Read height twice with a little error and the two readings are correlated at 0.996, so at nearly every node the best question on one is almost exactly the best question on the other. Whichever wins by a hair takes the whole credit for that node. Across a tree the credit lands on both, in shares that depend on tie-breaking and on which of the two the lottery happened to offer, and not on anything about the crowd.

…

With height alone the lone tree credits it 0.897; with the second reading beside it the two split the same work 0.860 to 0.101. The forest is the sharper case. Height alone earns 0.479 of the forest’s impurity; with its twin present the twin leads at 0.510 and height falls to 0.257, and weight, which did not change, moves from 0.521 to 0.233 because the twin took the nodes weight used to be offered.

The second reading of height is height plus measurement error, so by construction it cannot matter more than the first, and the ledger says it does.

Every entry in the ledger was made during growth, on the rows the model was fitted to, so the reading has no held-out version at all. A tree grown until its leaves are pure has memorised its rows, and the ledger credits whatever columns it memorised them with. On the drawn crowd the lone tree reaches an accuracy of 1.0 on its own rows and 0.845 on rows it never saw, and the ledger cannot tell which of its sixteen questions were the ones that generalise.

That is the extreme, and it is the honest one. A tree handed only the ticket grows 109 questions to depth 21, reaches an accuracy of 1.0 on its rows and exactly 0.5 held out, and the split reading hands the ticket all of the credit, as it must, since the ticket removed every unit of impurity the tree removed. The reading describes the growth accurately, and nothing in it says whether the growth generalised, which the held-out 0.5 says on its behalf.

A forest averages twenty or thirty ledgers, and averaging removes the part of the error that differs from member to member. It cannot remove a bias every member shares, and every member’s search prefers columns with many places to cut. On the parity puzzle the forest recovers the two-coin rule, 0.993 on its rows and 0.960 held out, and its averaged ledger still hands the noise column 0.519, more than the two real columns combined.

On the drawn crowd with all four columns and two offered per split, the ticket keeps 0.043 of the forest’s impurity across thirty members, at 77 of the 418 splits.

The second measure never looks inside the model. Score the fitted model once with every column intact. Then, for each column in turn, shuffle that column’s values among the people, leaving every other column and every label where it was, and score again. The shuffle keeps the column’s values exactly and destroys its pairing with the label, so whatever the score lost is attributable to that pairing and to nothing else, and the loss is the column’s raw score.

drop(feature) = intact score − score with that column shuffled

It would be simpler to overwrite the column with zeros or with fresh noise, and it would measure the wrong thing. Replacing the column changes two things at once, its pairing with the label and its own distribution, and a model that had never seen a height of zero would fail for the second reason as much as the first. Shuffling keeps every value the column had, in the same proportions, and moves only which person has which, so the drop is the pairing’s and nothing else’s.

One shuffle is one draw from a noisy quantity, and a column that happens to shuffle into nearly its original order will understate itself. The measure here shuffles each column five times from one seeded generator and averages the drops. A drop can come out negative, when the model was slightly better off without the column, and the average is clamped to zero after it is taken rather than each draw before, because the five drops are five estimates of one number and a negative one should be allowed to cancel a positive one.

Clamping each draw would push a useless column’s score upward.

drop_j = s − (1 / R) · Σ over R shuffles of s(column j shuffled)
share_j = max(0, drop_j) / Σ_k max(0, drop_k)

Everything so far scrambled the rows the model learned, and that makes the question a precise one, which columns did the model lean on while it learned. Scrambling rows it never saw asks which columns it leans on when it is being useful, and the two answers part company exactly on the models where the question matters, the ones that memorised. The ladder below is the scramble on the worked draw, both models, and the switch changes which rows are scrambled.

On its own rows the forest gives the noise column 0.018; held out its drop is −0.004 and its share exactly zero, with the coins at 0.510 and 0.490. The lone tree is at chance held out, 0.497, so there is nothing to lose, and its dots stay within 0.012 of the line. Breiman’s recipe scrambles each member’s own out-of-bag rows, which costs no held-out set at all.

The scramble here takes whichever rows it is handed and has no per-member out-of-bag variant, so this page uses held-out rows where the forests page would have used the rows each member left out.

The split measure needs one walk through a fitted tree. Permutation importance needs a baseline scoring pass and another pass for every feature in every repeat. The parity puzzle has three features and uses five repeats.

scoring passes = 1 + features × repeats
               = 1 + 3 × 5
               = 16

That cost grows with the number of features and repeats. The split measure is cheap to read from the fitted tree; permutation importance costs more because it measures changed predictions.

It is tempting to read the forest’s result as the scramble seeing through a column that fooled the ledger, and to expect it to do the same on any model. The lone parity tree says otherwise. It scores 0.537 on its own rows and 0.497 held out, chance either way, because it rooted on noise and never found the two-question structure.

Read from its splits, the noise column gets 0.807. Scramble that column and the tree falls from 0.537 to 0.504, most of the little it had above chance, while scrambling the second coin takes it only to 0.523 and scrambling the first changes nothing, so the scramble hands the noise column 0.710. Both readings name the noise column, and both are right.

That tree built itself out of noise, and it genuinely relies on it.

The disagreement needs a model that works, because only a model that found the real structure can be leaning on the coins while its ledger still credits the noise. The parity forest is that model, at 0.993 on its rows and 0.960 held out, and it is exactly there that the ledger, biased by the hundreds of thresholds it was offered in every member, keeps crediting a column the model no longer needs.

The drawn crowd’s forest with the ticket among its columns tells the story on people rather than coins. The ledger credits the ticket 0.107 and the scramble on the training rows still gives it 0.084, since thirty memorising trees do lean a little on the raffle numbers they memorised with. Held out, the ticket’s drop is −0.012 and its share exactly zero, while height takes all of it.

…

The three rows of bars are the three readings of one fitted forest, and the ticket’s bar shrinks from the first to the second to the third. The first row describes the growth, the second what the forest leans on when it is asked about the very rows it grew on, and the third what it leans on when asked about people it has never seen, which is the only one of the three that agrees with how the crowd was made.

The scramble has a failure of its own, and it is the mirror of the ledger’s. Scramble one of two columns carrying the same signal and the model leans on the other, so the drop is small and both look unimportant, where the ledger at least divided the credit between them. The drawn crowd’s second reading of height is the test.

…

With height alone, scrambling it held out costs the forest 0.360 of its accuracy. With the second reading beside it, scrambling height costs 0.148 and scrambling the second reading costs 0.184, and the two together, 0.332, come to less than the one column cost by itself, because whichever is scrambled the model still has the other.

The shares say the twin matters more than height, 0.540 to 0.434, which is a fact about which of two nearly identical columns the lottery handed the members more often. The lone tree does the opposite thing and is no more informative, since it won five splits on the twin and held out the twin’s drop is −0.001, because the tree leans on the first reading wherever it counts.

Both measures hand out zeros, and every zero on this page is a statement about one fitted model. The lone parity tree gives the first coin exactly zero by both readings because it never asked about it, and the forest fitted to the same rows gives that coin 0.479 on its rows and 0.510 held out.

The drawn crowd’s lone tree gives the ticket zero because height and weight finished the job first, and its forest, short of options at a third of its nodes, gives the ticket 0.107. Weight in that lone tree gets a held-out share of exactly zero, from a drop of −0.030, and the next Part shows what weight can do when nothing better is offered.

The drawn crowd’s rule assigns adult by height alone, with a little noise, and weight follows height only because both follow overall body size. Fit a model on weight and nothing else and it predicts adulthood usefully, the lone tree at 0.755 held out and the forest at 0.760, and every reading gives weight a share of one, because it is the only column there is.

Nothing in that report is wrong, and nothing in it says weight has any role in who is an adult. The model relies on a proxy, and reliance is what the scramble measures.

…

Offer height beside weight and the same forest’s held-out share for weight falls to 0.043, from a drop of 0.016, which is closer to the rule. Neither number is a finding about what makes a person an adult. The second reading of height in section 22 is the same lesson from the other side, since a column that is height plus measurement error cannot cause anything height does not, and the forest leaned on it more than on height.

A third way to ask the question is to refit without the column and see what the held-out accuracy loses, and the drawn crowd’s arrangements are those refits. Take the ticket away from the forest and its held-out accuracy moves from 0.870 to 0.880, so the ticket was worth nothing and slightly less than nothing; take it from the lone tree and the accuracy stays at 0.845 exactly, since the tree never used it. That agrees with the scramble held out and disagrees with the ledger, which credited the ticket 0.107.

The refit also shows the limit of every reading here. Weight scores 0.043 held out beside height and a refit that drops it would lose little, and weight alone still reaches 0.760. A column’s share is not its value; it is its value to this model given the other columns it had.

So a report of importances should say which question it answered, which model it answered it for, which rows were scrambled if any were, how many repeats under which seed, and how correlated the columns are, because without those the shares are a bar chart with no scale. On this page the same forest reports 0.107 for the ticket by the ledger, 0.084 by the scramble on its rows and 0.000 by the scramble held out, and the three are all correct answers to three questions.

A complete implementation of the split reading states which impurity the gains are measured in, that the credit is rows times gain and not gain alone, whether stand-in splits are counted as the 1984 book counted them, how a forest combines its members, by averaging shares or by summing credits, and that every fitted column appears in the report with a zero where it earned nothing.

A complete implementation of the scramble states the score it uses, how many repeats and under what seed, that the column is permuted rather than replaced, that the clamp follows the average, which rows are scrambled and whether out-of-bag rows are used per member, and what happens when no column’s scramble lowers the score. Both state that the shares are normalised to sum to one and that the raw quantities are available beside them.

Every row below was run before it was written down, and the words in the right column are the refusal’s own where the edge was refused.

The refusal worth remembering is the one for a model that never used its columns. A vector of zeros would type-check and sum to nothing, and a reader would take it for an answer; it is refused with a reason instead, and the ticket-only forest in section 17 is that refusal on real rows.