What Changed After the Model Was Fitted?

When Training and Use No Longer Share the Same Distribution

BOTIE

A predictor's evaluation can stop describing its environment when the frequencies or relationships in incoming data change. Detecting that mismatch is a separate problem from fitting the original predictor.

Lipton, Wang, and Smola's 2018 work studies label shift, where class proportions change under specific assumptions, and methods for detecting and correcting it. Our roof-category comparison is a simpler input-distribution audit. It is not their correction algorithm and does not assume that every observed change is label shift.

Sources for this history

Lipton, Wang, and Smola: Detecting and Correcting for Label Shift (2018)

Are We Still Predicting for the Same Greenhouses?

Botie's model was developed when most records came from glass-roofed greenhouses. The site expands, and plastic roofs now dominate the incoming reports. Even if the model code never changes, the population it sees has changed.

We need to notice that change before assuming the old evaluation still describes current performance. First we will compare simple category proportions. Then we will distinguish a changed input mix from a measured decline in prediction quality.

Keep category meanings aligned

Both reports use the order glass, plastic, mesh. Counts cannot be compared until that order agrees.

Reference counts

80
20
0

Current counts

40
60
0

An invented example for this lesson.

1 of 3

What we can now calculate

Normalisation removes volume differences. The remaining distance measures how the supplied category shares differ.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie needs to compare roof-category shares in a reference report and a new report. Both use the same category order.

Choose an example

Reference category counts

1

2

3

Current category counts

1

2

3

Result from these values

0.4

Follow the operations

1. Current category counts
1. 040
2. 160
3. 20
2. Add a list
```
100
```
3. Divide
1. 00.4
2. 10.6
3. 20
4. Reference category counts
1. 080
2. 120
3. 20
5. Add a list
```
100
```
6. Divide
1. 00.8
2. 10.2
3. 20
7. Subtract
1. 0-0.4
2. 10.4
3. 20
8. Absolute value
1. 00.4
2. 10.4
3. 20
9. Add a list
```
0.8
```
10. Set a constant
```
2
```
11. Divide
```
0.4
```
12. Result
```
0.4
```

Both reports must have positive totals and aligned, nonnegative counts. This is a distribution audit, not a performance estimate or an automatic retraining rule. Editing this example does not change the workshop tests.

1. Compare Like Categories Over Clear Windows

Choose a reference period and a current period. Keep the category vocabulary and collection rules consistent. A changed logging system can look like a changed population, so check how each report was produced.

Counts depend on how many observations were collected. Proportions describe the mix within each report.

Reference shares = [80, 20, 0] / 100 = [0.8, 0.2, 0]
Current shares = [40, 60, 0] / 100 = [0.4, 0.6, 0]

Both totals must be positive. An empty report has no empirical category distribution to compare.

2. Measure the Movement in Share

For discrete distributions over the same categories, total variation distance is half the sum of absolute probability differences. It ranges from zero for identical distributions to one for distributions with disjoint support.

Absolute differences = [|0.4 - 0.8|, |0.6 - 0.2|, |0 - 0|]
                     = [0.4, 0.4, 0]
Total variation = (0.4 + 0.4 + 0) / 2 = 0.4

We divide by two because moved share appears once as a loss and once as a gain. This is a descriptive distance between the supplied category distributions, not a hypothesis-test p-value.

3. Keep Different Kinds of Change Separate

Input shift concerns changed feature distributions. Label shift concerns changed outcome proportions under an assumption about class-conditional inputs. A changed relationship between inputs and outcomes is often called concept drift.

Our category audit can see roof proportions change. It cannot establish whether the temperature relationship changed, whether accuracy fell, or which assumption about outcomes holds.

A stable input histogram does not prove performance is stable either. Relationships or labels can change without a visible shift in this particular feature.

4. Connect the Alert to an Investigation

Inspect subgroup performance when labels become available, check collection changes, and compare against the intended deployment population. A monitoring distance is useful when it leads to a concrete investigation.

Thresholds for alerts should account for ordinary sampling variation, seasonality, and the cost of investigating. Do not automatically retrain because one small batch looks different.

The workshop calculates a transparent category distance. Its labels and proportions stay visible so the number has an interpretation.

Check yourself

Questions on Parts 1 to 4

Question 1 of 3

Doubling every count while preserving category shares changes this total variation distance.

Try it with Botie

Build with Botie

Write and run Python