What Must a Model Improve On?

A Prediction Needs a Reference for Comparison

BOTIE

Predictive modelling needs control comparisons. Without them, a score cannot show whether a method gained anything from its inputs beyond a simple summary of previous outcomes.

Modern evaluation libraries preserve this practice through baseline estimators that ignore input features and predict a training summary. This is not an invention claim for one library. Our invented greenhouse example uses the training mean to make the comparison explicit before introducing more capable predictors.

Sources for this history

Scikit-learn: DummyRegressor baseline strategies

Would the Average Have Done Just as Well?

Botie has a new model for greenhouse temperature. Its error sounds small, but small compared with what? If a simple rule would do just as well, the extra complexity has not yet earned its place.

We will start with a deliberately limited predictor: use the average training temperature for every new observation. That gives us a baseline, a result the more elaborate method should improve under the same evaluation conditions.

Learn a single training reference

The baseline sees the training outcomes and no features. Its only fitted value is their mean.

Training temperatures

10
14
18

Reference

14

An invented example for this lesson.

1 of 3

What we can now calculate

The training branch supplies the prediction. Only the scoring branch reads the held-out answers.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie needs a reference score before adding a more complicated temperature model. Fit one constant on training data and evaluate it on the held-out temperatures.

Choose an example

Training temperatures

1

2

3

Held-out temperatures

1

2

Result from these values

Follow the operations

1. Training temperatures
1. 010
2. 114
3. 218
2. Mean
```
14
```
3. Held-out temperatures
1. 012
2. 120
4. Subtract
1. 02
2. 1-6
5. Square
1. 04
2. 136
6. Mean
```
20
```
7. Result
```
20
```

This baseline uses no input features. Its score is a comparison, not a guarantee of acceptable temperature prediction. Editing this example does not change the workshop tests.

1. Choose a Baseline That Matches the Task

A regression baseline might predict a mean or median. A classification baseline might use the majority class. A forecast might repeat the latest observation or the matching season. The baseline should be simple but relevant.

For our temperature example we will use the mean, because we will score squared errors. Among constant predictions, the training mean minimises the sum of squared training errors.

Training mean = (10 + 14 + 18) / 3 = 14

This fitted constant ignores inputs such as sunlight and heating settings. It establishes what we can achieve before using those features.

2. Freeze the Rule Before Evaluating It

Apply the fitted constant to held-out observations. Do not replace it with the held-out mean. That would use information the predictor did not have when making those predictions.

Held-out predictions: [14, 14]
Held-out answers:     [12, 20]
Errors:               [2, -6]

A baseline must obey the same information boundary as the model it is compared with.

3. Score Both Methods on the Same Observations

Mean squared error averages the squared differences between predictions and answers. We need the same units, same held-out examples, and same metric for a fair comparison.

Baseline MSE = (2² + (-6)²) / 2
             = (4 + 36) / 2
             = 20

A model with lower error on different examples has not yet won this comparison. Shared evaluation examples let us compare their mistakes directly.

4. Ask What an Improvement Actually Buys

A small reduction in average error may not justify greater latency, maintenance, or instability. Inspect which observations improve and which get worse. Repeat the evaluation when the dataset is small or the fitting procedure is variable.

The baseline is a reference, not a target that guarantees usefulness once beaten. A temperature predictor may still be unsuitable for a particular control decision even after improving its average score.

The workshop builds the whole baseline evaluation from training outcomes and held-out answers. It must not learn its constant from the test branch.

Check yourself

Questions on Parts 1 to 4

Question 1 of 3

Which data determine the baseline's constant?

Try it with Botie

Build with Botie

Write and run Python