Would More Examples Help?

Before collecting more data or choosing a more flexible model, we can examine how training and validation errors change as the training set grows.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

Model Comparisons Depend on How Much Data We Have

BOTIE

A comparison at one dataset size can hide how two learning methods respond as more examples become available. A method that works well with few observations need not remain the best choice with many.

Perlich, Provost, and Simonoff's 2003 study compared tree induction and logistic regression through learning curves. Our supplied greenhouse errors are invented illustrations, not results from that study or a model trained in this widget. They let us inspect the interpretation before running larger fitting experiments.

Sources for this history

Should We Collect More Readings or Change the Rule?

Botie's temperature model makes errors on new days. We could collect more days, or choose a more flexible rule. Those changes address different problems, so it helps to inspect the behaviour we already have.

Fit the same kind of model using several training-set sizes and measure both training and validation error. The resulting learning curves show how the method responds to more examples. They provide evidence for a decision, not a promise about what the next batch will do.

Fit with different numbers of examples

Each column refers to the same model family fitted with a different amount of training data.

Training examples

An invented example for this lesson.

1 of 3

What we can now calculate

Each subtraction compares the same training size. Read these gaps beside the error levels before deciding what to try next.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie has training and validation errors for matching training sizes. We need to see their differences without confusing a narrow gap with low error.

Choose an example

Illustrative training errors

Illustrative validation errors

Result from these values

Follow the operations

1. Illustrative validation errors
1. 09
2. 16
3. 24
2. Illustrative training errors
1. 01
2. 12
3. 23
3. Subtract
1. 08
2. 14
3. 21
4. Result
1. 08
2. 14
3. 21

These errors are supplied teaching illustrations. No model is fitted by this construction. Editing this example does not change the workshop tests.

1. Change the Training Size, Not Everything at Once

Choose several training sizes. At each size, fit the same procedure from scratch, including preprocessing. Evaluate the fitted model on its training subset and on data excluded from that fit.

Keep the metric and validation design consistent. Repeat subsets or splits when possible so an unusually easy selection does not masquerade as a trend.

A learning curve varies the amount of training data. A validation curve instead varies a model setting. The axes answer different questions.

2. Inspect Both Curves Before Summarising Them

Low training error with much larger validation error suggests the fitted rule depends heavily on details of the training sample. If the gap shrinks as data increase, collecting more comparable data may help.

If both errors stay high and close together, the current representation or model may be too limited, although noise, poor measurements, or distribution mismatch can produce similar symptoms.

At 10 examples: gap = 9 - 1 = 8
At 50 examples: gap = 6 - 2 = 4
At 100 examples: gap = 4 - 3 = 1

The supplied illustration shows a narrowing gap and improving validation error. It is not a universal shape that all models must follow.

3. Do Not Mistake a Small Gap for a Good Predictor

Subtracting the two errors measures their difference, not their absolute quality. Equal errors can both be unacceptable. Compare the validation score with the baseline and with the needs of the application.

Training MSE = 20
Validation MSE = 20
Gap = 20 - 20 = 0

This gap is smaller than in the earlier illustration, yet the validation error is much worse. A single gap score cannot replace the two curves.

4. Use the Pattern to Choose the Next Experiment

A curve can suggest whether to investigate more data, better features, regularisation, or a different model family. Test that suggestion rather than treating the visual pattern as a diagnosis with certainty.

The workshop calculates gaps from supplied errors. In a real study, obtaining each error requires a complete fit and evaluation at that training size. We keep those measurements explicitly separate from the arithmetic that summarises them.

Final test data should not guide repeated decisions about which model or dataset size to choose. Use validation data for those development decisions.