Does Eighty Percent Mean Eighty Percent?

A model can rank failures well and still overstate its probabilities. We need to compare its confidence with what actually happens across similar predictions.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

A Correct Choice Can Still Come With Misleading Confidence

BOTIE

As classifiers improved, a separate question remained: did the number attached to a prediction accurately describe its uncertainty? A model could choose the right class often while being too confident about individual choices.

Guo and colleagues' 2017 study examined this issue in modern neural networks and evaluated calibration methods, including temperature scaling. Our invented sensor group teaches how to inspect reliability before attempting a correction. It does not reproduce their experiments or establish that one adjustment works for every model.

Sources for this history

Can We Trust the Number Beside the Prediction?

Suppose Botie's model gives a group of sensors an 80 percent chance of failure. If we saw many comparable predictions, we would expect failures in roughly eight out of ten cases. That interpretation is what gives the probability practical meaning.

Our small report has four such predictions, and only two sensors failed. That is evidence to inspect, not proof from four observations alone. We need to compare estimated probabilities with observed frequencies while keeping the sample size visible.

Group comparable probability estimates

All four estimates are 0.8, so their average also represents the group's stated confidence.

Failure probabilities

An invented example for this lesson.

1 of 3

What we can now calculate

The first pair compares probability with frequency. The separate score reports average squared error without pretending it measures calibration alone.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie needs to compare the group's confidence with its observed failure rate, while also reporting squared probability error.

Choose an example

Failure probabilities in one group

Observed failures

Result from these values

[[0.8,0.5],0.34]

Follow the operations

1. Failure probabilities in one group
1. 00.8
2. 10.8
3. 20.8
4. 30.8
2. Mean
```
0.8
```
3. Observed failures
1. 01
2. 10
3. 21
4. 30
4. Mean
```
0.5
```
5. Collect two results
1. 00.8
2. 10.5
6. Subtract
1. 0-0.2
2. 10.8
3. 2-0.2
4. 30.8
7. Square
1. 00.04
2. 10.64
3. 20.04
4. 30.64
8. Mean
```
0.34
```
9. Collect two results
```
[[0.8,0.5],0.34]
```
10. Result
```
[[0.8,0.5],0.34]
```

These tiny groups illustrate the arithmetic. They do not estimate calibration precisely or fit a correction. Editing this example does not change the workshop tests.

1. Calibration Is About Repeated Predictions

We cannot judge an 80 percent prediction as wrong merely because one sensor stayed healthy. A probabilistic statement allows both outcomes. Calibration concerns the outcome frequencies among many comparable probability estimates.

A useful reliability diagram groups predictions into probability ranges and compares each group's average probability with its observed positive fraction.

Grouping trades detail for enough observations to estimate a frequency. Different bin choices can change the appearance, particularly with little data.

2. Calculate One Reliability Group

Our four sensors all received the same estimated probability. Their binary outcomes sum to the number of failures. Averaging those outcomes gives the group's observed frequency.

Mean predicted probability = (0.8 + 0.8 + 0.8 + 0.8) / 4 = 0.8
Observed failure frequency = (1 + 0 + 1 + 0) / 4 = 0.5
Predicted minus observed = 0.8 - 0.5 = 0.3

This group looks overconfident about failure. Four examples are too few to establish a precise long-run discrepancy, so the report must preserve its count.

3. Score the Probability Errors Too

The binary Brier score averages squared differences between probabilities and zero-or-one outcomes. Lower is better. It penalises confident wrong predictions strongly.

Squared errors: [0.04, 0.64, 0.04, 0.64]
Brier score = (0.04 + 0.64 + 0.04 + 0.64) / 4 = 0.34

Brier score is not a pure calibration measure. It also reflects how usefully the model distinguishes different risks. Inspect reliability and predictive discrimination separately rather than collapsing both questions into one label.

4. Fit Corrections on Separate Data

A calibration method learns a mapping from scores to probability estimates. It needs labelled data excluded from the original model fit or an appropriate cross-validation procedure. Evaluate the corrected probabilities on data not used to fit that correction.

A model can be calibrated overall while failing for a subgroup or after a population change. It can also predict the same base rate for everyone and be calibrated while offering little useful ranking.

The workshop audits one group. It does not fit a calibrator or certify that four observations establish reliable probabilities.