When Should a Probability Become an Action?

A probability is evidence, not a complete instruction. We need to decide how likely a sensor failure must be before an inspection is worth its cost.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

Separating Statistical Evidence From a Decision

BOTIE

Prediction systems have long needed a second step between estimating an outcome and choosing an action. The same estimated risk can justify different actions when their consequences differ.

Current threshold-tuning documentation makes that separation explicit: estimating scores and selecting a cutoff are distinct tasks. Our invented sensor costs show one simple expected-cost rule. They are teaching units, not an estimate of real maintenance costs or a recommendation for an operational safety system.

Sources for this history

Scikit-learn: Tuning a decision threshold

Should We Inspect Every Sensor That Might Fail?

Botie's model assigns each greenhouse sensor a probability of failure. An inspection costs time, but missing a real failure also has a cost. A probability alone does not decide which action is preferable.

We need a rule connecting those probabilities to consequences. A threshold says how large the estimated probability must be before we act. The appropriate threshold depends on the decision we are making.

Start with probabilities

The model supplies evidence about failure, not an inspection decision.

Failure probability

An invented example for this lesson.

1 of 3

What we can now calculate

The costs determine a cutoff. The gate then turns each supplied probability into an action under that rule.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie has failure probabilities and a simplified cost table. Correct decisions cost zero; false alarms and missed failures have the supplied costs.

Choose an example

Estimated failure probabilities

False-alarm cost

Missed-failure cost

Result from these values

Follow the operations

1. Estimated failure probabilities
1. 00.1
2. 10.4
3. 20.8
2. False-alarm cost
```
1
```
3. Missed-failure cost
```
4
```
4. Add
```
5
```
5. Divide
```
0.2
```
6. Threshold gate
1. 00
2. 11
3. 21
7. Result
1. 00
2. 11
3. 21

This assumes credible probabilities, nonnegative costs with a positive sum, and zero cost for correct decisions. Different cost tables require a different derivation. Editing this example does not change the workshop tests.

1. Keep Probability and Action Separate

A probability describes an estimated frequency under similar circumstances. An action has consequences. Raising the threshold generally reduces positive decisions; lowering it generally increases them. Neither change requires retraining the score model.

The effect on false alarms and missed failures depends on the labelled data and the score ordering.

A conventional cutoff of one half is a convention, not a universal statement about the best decision.

2. Write Down the Cost Table

We will simplify the decision to inspect or leave alone. Correct decisions have zero cost. Inspecting a healthy sensor is a false alarm with cost one. Leaving a failed sensor alone is a miss with cost four.

For a failure probability p, inspection incurs the false-alarm cost only when the sensor is healthy. Leaving it alone incurs the missed-failure cost only when it fails.

Expected inspection cost = (1 - p) × 1
Expected leave-alone cost = p × 4

These expectations assume that the estimated probability is suitable for this population and that the stated cost table describes the decision. Real inspections can have costs even when they are correct, which changes the table.

3. Find Where the Preferred Action Changes

The two expected costs are equal at the cutoff. Above it, a missed failure becomes more expensive in expectation than a false alarm. We choose inspection at equality as our stated tie rule.

(1 - p) × 1 = p × 4
1 = 5p
p = 0.2
General cutoff = false-alarm cost / (false-alarm cost + miss cost)

The rule uses the relative costs. A larger missed-failure cost lowers the cutoff under these assumptions.

4. Choose and Test the Rule Without Reusing Final Answers

A cost-derived rule still needs evaluation. If we tune a threshold to a validation metric, we must keep the final test separate. Choosing the best cutoff on the final test and reporting that same result gives an optimistic estimate.

Calibration also matters: a score labelled as a probability must be credible for expected-cost arithmetic. A ranking alone can support threshold searches on validation data, but does not justify interpreting every score as a probability.

The workshop builds the cost-derived cutoff and applies it. The Python practice also evaluates a changed cost assumption.