When the Important Class Is Rare

If almost every sensor is healthy, predicting healthy every time can look accurate while missing every failure. We need to inspect what the score says about each class.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

Learning From a Class That Appears Only Occasionally

BOTIE

Classification datasets often contain many examples of one outcome and few of another. Researchers found that a method could favour the majority while performing poorly on the minority that mattered to the application.

Chawla and colleagues' 2002 SMOTE paper studied synthetic minority oversampling, creating additional minority examples between nearby observations. That is one response to imbalance, not a universal remedy. Our lesson first makes the evaluation problem visible; it does not generate synthetic sensor records.

Sources for this history

Chawla and colleagues: SMOTE (2002)

How Did a Model Miss Every Failure and Still Look Good?

Botie checks a batch containing 95 healthy sensors and five failed sensors. A rule that always says healthy gets most answers right. It also fails at the entire reason we might have built an alert system.

The class counts shape both the learning problem and the score. We need to keep the rare class visible, then decide how different mistakes should influence training and decisions.

Count both classes

The data contain many more healthy sensors than failed ones.

Healthy

Failed

An invented example for this lesson.

1 of 3

What we can now calculate

Division makes class-specific recall. Averaging those recalls gives each class equal weight in the summary.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie needs a score that gives each sensor class a place. Compute each class's recall before combining the results.

Choose an example

Correct predictions per class

Total observations per class

Result from these values

0.5

Follow the operations

1. Correct predictions per class
1. 095
2. 10
2. Total observations per class
1. 095
2. 15
3. Divide
1. 01
2. 10
4. Mean
```
0.5
```
5. Result
```
0.5
```

All supplied class counts are positive. Balanced accuracy does not encode application-specific mistake costs. Editing this example does not change the workshop tests.

1. Count the Mistakes the Task Cares About

Accuracy divides all correct predictions by all examples. If one class dominates the dataset, it also dominates that average. The problem is not that the arithmetic is wrong; the score answers a question that may be insufficient for our task.

Always-healthy accuracy = 95 / 100 = 0.95
Failure recall = 0 / 5 = 0

A high overall score does not establish that rare events are being found. Start by inspecting the class counts and the confusion matrix.

2. Give Each Class an Equal Place in the Summary

Recall asks what fraction of a class was correctly identified. Balanced accuracy averages those recalls, giving each class equal weight in the summary regardless of its count.

For the always-healthy rule, the healthy recall is perfect and the failure recall is zero.

Balanced accuracy = (1 + 0) / 2 = 0.5

This makes the neglected class visible. It still does not specify the cost of a false alarm or a missed failure, and it does not replace the individual class scores.

3. Separate Training Choices From Evaluation Choices

We can weight training losses, resample training examples, or change the decision threshold. These are different interventions. Loss weighting changes the fitting objective; resampling changes the examples presented; a threshold changes how scores become actions.

Fit these choices using training and validation data. Oversampling before splitting can place related or duplicated information into both training and evaluation sets.

A balanced training sample does not imply balanced deployment data. Keep the final evaluation distribution appropriate to the real task, and report the class counts used.

4. More Copies Are Not More Independent Evidence

Repeating a rare observation can make it influence training more, but it does not create a new independent observation. Synthetic examples introduce additional assumptions about which points are plausible.

For a rare class, score estimates can also be unstable because they depend on few cases. Report that limitation and inspect which mistakes occurred. The workshop constructs balanced accuracy from separate correct and total counts for both classes.

An evaluation set with no examples of a required class cannot estimate that class's recall. Do not quietly replace an undefined recall with a reassuring value.