Audit one group of failure probabilities

A journey through does eighty percent mean eighty percent? with Botie.

Choosing and Checking a Model · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

A Correct Choice Can Still Come With Misleading Confidence

Part 1 of 2

As classifiers improved, a separate question remained: did the number attached to a prediction accurately describe its uncertainty? A model could choose the right class often while being too confident about individual choices.

Sources for this history

Guo and colleagues: On Calibration of Modern Neural Networks (2017)
Scikit-learn: Probability calibration

← Previous lesson: When Should a Probability Become an Action?Read the full lesson and derivations