Reconstruct the prediction-time report

A journey through what could we know at prediction time? with Botie.

Data Preparation · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

When a Successful Evaluation Used Unavailable Information

Part 1 of 2

As data-mining systems combined many records and preprocessing steps, evaluation could accidentally give a model information unavailable in the real prediction setting. High scores then measured access to privileged evidence as well as predictive ability.

Sources for this history

Kaufman and colleagues: Leakage in Data Mining (2012)
Scikit-learn: Common pitfalls and data leakage

← Previous lesson: When the End Is Beside the Beginning Read the full lesson and derivations