What Could We Know at Prediction Time?

A model can appear remarkably accurate when a feature contains information that arrives after the decision. We need to reconstruct what was actually available when each prediction would have been made.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

When a Successful Evaluation Used Unavailable Information

BOTIE

As data-mining systems combined many records and preprocessing steps, evaluation could accidentally give a model information unavailable in the real prediction setting. High scores then measured access to privileged evidence as well as predictive ability.

Kaufman and colleagues examined leakage as information about a target that is not legitimately available to the prediction process. Our invented repair records make the timing issue explicit. Other forms include fitting preprocessing on test data or placing copies of an observation on both sides of a split.

Sources for this history

Can We Predict a Repair Using the Completed Repair Report?

Botie wants to predict at noon whether a greenhouse sensor will need repair later that day. The data export includes a morning reading, a noon reading, and a completed repair report entered that evening.

The evening report could make the prediction look easy. It may even state the answer. But it was not available at noon, so using it would test a different task from the one we intended to solve.

Fix the decision time

The task is to make the decision at noon. That time determines the boundary of available evidence.

Prediction time

12:00

An invented example for this lesson.

1 of 3

What we can now calculate

The gate compares the cutoff with each availability time. Selection keeps only the measurements whose evidence was available then.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie must predict repair need using only evidence available at the decision time. Later measurements remain in the export but cannot enter this prediction.

Choose an example

Availability hours

Prediction cutoff hour

Corresponding measurements

Result from these values

Follow the operations

1. Corresponding measurements
1. 019
2. 122
3. 299
2. Prediction cutoff hour
```
12
```
3. Availability hours
1. 09
2. 112
3. 218
4. Threshold gate
1. 01
2. 11
3. 20
5. Select real entries
1. 019
2. 122
6. Result
1. 019
2. 122

Times are simplified whole hours. A real system also needs dates, time zones, ingestion delays, and update history. Editing this example does not change the workshop tests.

1. Define the Decision Before Selecting Features

State what is predicted, for whom, and at what time. Without that definition, we cannot judge whether a feature was available legitimately.

Our task is a noon prediction of later repair need. A completed evening report fails the timing requirement even if it is stored in the same row as the morning measurements.

A column's name is not enough. Investigate how the value was created, updated, and joined into the table.

2. Compare Availability With the Cutoff

Each field has an availability time. Keep it only when that time is at or before the prediction cutoff. The workshop uses whole hours to make this boundary visible.

Prediction cutoff: 12
Availability times: [9, 12, 18]
Available by cutoff: [1, 1, 0]

The timestamp of the underlying event can differ from the time the system learned about it. A report describing a morning event but uploaded in the evening still cannot be used at noon.

3. Keep Fitting Inside the Evaluation Boundary

Leakage does not require an obviously future-looking column. A mean, vocabulary, selected feature, or imputation reference learned using held-out observations has also crossed the evaluation boundary.

Split first according to the intended prediction setting. Fit preprocessing and the model on training data. Apply those fitted choices to validation or test data without relearning from the held-out group.

Pipelines help with the order of fitting operations. They cannot determine whether a raw field was available at the intended decision time. That requires knowledge of the data collection process.

4. Test the Procedure You Can Actually Run

A realistic evaluation replays the information constraints of deployment. Time order, delayed labels, related entities, and record updates may all matter. A random row split is not automatically wrong, but it answers the wrong question when it ignores the dependency we care about.

The workshop returns the eligible measurements in their original order. It does not decide whether those features are useful or ethically appropriate; it checks the stated timing boundary.

Removing leakage may lower the score. That is a better measurement of the intended task, not a reason to put the unavailable information back.