When a Measurement Is Missing

An empty thermometer reading does not mean the greenhouse was at zero degrees. We need to distinguish what was measured from what we supplied to keep a calculation working.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

The Missing Observation Has a Story Too

BOTIE

Incomplete records posed a problem for statistical inference long before modern neural networks. Ignoring an absent observation can change the group we describe, especially when absence is related to the value we wanted to measure.

Rubin's 1976 paper established conditions under which the missingness process can be ignored for particular forms of inference. It did not say that replacing every blank with an average is universally valid. Our invented greenhouse example uses mean replacement as a transparent baseline so we can inspect exactly what it assumes.

Sources for this history

What Should We Do With an Empty Cell?

Botie opens the greenhouse report and finds a blank temperature. Perhaps the sensor lost power, or perhaps it stopped reporting when the greenhouse became too hot. Those possibilities have different implications.

A numerical method may still require a value in that position. We can supply a replacement, but we must keep clear that it is an estimate. Filling a cell makes an array usable; it does not recover the measurement that was never recorded.

Leave the blank visible

A blank is an unknown temperature. Writing zero there would assert something we did not observe.

Training readings

10
missing
14

An invented example for this lesson.

1 of 3

What we can now calculate

The training branch supplies one reference. The report branch retains its measurements and uses that reference only at absent entries.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie's report has absent temperatures. Learn a replacement from observed training readings, then fill only the missing report entries.

Choose an example

Training temperatures

New greenhouse report

Result from these values

Follow the operations

1. New greenhouse report
1. 0null
2. 118
2. Training temperatures
1. 010
2. 1null
3. 214
3. Keep observed readings
1. 010
2. 114
4. Mean
```
12
```
5. Fill missing readings
1. 012
2. 118
6. Result
1. 012
2. 118

Mean replacement is a baseline. It cannot recover the true missing values or represent uncertainty about them. Editing this example does not change the workshop tests.

1. Separate Missing From Zero

Zero is a possible temperature. Missing means we do not know the temperature. These must remain different values. The workshop uses a blank cell represented internally by null; Python uses a floating-point missing marker called NaN.

First identify which entries were observed. Do not let the missing marker participate in an ordinary average.

Training readings: [10, missing, 14]
Observed readings: [10, 14]

The set of observed readings may itself be biased. If the hottest sensors stop reporting, its average will not describe the missing temperatures well.

2. Learn the Replacement From Training Data

Mean imputation uses the average of observed training values as a replacement. Imputation means supplying missing entries. We estimate that reference once during fitting, then reuse it on later data.

Training reference = (10 + 14) / 2 = 12
New report:    [missing, 18]
Filled report: [12, 18]

Including the new report's measured temperature when choosing the reference would change the fitted preprocessing after seeing new data. For evaluation, it would let held-out observations influence the preparation.

3. Keep Track of What Was Replaced

A missingness indicator records which values were supplied. A one in that indicator means the original entry was absent; a zero means it was measured. That information can matter if missingness is related to how the records were collected.

New report:          [missing, 18]
Missing indicator:   [1, 0]
Filled temperatures: [12, 18]

The indicator preserves a distinction, not a guarantee. It cannot recover the absent reading. Mean replacement also understates variability if we later treat supplied values as independent observations.

4. Choose the Assumption Before the Convenience

A median can resist extreme observed values. A fixed replacement may have domain meaning. More elaborate methods use relationships among features, and some models handle missing entries directly. Each choice still needs evaluation under the intended conditions.

If every training value is missing, there is no observed mean to calculate. Stop and choose a documented fallback or remove that feature for a reason. Silently inventing a zero would hide the failure.

Botie's construction must learn from the training readings and fill the new report. It must preserve every value that was actually observed.