Have We Really Held Out a New Example?

Several rows can belong to the same greenhouse, patient, or device. If we want to predict for a new entity, a random row split can put familiar entities on both sides of the evaluation.

10 sections · 3 questions · 2 coding challenges

1 / 10

The arrow keys move between sections too.

Repeated Measurements Need a Different Evaluation Boundary

BOTIE

Repeated observations are common in scientific datasets. Measurements from one person, device, or location can share properties that distinguish them from other groups. Treating all rows as independent can therefore overstate performance on new groups.

Grouped validation procedures address the evaluation boundary by keeping a group's observations together. Scikit-learn documents these splitters separately from ordinary random splits. Our invented greenhouses make the group identity visible without claiming that one split design is appropriate for every prediction task.

Sources for this history

Scikit-learn: Cross-validation for grouped data

Was the Greenhouse New, or Just the Row?

Botie records several readings from each greenhouse. We want to know whether a model will work for a greenhouse it has never encountered. Randomly splitting individual readings can put one greenhouse's morning reading in training and its afternoon reading in validation.

Those are different rows, but they share the same greenhouse. The model may benefit from familiar construction, sensor behaviour, and location. To answer the new-greenhouse question, the whole greenhouse must stay on one side of the split.

Different rows can share an identity

Each greenhouse has several measurements. The group ID is attached to every row.

Record groups

An invented example for this lesson.

1 of 3

What we can now calculate

Distinct IDs remove repeated records from the audit. Each remaining match identifies one greenhouse shared by both sets.

Explore the full example

Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.

BOTIE

Botie needs to know whether any greenhouse appears on both sides of a proposed split. Repeated rows must not count the same overlapping greenhouse more than once.

Choose an example

Training greenhouse IDs

Validation greenhouse IDs

Result from these values

Follow the operations

1. Training greenhouse IDs
1. 0A
2. 1A
3. 2B
2. Keep distinct items
1. 0A
2. 1B
3. Validation greenhouse IDs
1. 0C
2. 1C
4. Keep distinct items
1. 0C
5. Compare category membership
0
0
6. Flatten a list
1. 00
2. 10
7. Add a list
```
0
```
8. Result
```
0
```

This checks the supplied group identity only. It cannot discover hidden relationships or incorrectly assigned IDs. Editing this example does not change the workshop tests.

1. State What Must Be New

A new reading from a familiar greenhouse and a reading from a new greenhouse are different generalisation tasks. The split should reflect the one we intend to claim.

For a new-greenhouse evaluation, the greenhouse ID defines the group. For another task, the relevant group might be a patient, author, device, household, or location.

Grouping is a modelling decision based on dependencies, not simply a request to split on whichever column is most convenient.

2. Keep All Records From a Held-Out Group Together

Choose held-out group identities, then select their rows. The remaining groups supply training rows. We should be able to list the group IDs on both sides and verify that their intersection is empty.

Training group IDs:   [A, A, B, B]
Validation group IDs: [C, C]
Shared distinct groups: []
Overlap count: 0

A zero overlap count is necessary for this grouping rule. It does not rule out other leakage, such as copies with different IDs or preprocessing fitted on all rows.

3. Weight the Report Deliberately

If one greenhouse contributes hundreds of rows and another contributes only a few, a pooled row score mostly describes the larger group. Depending on the goal, we may also want a score per greenhouse and an average across greenhouses.

Those are different summaries. Report which one you used and keep group sizes visible. Do not call a group-average score a row-average score.

The workshop audits identity overlap. It deliberately does not fit a model or choose how group errors should be weighted.

4. Combine Grouping With Time When Necessary

A future reading from a new greenhouse may require both a time boundary and a group boundary. Neither a random grouped split nor a simple chronological split automatically handles both.

Also check whether held-out groups contain the classes needed for evaluation. Very few groups make uncertainty difficult to estimate even if the row count is large.

The Python challenges select held-out rows by group identity and audit a separate proposed split. That makes the evaluation design inspectable before any fitting happens.