Can We Repeat the Same Experiment?
Changing both a model and its sampled observations makes it difficult to explain a changed result. We need to keep track of the random choices so we can repeat a comparison.
10 sections · 6 questions · 2 coding challenges
The arrow keys move between sections too.
Giving Simulated Experiments a Repeatable Starting Point
BOTIE
NumPy separates the bit generator from the interface that draws values. Its PCG64 documentation specifies how an initial seed establishes a stream. Our lesson uses this distinction to explain repeatability; the workshop receives already chosen indices and does not implement a random-number generator.
Sources for this history
Did Our New Method Help, or Did We Draw Easier Examples?
Botie compares two ways of checking greenhouse sensors. If each method sees a different selection of readings, a changed score could come from the method or from the selection. We have changed two things at once.
A reproducible draw lets us hold the selection still while we investigate a change. It does not make that one selection representative of every possible draw. We need both repeatability and an honest account of variation.
Name the observations before drawing
An index identifies a reading. Keeping the indices makes the selected observations inspectable.
Index
- 0
- 1
- 2
- 3
Reading
- 10
- 20
- 30
- 40
What we can now calculate
Gather preserves the recorded choices. The mean describes that resample, so changing the indices can change the result even when the underlying report stays the same.
Explore the full example
Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.
BOTIE
Result from these values
26.6667
Follow the operations
1. Original greenhouse readings
- 010
- 120
- 230
- 340
2. Recorded sample indices
- 01
- 11
- 23
3. Gather selected entries
- 020
- 120
- 240
4. Mean
26.6667
5. Result
26.6667
The indices are supplied. This construction does not generate randomness or establish that a sample is representative. Editing this example does not change the workshop tests.
1. A Seed Starts a Sequence
We have stored readings in arrays and named readings in dictionaries. Generating random choices requires a different kind of object. A random generator contains a changing internal state and provides methods that use it to produce values. It is not itself the list of values it produces.
A seed gives us a repeatable way to initialise that state. Each draw advances it. Another call to the same generator continues the sequence; creating a new generator with the same setup starts a replay.
We will create the generator in separate steps so that each piece has a clear purpose.
Separate the seed, the generator, and the drawn array
Our seed is one integer. PCG64 is the algorithm that maintains the state and produces random bits. Generator supplies methods for turning those bits into the kinds of draws we request.
The integers method below requests three positions. Its lower limit is included and its upper limit is excluded. With four readings, the valid positions therefore run from zero through three. The size argument sets the number of requested entries.
import numpy as np
seed = 42
bit_generator = np.random.PCG64(seed)
rng = np.random.Generator(bit_generator)
positions = rng.integers(0, 4, size=3)
print(positions.tolist())
print(positions.shape)[0, 3, 2]
(3,)The result is an integer array containing three positions. The generator remains a separate object, ready for another request. This output is from the Python environment used by the site's challenges; record the library version and generator when preserving an experiment.
A new draw and a replay are different requests
First we draw twice from one generator. Then we initialise another with the same algorithm and seed. Passing the PCG64 object directly into Generator combines the two construction steps from the previous example.
The array_equal function compares two arrays' shapes and entries and returns one Boolean result.
import numpy as np
rng = np.random.Generator(np.random.PCG64(42))
first = rng.integers(0, 4, size=3)
second = rng.integers(0, 4, size=3)
replay_rng = np.random.Generator(np.random.PCG64(42))
replayed = replay_rng.integers(0, 4, size=3)
print(first.tolist())
print(second.tolist())
print(np.array_equal(first, replayed))[0, 3, 2]
[1, 1, 3]
TrueThe second request continued the first generator's sequence. The separate generator reproduced the first request. Consecutive draws can happen to contain the same values, so inequality is not the definition of a fresh draw. Advancing the generator's state is what makes it a new request.
2. Draw Positions, Then Retrieve Readings
Keeping the positions separates the choice of observations from the retrieval of their values. Once we have the positions, the lookup is completely determined. An integer index array can request the same position more than once, which is useful for sampling with replacement.
Sampling without replacement does not repeat a selected position. Bootstrapping deliberately allows repeats; splitting data into disjoint groups should not put the same observation in both groups.
Readings: [10, 20, 30, 40]
Indices: [ 1, 1, 3]
Selected: [20, 20, 40]
Selected mean = (20 + 20 + 40) / 3 ≈ 26.6667The repeated reading counts twice in this resample because its position was selected twice. It is not evidence that Botie independently measured that greenhouse twice.
Retrieve values in the order the indices request them
The first array holds temperatures. The second holds integer positions into that array. Square brackets apply the recorded choices, including their order and repetitions. We round the displayed mean to four decimal places.
import numpy as np
readings = np.array([10, 20, 30, 40], dtype=float)
indices = np.array([1, 1, 3])
selected = readings[indices]
print(indices.tolist())
print(selected.tolist())
print(np.round(selected.mean(), 4))[1, 1, 3]
[20.0, 20.0, 40.0]
26.6667The positions are whole numbers because they identify entries. The selected temperatures are floating-point values because that is what readings stores. We should not turn the index array into a set, since doing so would discard the repeated selection.
Choose readings with or without replacement
default_rng is NumPy's convenient way to create a generator. It currently uses PCG64 by default; explicitly naming the bit generator, as earlier, also records that choice. choice draws from the values we supply. With replace=False, no input position can be chosen twice in one draw.
A size tuple asks for a table of draws. Here replacement is allowed for the second request, so a draw can include the same observation more than once.
import numpy as np
readings = np.array([10.,20.,30.,40.])
rng = np.random.default_rng(42)
without_replacement = rng.choice(readings, size=3, replace=False)
print(without_replacement.size)
print(np.unique(without_replacement).size)
rng = np.random.default_rng(42)
resamples = rng.choice(readings, size=(2,3), replace=True)
print(resamples.tolist())
print(np.round(resamples.mean(axis=1),4).tolist())3
3
[[10.0, 40.0, 30.0], [20.0, 20.0, 40.0]]
[26.6667, 26.6667]The first input has distinct values, so its three unique selected values also demonstrate three distinct positions. In a dataset with equal-valued observations, different positions could still hold equal values. The two resamples happen to share a mean here, although their selected observations differ.
3. Keep a Fair Comparison Repeatable
If Botie compares two methods, both should see the same recorded draws before we attribute a changed result to the method. We can keep several draws in one two-dimensional index array: one row per draw, one position per selected observation.
Indexing a one-dimensional readings array with this index table produces a table of readings in the same arrangement. Each index is replaced by the value it selects.
The two rows below are illustrative draws. The coding challenge will ask you to summarise a different pair.
An index table becomes a table of selected readings
The integer table has two draws with three selections each. After retrieval, averaging along axis one combines the selections within each draw. It leaves one mean per draw.
import numpy as np
readings = np.array([10, 20, 30, 40], dtype=float)
indices = np.array([
[0, 1, 2],
[1, 2, 3],
])
selected = readings[indices]
print(indices.shape)
print(selected.tolist())
print(selected.shape)
print(selected.mean(axis=1).tolist())(2, 3)
[[10.0, 20.0, 30.0], [20.0, 30.0, 40.0]]
(2, 3)
[20.0, 30.0]The same original readings gave different averages under these two selections. Saving the indices lets us repeat either calculation exactly, provided we also keep the original data and its order.
Shuffle related arrays with one shared permutation
A permutation rearranges every position without replacement. Generate it once, then apply it to both features and labels. Calling a shuffle separately for each collection can destroy their correspondence.
import numpy as np
names = np.array(["north","middle","south","annex"])
readings = np.array([10,20,30,40])
rng = np.random.default_rng(42)
order = rng.permutation(readings.size)
print(order.tolist())
print(names[order].tolist())
print(readings[order].tolist())[3, 2, 1, 0]
['annex', 'south', 'middle', 'north']
[40, 30, 20, 10]The same order moves names and temperatures together. A saved permutation can define a repeatable split, but random splitting is not appropriate when time or related groups require a different evaluation design.
4. Do Not Select the Seed That Flatters the Model
Trying many seeds and reporting only the best score includes the luck of choosing that run. Decide how to evaluate a method before inspecting those outcomes, and keep final test data separate from development choices.
A reproducible experiment needs more than a seed. Preserve the input data, its ordering, the generator and library version, and the sequence of requests. Keeping the actual selected indices also makes the experiment easier to inspect.
Seed: one starting value
Generator: an object with changing state
Indices: an array of selected positions
Selected readings: the values at those positionsThose four things play different roles. In practice, we will first replay a draw, then summarise saved selections. A successful replay confirms that we repeated the calculation; it does not establish that the selection represents all the situations our method will face.
Generate a small simulated sensor report
A distribution describes the rule for drawing values. normal draws from a normal distribution with a supplied centre, loc, and standard deviation, scale. The size determines the output shape. The standard deviation is not a hard bound on possible deviations.
These are simulated values under a chosen assumption, not recorded thermometer measurements. Uniform draws over an interval would express a different assumption; a seed only fixes the sequence within the chosen rule.
import numpy as np
rng = np.random.default_rng(42)
simulated = rng.normal(loc=20, scale=2, size=(2,3))
print(simulated.shape)
print(np.round(simulated,1).tolist())(2, 3)
[[20.6, 17.9, 21.5], [21.9, 16.1, 17.4]]The layout represents two mornings and three greenhouses. It does not claim to reproduce real temperature patterns. Changing the distribution, its parameters, the request order, or the environment can change a simulated experiment.
Check yourself
Questions on Parts 1 to 4
Question 1 of 6
Calling the same generator again automatically restarts the original sequence.
Both draws use the same generator object.
import numpy as np
rng = np.random.Generator(np.random.PCG64(42))
first = rng.integers(0, 4, size=3)
second = rng.integers(0, 4, size=3)Try it with Botie
Build with Botie
Write and run Python