One Calculation for a Whole Report
Botie applies the same rule to many readings. We will first write the repeated steps, then express them with array operations while keeping the intermediate values and memory costs understandable.
11 sections · 4 questions · 2 coding challenges
The arrow keys move between sections too.
Moving Repeated Arithmetic Into Numerical Routines
BOTIE
NumPy exposes common elementwise operations as universal functions, often shortened to ufuncs. This lets us express the numerical rule at array scale. It does not eliminate the need to inspect temporary arrays or prove that the rule matches the intended calculation.
Sources for this history
How Do We Repeat a Rule Without Hiding What It Does?
Suppose we need to compare every greenhouse reading with a target. A Python loop can visit each entry, subtract the target, and save the result. Once we understand those steps, an array expression can ask NumPy to perform that same repeated calculation.
This is vectorisation. It is useful because common numerical loops can run inside compiled library routines, but a short expression is not automatically correct or economical. We still need to understand the operations and the shapes it creates.
Start with observations and a reference
Each greenhouse is compared with the same target.
Reading
- 18
- 22
- 26
Target
- 20
- 20
- 20
What we can now calculate
Each reading contributes before the reduction. The construction exposes the intermediate array so a cancellation mistake is visible.
Explore the full example
Choose an example, then change a value. Follow the calculation below to see which result changes and why. These are small teaching examples; the construction later asks you to choose the operations yourself.
BOTIE
Result from these values
14.6667
Follow the operations
1. Greenhouse temperatures
- 018
- 122
- 226
2. Reference temperature
20
3. Subtract
- 0-2
- 12
- 26
4. Square
- 04
- 14
- 236
5. Mean
14.6667
6. Result
14.6667
The target is supplied. This example evaluates a fixed reference rather than learning a prediction rule. Editing this example does not change the workshop tests.
1. Follow the Repeated Steps Before Shortening Them
A for statement visits each value in a collection. The indented instructions run for the current value, then repeat for the next. We can append each result to an initially empty list.
An array subtraction performs the corresponding elementwise operation. Elementwise means each output is calculated from the corresponding input entry or aligned entries.
The loop and the array expression answer the same question
The loop version makes the repeated operation visible. The array version expresses that rule for the collection.
import numpy as np
readings = np.array([18,22,26])
differences = []
for reading in readings:
differences.append(int(reading - 20))
print(differences)
print((readings - 20).tolist())[-2, 2, 6]
[-2, 2, 6]int converts each scalar to a Python integer for the list display. The shorter form avoids our explicit Python loop. np.vectorize can make a Python function convenient to call over arrays, but it is not a promise of compiled-loop speed.
2. Combine Elementwise Steps With a Reduction
The size of an error and its direction answer different questions. abs discards the sign. square squares each entry. sqrt takes square roots. Reducing afterwards, with mean for example, combines the separate contributions.
Keep intermediate arrays named while learning the calculation. That makes it possible to check whether a sign was lost too soon or a reduction happened along the wrong axis.
Calculate error contributions before averaging them
These residuals are observations minus a target. We print each transformation before reporting the average squared error and its square root.
import numpy as np
residuals = np.array([18.,22.,26.]) - 20
squared = np.square(residuals)
print(np.abs(residuals).tolist())
print(squared.tolist())
print(np.round(squared.mean(), 4))
print(np.round(np.sqrt(squared.mean()), 4))[2.0, 2.0, 6.0]
[4.0, 4.0, 36.0]
14.6667
3.8297Squaring prevents opposite signs from cancelling. Taking a square root after averaging returns to the original units; averaging square roots would be a different calculation.
3. Choose Between Values Without Dropping Positions
A Boolean mask selects a subset and can shorten an array. where instead chooses one of two values at each position, preserving the aligned output shape. Its arguments are a condition, the value when true, and the value when false.
Use parentheses around each array comparison when combining them with & for and, | for or, and ~ for not. Python's plain and and or expect single truth decisions and are not substitutes for these elementwise operators.
Keep a report while marking readings inside a band
The mask identifies readings inside our illustrative operating band. where writes a label at every original position, while Boolean indexing keeps only qualifying readings.
import numpy as np
readings = np.array([18,22,26])
inside = (readings >= 20) & (readings <= 25)
print(inside.tolist())
print(np.where(inside, "within band", "review").tolist())
print(readings[inside].tolist())[False, True, False]
['review', 'within band', 'review']
[22]The labels still correspond to all three original readings. The selected list contains only one. A condition does not need to mean deletion.
4. Use a Named Operation When It Expresses the Rule
maximum can compare each entry with a lower bound; minimum can compare with an upper bound; clip applies both limits. These operations change values rather than filtering observations. Preserve the original if the clipped values are only intended for a display.
Other ufuncs used later include exp and log for exponential and logarithmic calculations, and sin and cos for angles in radians. Their input assumptions matter: a real logarithm needs a positive value, and a negative value has no real square root.
Limit the display without rewriting the sensor reading
The clipped result puts every displayed entry inside the requested range. It does not establish that the sensor's out-of-range reading was wrong.
import numpy as np
readings = np.array([18,22,26])
display = np.clip(readings, 20, 25)
print(display.tolist())
print(readings.tolist())
ratios = np.array([1.,2.,4.])
print(np.round(np.exp(np.log(ratios)), 6).tolist())[20, 22, 25]
[18, 22, 26]
[1.0, 2.0, 4.0]The original readings are unchanged. The final line illustrates reversing a logarithmic transformation for positive ratios, subject to floating-point precision. It is not a reason to logarithmically transform every temperature report.
5. Check the Arrays a Short Expression Creates
Broadcasting can create a large output even when both inputs are modest. Comparing every observation with every reference can produce a three-dimensional intermediate: observation, reference, feature. A batch reduces how many observations we handle at once.
The nbytes attribute reports an array's numerical storage size. A copy and a view can expose equal nbytes even when the view shares storage, so it is not the total unique memory used by a whole program. Profile real workloads before making speed claims.
Count storage before scaling the example
Each float64 entry uses eight bytes. The new axis below creates a view; square creates a numerical result array.
import numpy as np
readings = np.array([[18.,22.,26.],[19.,23.,27.]])
print(readings.nbytes)
expanded = readings[:, None, :]
print(expanded.shape)
print(np.shares_memory(readings, expanded))
print(np.square(readings).shape)48
(2, 1, 3)
True
(2, 3)Know which axes and temporary results an expression introduces before applying it to millions of records. Vectorisation is a way to express repeated work, not a guarantee that all work fits in memory.
Check yourself
Questions on Parts 1 to 5
Question 1 of 4
Which operation preserves one output position per reading?
Use this setup. The code is given; its result is for you to work out.
import numpy as np
readings = np.array([18,22,26])
inside = (readings >= 20) & (readings <= 25)Try it with Botie
Build with Botie
Write and run Python