Statistics and Probability Primer

Five people's measurements fit in a small table. Thousands do not. We need to describe where values gather, how much they differ, and what an observed group can tell us about a larger one.

Start with the observations. A summary is useful only when we know what information it keeps.

Five people, each measured twice. That is the whole dataset, and it carries every idea on this page.

height (cm):  160   165   170   175   180
weight (kg):   58    66    68    74    74

Two words are worth fixing before anything is computed. An observation is one of the five people, one row, everything measured about a single case. A variable is one of the two measurements taken across all of them, one column, the same quantity recorded for everybody.

Almost everything in this primer is a statement about a column, or about how two columns relate. The rows are what we have; the columns are what we describe.

The distinction earns its keep because the two directions answer different questions. Reading across a row describes one person and nobody else, which is what a model does each time it makes a prediction. Reading down a column compares the same quantity across everybody, and every summary on this page is computed that way, down a column. The units belong to the column too, centimetres for one and kilograms for the other, which is a small point now and returns in section 10.

Five heights can simply be read. Five thousand cannot, and neither can the columns a model is usually handed.

A summary trades detail for something a person can hold in mind. Two numbers about a column, one saying roughly where its values sit and one saying how spread out they are, answer most of the questions anyone asks of it, and they can be compared across columns and across datasets in a way that a list of raw values cannot.

The cost is real and worth stating up front. Every summary discards something, and the rest of this primer is partly about noticing what each one throws away.

The mean is the most familiar summary. Add the values and divide by how many there are.

mean height = (160 + 165 + 170 + 175 + 180) / 5 = 850 / 5 = 170
mean weight = (58 + 66 + 68 + 74 + 74) / 5 = 340 / 5 = 68

There is a picture that explains why the mean is the number it is. Put the values on a line and imagine a plank with a weight at each value. The mean is the point where the plank balances, because the pull of the values above it exactly cancels the pull of the values below.

Once there is a centre, every value can be described by its distance from it. Subtract the mean from each value, and what is left is called the deviation.

height:          160   165   170   175   180
height − mean:   −10    −5     0     5    10      sum = 0

weight:           58    66    68    74    74
weight − mean:   −10    −2     0     6     6      sum = 0

A deviation carries two pieces of information. Its sign says which side of the centre a person is on, and its size says how far out they sit. The shortest person is 10 below the mean height and the tallest is 10 above it, the same distance in opposite directions, which is what lets the two cancel. Every summary that follows, the variance, the covariance and the correlation, is built from these signed distances.

That the deviations sum to zero is not a coincidence of these numbers. It is the balance point from the last section, stated arithmetically, and it holds for every column there has ever been. It is also about to be inconvenient.

The mean alone leaves out something important. Consider two groups of five.

Identical means, and nobody would describe them as the same. One group is nearly uniform and the other ranges over sixty centimetres. Whatever separates them is not centre, and the mean cannot see it at all.

The deviations already see what the mean cannot. Subtract 170 from each value and the two groups stop looking alike.

tightly packed:    −1     0     0     0     1      sum = 0
widely spread:    −30   −15     0    15    30      sum = 0

Both lists sum to zero, as every list of deviations does, so adding them up cannot tell the groups apart either. The sizes can. One group never strays more than a centimetre from its centre and the other strays thirty.

So a second summary is needed, one that answers how far from the centre the values tend to sit. The deviations from section 4 are exactly the raw material for it, and the whole difficulty, which section 6 opens with, is using their sizes without letting their signs cancel.

The obvious first attempt is to average the deviations, and it fails immediately, for the reason section 4 gave.

(−10 + −5 + 0 + 5 + 10) / 5 = 0

It gives zero for every column that has ever existed, because the positive and negative deviations cancel by construction. The cancellation is the problem, so the fix is to stop the signs from cancelling. Two ways suggest themselves, taking the absolute value of each deviation or squaring it, and both are used in practice.

Squaring is the one that leads here. Squares are easier to work with algebraically, which matters once calculus starts differentiating them, and they punish a large deviation far more than several small ones, which is often the behaviour wanted. Averaging the absolute values instead gives the mean absolute deviation, a perfectly respectable statistic that this primer simply does not follow.

sum of squared deviations = 250
variance = 250 / 5 = 50

That is the variance, the average squared deviation from the mean, and dividing by 5 says something specific about what we are doing. We are treating these five people as the complete group we want to describe, not as a sample of anybody else.

population:  σ² = Σ(xᵢ − μ)² / N          divide by 5, giving 50
sample:      s² = Σ(xᵢ − x̄)² / (n − 1)    divide by 4, giving 62.5

Variance has one awkward feature. Squaring the deviations squared the units too, so a variance of 50 is 50 square centimetres, which is not a quantity anyone has intuition about.

Take the square root and the units come back.

standard deviation of height = √50 ≈ 7.07 cm

Taking the square root restores the original unit, so a standard deviation can be compared with the measurements themselves. Repeat the same calculation for weight.

weight variance = 176 / 5 = 35.2 kg²
weight standard deviation = √35.2 ≈ 5.93 kg

Three experiments in the box below separate centre from spread. Move one point a little and both shift slightly. Move one point a long way and the spread jumps while the centre barely stirs. Then drag each of the five points up by the same amount, one after another. The centre travels with them and the spread ends up where it started, give or take the precision of a drag, because a shift shared by everybody changes nobody’s distance from the centre.

Everything so far describes one column at a time. The interesting question sits between the columns.

When a person is taller than average, are they also heavier than average? If they tend to be, then the two quantities carry information about each other, and knowing one tells you something about the other. That is exactly what a prediction is, so a summary of how two columns move together is the last descriptive idea this primer needs.

Look down the two columns together and the pattern is already visible. The shortest person, at 160 cm, is also the lightest at 58 kg. The tallest, at 180 cm, shares the heaviest weight of 74 kg. Each step up in height comes with a step up in weight, apart from the last, where 175 and 180 weigh the same. Nobody is tall and light, and nobody is short and heavy.

What is missing is a number for “tend to”. Reading five rows is easy and reading five thousand is not, which is the problem section 2 raised for one column, now asked of two at once. The summary wanted has to say how strongly the two columns move together, and the next two sections build it out of the deviations each column already has.

The deviations already hold the answer, once they are paired up person by person rather than looked at one column at a time.

Multiply each person’s height deviation by their own weight deviation. Somebody below average in both gives a positive product, since a negative times a negative is positive. Somebody above average in both also gives a positive product. The product only comes out negative when the two disagree, tall but light, or short but heavy.

sum of products = 200

Reading down the table, the 160 cm person is ten below the mean in height and ten below it in weight, and that agreement in both directions gives the largest product, 100. The 170 cm person sits exactly on both means and contributes nothing, since a deviation of zero times anything is zero. A short, heavy person would have contributed a negative product, a positive weight deviation times a negative height one, and would have pulled the sum down.

The products have units of their own, centimetres times kilograms, which is about to matter.

Average those products and the result is the covariance.

covariance = 200 / 5 = 40

Its sign is the part that reads directly. Positive means the two columns tend to move together, negative means one rises as the other falls, and near zero means the agreements and disagreements roughly cancel.

The sum of 200 is also worth remembering. The regression page built that exact number on its way to a slope without giving it a name, and section 14 collects the debt.

The repair follows from what went wrong. Covariance carries the scale of height and the scale of weight, both at once, so dividing by a measure of each column’s own spread should leave the relationship and take the units away.

The standard deviations are exactly those measures, so divide by both.

r = covariance / (SD of height × SD of weight)

r = 40 / (7.07 × 5.93) ≈ 0.953

r = average of  ((xᵢ − x̄)/σₓ) × ((yᵢ − ȳ)/σᵧ)

That pure number is the correlation, written r, and it always lands between −1 and 1. Sign and strength read separately.

r = (2.384 + 0.238 + 0 + 0.715 + 1.430) / 5 = 4.767 / 5 ≈ 0.953

The bounds come from that same table. Each column now has a spread of exactly one, so the products can only average 1 when every person sits in the same place in both columns, as many standard deviations up or down in weight as in height, which is a perfect rising line. Every departure from it, somebody further out in one column than the other, lowers the average, and a perfect falling line pins it at −1.

Our 0.953 says these five people sit very close to a rising line, which the plot confirms at a glance. Drag one person well off the trend in the box above and watch r fall away from 1, then press the button for no pattern and watch it drop near 0.

Two qualifications, and both matter more than the formula.

Now the payoff. The regression page handed you a formula for the slope, derived from calculus, and it looked like a ratio of two sums.

slope = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²

Divide the top and the bottom by the same count of five, which changes nothing, and both halves turn into things this primer has named.

slope = covariance(x, y) / variance(x)
slope = 40 / 50 = 0.8

Read in two steps, that says everything. The covariance measures how height and weight move together. The variance measures how much height moves on its own. Their ratio is how much of the joint movement should be credited to each centimetre of height.

covariance is in  cm · kg
variance is in    cm²

slope is in      cm · kg / cm²  =  kg / cm

One more connection, and this one needs its conditions stated before the identity rather than after it. All of the following have to hold.

Under exactly those conditions,

R² = r²
0.909 = 0.953²

The squaring is where the difference between them sits. Correlation keeps its sign, so it can tell a rising relationship from a falling one. R² measures the share of variation the fit accounts for, which cannot sensibly be negative here, and squaring is what removes the direction.

variation the line accounts for = 0.8² × 250 = 160
total variation in weight = 176
R² = 160 / 176 ≈ 0.909
left over = 176 − 160 = 16

r² = 40² / (50 × 35.2) = 1600 / 1760 ≈ 0.909

Everything up to here described five people. The second half of this primer asks a different question, which is what those five say about anybody else.

Three words keep it straight. The population is the whole group we care about, or more generally the process that produces observations. The sample is what we actually got, our five people. A statistic is a number computed from the sample, and a parameter is the corresponding number for the population, which is usually unknown and is what the statistic is trying to estimate.

Our mean height of 170 was a statistic all along. Whether it is close to the population mean is a question the five people cannot answer by themselves.

One debt from section 6 comes due here. While the five were the whole group, their spread divided by five. The moment they stand for a larger population, the spread being estimated is the population’s, and the sample mean sits a little closer to these five values than the population mean would, so the deviations measured from it run slightly small. Dividing by one fewer corrects for that.

as the whole group:   variance = 250 / 5 = 50       SD ≈ 7.07 cm
as a sample:          variance = 250 / 4 = 62.5     SD ≈ 7.91 cm

The mean needs no such correction, and 170 is the estimate of the population mean either way. Only the spread is adjusted, and only because the centre it is measured from was itself estimated from the same five people.

Statistics looks at observations and reasons back to the process. Probability goes the other way, starting from a process and describing what it might produce, and the second half of this primer needs a little of it.

It is needed for one reason. To say what five people tell us about anybody else, there has to be a way of describing the process that produced the five, and that is what probability provides. The box in section 19 is built on exactly this. A population with a fixed mean height sits underneath it, and every press draws people from that process rather than from a list somebody wrote down.

Four words carry the whole of what is needed here. An outcome is something that could happen. A probability is a number saying how likely an outcome is. A random variable attaches a number to an uncertain outcome. And a distribution says which values that number can take and how often each occurs.

A distribution has more to it than a centre and a spread, and it is worth saying so before any particular shape arrives.

Two columns can have identical means and identical standard deviations and still look nothing alike. One might be symmetric and another lean heavily to one side. One might have a single peak and another two, which usually means two different groups have been mixed together. One might have long tails that occasionally produce extreme values where another never does.

Section 3 already met the lopsided case. Four incomes of 25,000 and one of 500,000 lean entirely to one side, and the mean of 120,000 lands where nobody is. That is skew, and neither a centre nor a spread can report it, since a column shaped symmetrically about 120,000 could be built to share both numbers while looking nothing like those five incomes.

The mean and the standard deviation summarise a distribution. They do not describe it. That is why the shape of a column is worth looking at before either number is trusted, and why section 21 introduces one particular shape as one distribution among many rather than as what data looks like.

Here is the fact that makes the second half of this primer necessary. Collect five people, compute the mean, and it will not be the population mean. Collect five different people and it will be different again.

The box below draws people from a population whose true mean height is 170 cm. Press for more and watch the solid line, the mean of everyone drawn so far, move around the dashed line, which marks the truth.

With a handful of people the solid line wanders noticeably. With a few hundred it settles close to the dashed line and stops making large excursions.

That settling has a name. The law of large numbers says that as the number of independent, representative observations grows, the sample mean tends toward the population mean.

The mechanism is cancellation. Each person drawn is as far from the true mean as people ever are, since the population’s own spread does not shrink. But the people above the truth and the people below it pull the average in opposite directions, and the more of them there are, the more completely those pulls cancel. The typical gap between a sample mean and the population mean shrinks with the square root of the count, which is slower than it sounds.

typical gap ≈ population SD / √count

  5 people:   7 / √5    ≈ 3.1 cm
 40 people:   7 / √40   ≈ 1.1 cm
400 people:   7 / √400  ≈ 0.35 cm

The 7 is the spread of the population behind the box in section 19, which draws heights with a true mean of 170 cm and a standard deviation of 7 cm. Four times the people halves the typical gap, and a hundred times the people cuts it to a tenth. The box opens on forty people already drawn, and their mean reads 167.24 cm, nearly three centimetres under the truth, a larger gap than is usual for forty and exactly the kind of excursion a few more presses settle.

The law is the reason averages are worth computing at all, and the reason more data usually helps. It is also narrower than it first sounds, and the two qualifications are the ones from the last section. The observations have to represent the population you mean to describe, and collecting more of a biased sample does not undo the bias.

The curve in the box above is the normal distribution, the bell shape, and it is worth introducing as one distribution among many rather than as what data looks like.

It is symmetric about its mean, most of its probability sits near the centre, and progressively less lies further into the tails. Two numbers fix it completely. The mean slides the bell left or right, and the standard deviation makes it narrow and tall or wide and flat.

Two different convergence ideas are easy to run together, and keeping them apart is worth a section, because the box above demonstrates one of them and not the other.

The box above draws individual people and piles them into bins, so what fills in as you press is the shape of the population itself, and what settles is the one sample mean. That is the empirical distribution converging, and the law of large numbers alongside it.

It is not the central limit theorem, which is a claim about a different quantity entirely. Demonstrating that one would mean drawing many separate samples, computing the mean of each, and plotting the distribution of those means rather than of the people. The striking part of the result is that this distribution of means comes out approximately normal even when the population it drew from is not, and the population here already is normal, so the box could not show it even in principle.

Everything in the last few sections applies directly to a fitted model, because a fitted coefficient is a statistic like any other.

The training data is the sample. The slope of 0.8 from section 14 was computed from five people, and five different people would have given a different number. More relevant data makes such estimates steadier, for exactly the reason sample means steady. And more data from the wrong source makes the model more confident about the wrong thing.

Every idea in this primer reappears, usually without announcement, in the pages that follow.

Two rows of the table are this primer’s own numbers wearing other names. Standardising a column subtracts its mean and divides by its standard deviation, which for the five heights means 170 and 7.07, and the result is exactly the unit-free column section 12 multiplied by its partner. The slope of 0.8 is the covariance of 40 over the variance of 50 from section 14, so a regression coefficient is a statement about how two columns move together, scaled by how much one of them moves alone.

The last row is the second half of the primer in one line. A score measured on the rows a model was fitted to says how well it summarised its sample, and the question that matters is how it would have come out on different rows, which is why data is held back before fitting.