Linear Algebra Primer

A person may arrive as height, weight, and age; an image arrives as many pixels. A model needs to keep related numbers together without losing what each position means.

Linear algebra describes those arrangements and the operations connecting them. We will start with the objects before the notation.

One observation usually contains several measurements. A person might be recorded as a height, a weight and an age, and those three numbers describe one example rather than three, so they should travel through a model together rather than separately.

(170, 72, 35)

That grouping is a vector. Three things are true of it and worth fixing now. Each position has a defined meaning, so the first entry is the height and stays the height. The order cannot be shuffled, because shuffling it would silently change what every entry means. And the number of entries is called the dimension, so this is a three-dimensional vector.

It is tempting to define a vector as “a list of numbers” and stop, though a list is only how it is written down. The useful idea is that several related quantities have been packaged into one mathematical object, so that one symbol can stand for a whole observation and one operation can act on all of it at once.

A two-entry vector also has a picture, drawn as a point or an arrow on a pair of axes. We stay in two dimensions throughout this primer for exactly that reason, and the geometry is worth having because every operation we see there keeps working in dimensions nobody can draw.

The pair (3, 4) can be read two ways, and both are used constantly.

Read as a location, it is a position in the plane, three across and four up from the origin. Read as a movement, it is a displacement, three across and four up from wherever you happen to be standing. The numbers are identical either way.

Below is one arrow on its own. Drag the tip, and watch the two dashed walks change. The horizontal walk is the first entry and the vertical walk is the second, so moving the tip sideways changes one number and leaves the other alone.

Two things can be done to vectors before anything is measured on them, and everything later is built out of the pair. They can be added, and they can be scaled.

Adding two vectors means doing one movement and then the other, and in coordinates it is as simple as it sounds. Add the matching entries.

(a₁, a₂) + (b₁, b₂) = (a₁ + b₁,  a₂ + b₂)

(3, 4) + (1, −2) = (3 + 1,  4 − 2) = (4, 2)

Scaling means multiplying a vector by a single number, which stretches or shrinks it without changing the line it lies on.

c·(a₁, a₂) = (c·a₁,  c·a₂)

These two operations are the whole of what makes the rest possible. Weighted combinations, matrix transformations, eigenvectors and the parameter update every model performs are all additions and scalings wearing different names.

Section 3 could stretch an arrow and could add two of them, and never once said how big an arrow is. A model needs that number constantly. How far a point sits from the origin, how big a parameter set is, and in the next section how far apart two examples are, are all the same measurement, and it is called the length.

A vector’s entries describe its movement along each axis separately. Its length is the direct distance from its start to its end, and that is a right triangle, so Pythagoras measures it. The triangle is right-angled because the horizontal walk and the vertical walk run along the two axes, which meet at a right angle, so the arrow itself is the hypotenuse.

horizontal component  3
vertical component    4

length = √(3² + 4²) = √(9 + 16) = √25 = 5

Written generally, with the bars meaning length,

‖v‖ = √(v₁² + v₂²)

Run the table from section 3 through it and the lengths behave the way the table said. Scaling by a number multiplies the length by the size of that number, and the sign is invisible to length, which is why the table could say same length, pointing the opposite way.

Three dimensions add another squared component under the root, and higher dimensions carry on the same way, one squared entry each. The picture stops being drawable long before the arithmetic stops working.

‖(2, 3, 6)‖ = √(2² + 3² + 6²) = √(4 + 9 + 36) = √49 = 7

‖v‖ = √(v₁² + v₂² + … + vₙ²)

Before measuring the distance between two points, it is worth being explicit about subtraction, because distance is built directly on it.

Take the points a = (1, 2) and b = (4, 6), and ask what movement carries a to b. Subtract the starting coordinates from the ending ones.

b − a = (4 − 1,  6 − 2) = (3, 4)

Distance is then one more step. Find the difference, and measure its length.

distance(a, b) = ‖b − a‖

‖(4, 6) − (1, 2)‖ = ‖(3, 4)‖ = 5

The box below has two arrows, a in indigo and b in amber. Drag either tip and watch the two lengths and the distance between the tips follow. The dot product readout is the subject of section 8.

Two of the models ahead run almost entirely on the previous section. An example, once its measurements are collected into a vector, is a point in a space with one axis per feature, and examples that resemble each other land near each other in it.

Here is a calculation that appears inside almost every model, written out before it is given a name.

A prediction is rarely made from one measurement, and when it is made from several there is no reason every one of them should count equally. What is missing is a way to let each feature count by its own amount. The fix is one number per feature, called its weight, saying how strongly that feature should count.

Suppose a prediction uses two feature values and carries one weight for each.

3 × 4  =  12      first feature times its weight
4 × 3  =  12      second feature times its weight
         ────
           24      added together

The claims in that last sentence are easiest to believe by holding the features at (3, 4), keeping the first weight at 4, and changing only the second.

The features never changed. Everything the prediction did differently came from the weights, which is why the weights are what a model has to learn and the features are what it is given.

That is the shape of a linear model’s prediction, and with ten features it is ten multiplications and nine additions written out in a row. The next section is the notation that compresses it.

The operation from the last section has a name. The dot product takes two vectors of the same length and produces a single number, by multiplying the entries position by position and adding.

a · b = a₁b₁ + a₂b₂

(3, 4) · (4, 3) = 3·4 + 4·3 = 12 + 12 = 24

For any number of entries, written with a summation sign,

a · b = Σᵢ aᵢbᵢ

The two vectors must have the same number of entries, and the reason is not a technicality. Every feature needs a weight of its own, so a feature with no matching weight would have nothing to multiply by and no way to contribute.

score = x · w + b

The dot product also has a geometric reading, and it needs stating carefully, because the loose version of it is misleading.

Hold both arrows at fixed lengths and swing one of them. The dot product is largest when they point the same way, falls as they open out, passes through exactly zero when they are perpendicular, and goes negative once they oppose each other. Try it in the box above by dragging b to (−4, 3), a quarter turn from a, and the readout shows 0.

Now change a length rather than an angle, and the value moves too. So the dot product is reporting two things at once, how aligned the arrows are and how long they are, and it does not separate them.

a · b = ‖a‖ · ‖b‖ · cos θ

(10, 0) · (10, 10) = 100    lengths 10 and 14.14, cos θ = 0.71
(2, 0)  · (2, 0)   =   4    lengths 2 and 2,      cos θ = 1.00

cos θ = (a · b) / (‖a‖ · ‖b‖)

One consequence ties the sections together. Dot a vector with itself. It is perfectly aligned with itself, so the cosine is 1, and the recipe multiplies each entry by itself and adds, which is the squared-length calculation from section 4.

v · v = v₁² + v₂² = ‖v‖²

For the vector (3, 4), its dot product with itself gives the squared length.

(3, 4) · (3, 4) = 3² + 4² = 9 + 16 = 25
length = √25 = 5

The dot product therefore connects three ideas: length, distance and alignment.

A dot product takes a vector and produces one number. A layer of a network, or any transformation of one representation into another, needs several numbers out rather than one.

The fix is unglamorous. Give each output its own weight vector, and do one dot product per output. Suppose two outputs are wanted from two inputs.

o₁ = 2x + 1y
o₂ = 1x + 2y

Each line is a weighted sum with its own pair of weights, (2, 1) for the first and (1, 2) for the second. Collect those pairs as rows, and the collection is a matrix.

A = ⎡ 2  1 ⎤
    ⎣ 1  2 ⎦

So a matrix is not a new idea so much as a container. It packages several related weighted sums so they can be written, and performed, as one operation.

Rows rather than columns, because a row is what meets the input. Each row is read against the input vector exactly as section 8 read a weight vector against a feature vector, so writing the weights as rows keeps every one of the dot products visible on its own line.

A matrix is a rectangle of numbers, described by how many rows and columns it has, in that order. The matrix above has two rows and two columns, so it is a 2 × 2.

The shape says what it accepts and what it returns, and the rule falls straight out of the last section. Each row is one output’s weight vector, so the number of rows is the number of outputs. Each row has one weight per input, so the number of columns is the number of inputs.

That is the convention used throughout this site and on the neural network pages later, rows correspond to outputs and columns correspond to inputs. A 3 × 2 matrix would take a two-entry vector and return a three-entry one.

⎡ 1  0 ⎤              (1, 0) · (3, 1) = 3
⎢ 0  1 ⎥ · (3, 1)      (0, 1) · (3, 1) = 1
⎣ 1  1 ⎦              (1, 1) · (3, 1) = 4

result (3, 1, 4)

Applying a matrix to a vector is the two dot products of section 10, carried out and stacked.

A = ⎡ 2  1 ⎤        A·(x, y) = (2x + 1y,  1x + 2y)
    ⎣ 1  2 ⎦

first row  (2, 1) · (3, 1) = 2·3 + 1·1 = 7
second row (1, 2) · (3, 1) = 1·3 + 2·1 = 5

A·(3, 1) = (7, 5)

Each output came from one row, and each row’s contribution was a dot product with the input. That is the whole operation, and it is why the number of columns has to match the number of entries going in.

Written for any shape, the output’s entry i is row i dotted with the input, and the sum runs along the row, one term per column.

(A·v)ᵢ = Σⱼ Aᵢⱼ·vⱼ

Two things fall out of that line. The output has one entry per row of A, so its dimension is the row count whatever the input’s was. And a matrix with a single row is just a weight vector, so the dot product of section 8 is the one-output case of this operation rather than a different operation.

outputs = A·x + b

Two particular vectors are worth naming, because everything else is built from them. Call them the basis vectors.

e₁ = (1, 0)      e₂ = (0, 1)

One step east and one step north. They matter because every vector in the plane is a scaled copy of the first plus a scaled copy of the second, which is what its coordinates have been saying all along.

(x, y) = x·e₁ + y·e₂

Now send both of them through the matrix.

A·e₁ = A·(1, 0) = (2, 1)
A·e₂ = A·(0, 1) = (1, 2)

And because a matrix distributes over addition and scaling, the two operations from section 3, knowing where the basis vectors land is enough to know where anything lands.

A·(x, y) = x·(A·e₁) + y·(A·e₂)

3·(2, 1) + 1·(1, 2) = (6, 3) + (1, 2) = (7, 5)

This is what justifies the claim that a matrix is completely described by its columns. Two arrows fix the whole transformation.

Since the basis vectors fix everything, the natural picture is not one arrow moving but the entire plane being carried somewhere. The box below draws it. The faint grid is the plane before, the darker one is after, the two arrows are where the basis vectors land, and the shaded shape is what became of the unit square.

The presets are worth walking one at a time, because the four families cover most of what a matrix can do. Scaling stretches along the axes. Reflection flips the plane across a line. Rotation turns everything about the origin by a fixed angle. Shearing slides one axis along while leaving the other where it was, which is what turns a square into a leaning parallelogram.

Take one arrow, send it through A, and compare where it pointed with where its image points. Usually the two differ, because the matrix has both stretched the arrow and turned it.

Which raises a question worth asking before any terminology arrives. Are there directions this matrix stretches without turning at all?

A·(1, 1) = (2·1 + 1·1,  1·1 + 2·1) = (3, 3)

A direction with that property has a name, and so does the number. The direction is an eigenvector of the matrix, and the factor it is scaled by, 3 here, is the matching eigenvalue.

This matrix has a second one. Try (1, −1) and it comes back as (1, −1) exactly, an eigenvector with eigenvalue 1, a direction the matrix leaves completely alone.

Drag the indigo arrow in the box below and watch the amber image swing away from it, then sweep onto one of the dashed lines and see the two fall into line.

“Stretch factor” undersells the eigenvalue, because its size and its sign say different things and it can do more than stretch.

Every row of that table can be seen in the box in section 14, where the two arrows show where the basis vectors land. The matrices below are chosen so that the basis vectors themselves are the preserved directions, which makes each case checkable by hand from section 13.

⎡ 2   0  ⎤ · (1, 0) = (2, 0)    = 2 · (1, 0)
⎣ 0  0.5 ⎦ · (0, 1) = (0, 0.5)  = 0.5 · (0, 1)

The negative case is the one that needs the earlier phrasing tightened. An eigenvector is not quite a direction the matrix leaves pointing the same way, it is a direction whose line the matrix preserves. A negative eigenvalue flips the arrow end for end while leaving it on that same line, which still counts.

Size and sign are read differently once the matrix comes from data rather than from a preset. On the PCA page every eigenvalue is an amount of variation along its direction, so there the size is what ranks the directions, which is where section 18 ends up.

Now that the idea has been seen and computed, the notation adds nothing new. It says exactly what section 15 observed.

A·v = λ·v

The eigenvector equation says that multiplying by the matrix preserves the vector’s axis and scales its magnitude. For this matrix, the vector (1, 1) is multiplied by three.

A(1, 1) = (3, 3) = 3(1, 1)

The eigenvalue is three.

Finding eigenvectors algebraically is a separate skill, and this primer does not need it. Recognising what one is is enough for everything that follows.

Here is why any of the last four sections matter, kept to what the later page will actually use.

A dataset usually varies more in some directions than others, and those directions need not line up with the features that were measured. When two features move together, height and weight say, the direction of greatest variation runs diagonally between their axes rather than along either of them.

Principal component analysis builds a matrix describing how each pair of features varies together, called a covariance matrix. That matrix has eigenvectors, and they turn out to name directions the data varies along independently, each eigenvalue saying how much variation lies along its direction.

Everything in this primer reappears, in this order, across the pages that follow.

Read down the table and the pieces chain into one picture of a model. A dataset is a matrix whose rows are example vectors. A model’s parameters are a vector with one entry per feature. A prediction for one example is the dot product of the two plus an offset, the line from section 8. A layer of a network is a matrix that carries that row to a new one, section 12. And principal component analysis takes the eigenvectors of a matrix built from the dataset, section 18. Each page that follows is one of those, said in its own vocabulary.