A Vector for a Word

A tokenizer finishes by handing every piece of text a number, and that number is a name. It says which row of a lookup table the word lives in, so we can check whether two pieces of text are the same piece of text, and that is genuinely all it promises. Nothing about the numbers being ordered makes the words ordered, and nothing about two numbers being close makes the words close, which is easy to say and worth measuring before we accept it.

The corpus this page works on is twenty-four short documents, twelve of them about cooking and twelve about sailing, and they hold twenty-three distinct words between them. The numbers are dealt out commonest first, so a number does carry exactly one fact about a word, which is roughly how common it was. What it does not carry is anything about what the word means.

Over all 253 pairs of words, the correlation between how far apart two words’ numbers are and how alike they later turn out to be is −0.0852, which is small enough that the scatter of those pairs shows no slope at all.

Fitting the table…

Every word of the table is a direction, drawn on the sphere those directions live on. Coloured by subject the words fall into two caps on opposite sides. Coloured by token number, which is the same twenty-three dots in the same places, the colours are scattered over the whole sphere, because the number a word was dealt says nothing about where it ended up.

The corpus is a design and not a piece of prose, and saying so early saves confusion later. Each document is six words long, each topic is spelled from ten words of its own, and the two topics share only the words and, the and we, one per document. Within a topic the documents step through the ten words in a cycle, so that every topic word keeps the same kind of company as every other one. A cooking document reads oven bake stir the whisk dough and a sailing document reads harbour anchor and tide mast rope.

The reason to use something so plain is that we have to be able to judge the answers. When the page reports that sail came out near rope and far from oven, that is checkable against the texts in a way that a result on a corpus of a billion words is not, and where the design produces an answer a reader would not have predicted the page says so instead of quietly choosing a different example.

Replace each word’s single number with several numbers, and read those numbers as coordinates. The word is then a point somewhere, and points can be compared in ways names cannot, since two points have a direction from one to the other and an amount of separation between them. That is the whole idea, and everything the rest of the page does is a consequence of it.

How the coordinates are worked out is a separate subject with pages of its own. For the record, the ones used here come from counting how often each pair of words falls within two positions of each other, scoring each pair by how much more often that happened than chance would explain, and squeezing the resulting table down to four numbers per word. Nothing else on this page leans on that choice, and Part 5 turns the whole table to prove it.

sail = (0.5369, 0.5223, 0.2690, −0.2075)

It is tempting to look at the first coordinate of sail, which is 0.5369, and ask what it measures. The answer is that it measures nothing on its own. It is not a score for how nautical a word is, and it does not become one if we line up the words with a large first coordinate and hunt for what they have in common. The coordinates exist so pairs of them can be multiplied together, and what the fit determined was the collection of those products and not the individual numbers.

This is not a philosophical caution, it is a checkable fact about the table, and Part 5 checks it by turning the whole space and watching every coordinate move while every comparison between words stays exactly where it was. Until then it is enough to read a coordinate as a bookkeeping entry and to treat only the comparisons as meaningful.

Given two points there are two obvious readings of close. One is the ordinary distance between them, the length of the line joining the two points, which is what a ruler would say. The other is the angle between the two arrows drawn from the origin out to each point, which asks whether they lie in the same direction and ignores how far along that direction each one has got.

The two readings genuinely disagree, and the disagreement is not rare or delicate. A word placed far out along a direction and a word placed just a little way along the same direction point identically and sit a long way apart, so the ruler calls them distant and the angle calls them identical. Which of the two we should use depends on which quantity, the direction or the distance from the origin, was the one the fit was determining, and that question has an answer.

ordinary distance between a and b = √( Σᵢ (aᵢ − bᵢ)² )

The angle between two arrows is got from their dot product, which is the sum of the products of matching coordinates, divided by the two lengths. Dividing by the lengths is what removes them from the answer, and what is left is the cosine of the angle, a number between −1 and 1. One means the two arrows point the same way, zero means they are at right angles, and −1 means they point in opposite directions.

cosine(a, b) = (a · b) / (‖a‖ ‖b‖),   where a · b = Σᵢ aᵢ bᵢ

The two bars around a letter stand for the length of its arrow, which is the ordinary distance from the origin out to the point, the square root of the sum of its squared coordinates. For sail, whose four numbers Part 1 gave, the length comes out a little under one.

‖a‖ = √( Σᵢ aᵢ² )
‖sail‖ = √( 0.5369² + 0.5223² + 0.2690² + (−0.2075)² ) ≈ √0.6765 ≈ 0.8225

Three things about that formula are worth stating before we use it. It is symmetric, so the order of the two words does not matter. It is entirely blind to length, so doubling every coordinate of one word changes nothing. And it is undefined when either arrow has no length at all, since a point sitting exactly at the origin has no direction to compare, and the correct behaviour there is to refuse, and never to answer zero, which would falsely report the word as being at right angles to everything.

Four numbers per word is already too many to picture, so before we trust the measure on the real table let us check it on a drawing. Five words in two dimensions, arranged so that every answer can be read off graph paper. Two of them lie along the horizontal axis at different distances out, two lie along the vertical axis at different distances out, and the fifth sits at the origin with no direction at all.

king · queen = 1 × 2 + 0 × 0 = 2
cosine similarity = 2 / (1 × 2) = 1.0000

king · man = 1 × 0 + 0 × 1 = 0
cosine(king, man) = 0 / (1 × 1) = 0.0000

man · woman = 0 × 0 + 1 × 3 = 3
cosine(man, woman) = 3 / (1 × 3) = 1.0000

distance(king, man) = √( (1 − 0)² + (0 − 1)² ) = √2 ≈ 1.4142
distance(man, woman) = √( (0 − 0)² + (1 − 3)² ) = √4 = 2.0000

The same disagreement shows up on the real table, and it shows up at the very top of a ranking and not somewhere down in the noise. Stand at sail, whose own position has length 0.8225, and ask what is nearest. By angle the answer is rope, at a cosine of 0.9825, with mast second at 0.9445. By the ruler the order is reversed, mast at a gap of 0.2750 and rope at 0.3046.

The reason is visible in the third column. rope has length 1.0713 and mast has length 0.8281, and sail is 0.8225 long, so mast is at almost exactly the same distance from the origin as sail is while rope is well beyond it. The ruler charges rope for that extra reach even though the two words point more nearly the same way than sail and mast do, and the question is whether the extra reach is a fact about meaning.

sail = (0.5369, 0.5223, 0.2690, −0.2075)     length 0.8225
rope = (0.5948, 0.6715, 0.3906, −0.4364)     length 1.0713

sail · rope ≈ 0.3193 + 0.3507 + 0.1051 + 0.0906 = 0.8657
cosine ≈ 0.8657 / (0.8225 × 1.0713) ≈ 0.9825

…

The same word’s neighbours ranked twice. Rows in pink changed place between the two columns, and the length column is where the explanation lives.

So what is a vector’s length a fact about? The usual answer, and the one I expected to be able to report, is that it tracks how often the word was seen, because a common word is adjusted more times during a fit and so drifts further from wherever it started. On this table the relationship is real and runs the other way. The correlation between how often a word was used and how long its vector came out is −0.6284, which is to say the commoner a word is here, the shorter its position.

The three commonest words are and, the and we, used eight times each, and they have the three shortest positions in the table at 0.3708, 0.3667 and 0.5770. Words like deck and dough, used five times, are the longest at 1.1628. That is a consequence of how these particular coordinates were worked out, since a word that turns up beside everything is not surprising beside anything, and a scoring rule built on surprise gives it very little to say. A different way of finding the coordinates would move this number and could easily reverse its sign.

…

Each dot is a word, placed by how often the texts used it and how long its position came out. The three dots on the right, low down, are the words both topics share.

The direction of that correlation is not the point, and it would be a mistake to carry away the number rather than the argument. What matters is that the length is answering a question about the word’s frequency and distinctiveness, which is a different question from the one we are asking when we ask whether two words resemble each other, and dividing it out is how we stop it from contributing to an answer it has no business in.

Once every word has a position and we have a way of scoring a pair, the first new question falls out. Take a word, score it against every other word in the vocabulary, sort, and read off the top of the list. Nothing in the fit was asked to produce a list of neighbours; the list is a consequence of every word having been placed somewhere, which is why the answer is worth looking at and is not merely definitional.

A list of neighbours looks convincing whatever it contains, so it is worth checking it in both directions. In the direction that works, sail and rope score 0.9825 while sail and flour score 0.0661 and sail and oven score −0.0518. Words from the same topic are near one, words from different topics are near zero, and the fit was never told which topic any word belonged to.

In the direction that does not work, sail and boat score only 0.3638, which is far lower than a reader who knows English would predict for two words that name parts of the same activity. That is a fact about the corpus and not about the method, and tracing it turns up something more interesting than the miss itself.

Counting the company each word keeps, with the same window of two positions the fit used, gives the explanation and overturns the obvious guess along the way. sail and rope, at a cosine of 0.9825, never occur within two positions of each other anywhere in the twenty-four documents, so their direct company is exactly 0. What they do share is the company they each keep with crew and deck. Meanwhile sail and boat do occur together, twice, and still come out at 0.3638, because boat’s heaviest company is harbour and anchor, neither of which sail ever meets in these texts.

…

The company each word of the pair keeps, with the words both of them were seen beside picked out in colour. Switch to sail and boat and watch the direct count go up while the cosine goes down.

That is worth pausing on, since it is easy to read a cosine as a count of how often two words appeared together and it is not. Two words are placed together when they are used in the same kind of position, whether or not they were ever used in the same breath, and it is the second-hand evidence that carries most of the work here. It also sets the honest limit on what a high cosine claims, which is similar company and not similar meaning.

The second question is the smaller one, and it is worth separating from the first because it is the one that gets used inside larger systems. Given two words we already have in mind, how alike are they? That is a single number, computed without touching the rest of the vocabulary, and it can be compared across pairs because the scale is fixed at both ends.

Being able to compare across pairs is the part that a number could never give us. Saying that oven and bake score 0.9267 while sail and flour score 0.0661 puts the two pairs on one scale, and a system downstream can set a threshold on that scale and have it mean roughly the same thing everywhere in the vocabulary.

Some questions we can put to a table of positions have no answer, and the reasons are arithmetic ones and not matters of policy. A word the texts never contained has no row to read, and there is nothing to return, since the whole content of a position is what the fit learned about that word and it learned nothing. Asking for the position of kettle here is that case.

The other case is a point sitting exactly at the origin. It has no direction, so the angle between it and anything else does not exist, and returning zero would be a claim that it stands at right angles to every word, which is a much stronger statement than the truth. The consistent treatment is to refuse the comparison and to leave such a word out of every neighbour list, which is why the fifth word of the sketch never appears in an answer.

…

Each row is a question put to one of the two tables. Some have an answer and some do not, and the ones that do not are undefined rather than merely awkward.

A dense embedding table stores one row per word and one value per coordinate.

example table = 23 words × 4 coordinates = 92 values
larger table = 50,000 × 300 = 15,000,000 values

Compared with storing one ID per word, the number of coordinates determines the storage multiplier. Increasing the vocabulary does not reduce that multiplier.

Answering a question costs the whole table as well. A nearest-neighbour query here scores the word against all 22 others before it can sort, and in general it scans every row, because a cosine gives no way to rule a word out without computing it. That is the honest place where the older approach wins. Checking whether two pieces of text are the same word is one integer comparison and it is exact, where the nearest-neighbour question is a scan and its answer is a ranking that can be argued with.

The observation that made these tables famous is about differences between positions. If the step from one word to a second word is much the same step as the one from a third word to a fourth, then taking the first word, undoing the first step and applying it to the third should land near the fourth. Written as a sentence, that is the claim that the difference between two positions can stand for a relation between two words.

Nothing in fitting a table asks for this. A fit is told to place words so that words keeping the same company end up together, and whether the resulting arrangement also lines up parallel differences is something to be checked afterwards and is not guaranteed. It holds well on large tables and it holds patchily on small ones, and this page reports both.

answer ≈ nearest word to  (first + third − second)

The arithmetic is done on unit vectors rather than on the positions themselves. Each of the three words is first divided by its own length, so that it becomes an arrow of length one pointing the same way, and only then are the three added and subtracted. The standard form is called 3CosAdd, after Levy and Goldberg, and the reason for the normalising is the measurement of Part 2.

If the raw positions were added, a word that happens to be long would dominate the sum, and we have already established that length here records how ordinary a word is rather than what it means. The result would be an answer driven by which of the three words was the rarest. Dividing each one by its length gives all three an equal say, so the sum is a statement about directions alone.

q = first/‖first‖ − second/‖second‖ + third/‖third‖

The five-word drawing is small enough to follow every step, so let us put the famous question to it. What is to woman as king is to man? The three positions are (1, 0), (0, 1) and (0, 3), and their lengths are 1, 1 and 3.

…

Every intermediate of the calculation, on the drawing. Untick the switch to see what the next step is about.

Notice what would happen if we did not strike out the three words of the question. The sum came to (1, 0), which is exactly king’s own direction, so king scores a cosine of 1.0000 against it and takes first place, with queen tying behind. The analogy appears to answer itself, and the answer carries no information at all.

This happens because the word that was added first is a whole unit of the sum, and the other two frequently cancel a good part of each other. The published accounts of the analogy result usually say that the question words are excluded and leave it at that, so I measured how often it actually matters.

Over every ordered triple of the twenty-three words, which is 10,626 questions, the top answer without exclusion is one of the three words already in the question 4,118 times, a share of 0.3875. That is often and not always, and it is worth saying so plainly instead of repeating the stronger claim.

The split within those 4,118 is the more interesting half. 4,048 of them return one of the two words that were added and only 70 return the word that was subtracted, which is what the arithmetic predicts, since subtracting a word’s direction pushes the sum away from it rather than towards it.

…

The bar is every question asked without exclusion, split by what came back. The three rows underneath are single questions asked both ways.

The sketch was built so the analogy would work. The documents were not, and putting the same machinery to them is where the claim gets tested. The corpus has a parallel structure by construction, since each topic steps through its ten words in the same cycle, so flour occupies the position in cooking that sail occupies in sailing, and sugar the position that wind does. The question “what is to sugar as sail is to flour” therefore has a designed answer, which is wind.

It does not come out. The sum lands nearest rope at 0.7847, then deck at 0.7323 and mast at 0.7210, and wind is nowhere in the first three. All three answers are sailing words, so the arithmetic did carry the topic across correctly, and what it did not carry is the position within the cycle. On a table of twenty-three words fitted to a hundred and forty-four word occurrences, the differences between positions are simply not determined finely enough to encode which step of a cycle a word sits at.

…

The same calculation on the fitted table. The topic transfers and the position within the topic does not.

Part 1 claimed that a single coordinate names nothing, and here is the demonstration. Take the fitted table and turn every word by the same angle in the plane of its first two coordinates, in the way a sheet of graph paper can be rotated under a set of fixed points. Every word moves in the same way, and every coordinate of every word changes.

At a quarter turn, sail’s first coordinate goes from 0.5369 to −0.5223, which is a swing across most of the range the table uses and includes a change of sign. Its length does not move at all, and neither does any pair’s cosine, the largest change over all 253 pairs being smaller than 1e-15, which is rounding and not a change. All 23 words keep the same nearest word.

…

Drag the slider and watch the top row of dots become the bottom row. The two panels at the bottom hold the neighbour lists before and after, and they do not change.

The turned table is not a damaged copy of the original. It is a table that gives the same answer to every question we know how to ask, which means that whatever the fit determined, it did not determine the coordinates. It determined the dot products, and the coordinates are one of infinitely many ways of writing those down.

Reading a coordinate as a named feature is ruled out by that, since a turn destroys the reading without changing anything a reader could measure. So is any arithmetic that puts two tables’ coordinates side by side, whether that is comparing a word’s third coordinate here against its third coordinate there or adding one table’s position for a word to another table’s, since the two are written along axes that have nothing to do with each other.

One small thing does change under a turn, and it is worth reporting because it looks like a contradiction and is not. The full ranking of neighbours survives for 21 of the 23 words and not for all of them. The two exceptions swap a pair of words whose cosines are equal to sixteen digits, so what moved is which of two exactly tied words the sort happened to put first, and a tie has to be broken somehow.

The definition gives each word in the vocabulary exactly one position, and a vocabulary is a list of spellings. Where one spelling is used for two unrelated things, the fit has one point to describe both, and the arithmetic that places it is driven by every context it appeared in at once. There is no choice available, since a single point that has to be near two separated groups ends up between them, at a middling distance from each and inside neither.

I built the case rather than waiting for it, since the documents have no ambiguous word of their own. Every use of pan in the cooking texts and of rope in the sailing texts was respelled sheet, which is a real ambiguity in English, since a sheet is a baking tray in one trade and a rope that controls a sail in the other. The texts are otherwise untouched and the fit is the same one.

…

What the merged spelling is now nearest, against what the two words it replaced were nearest when they were separate.

The result is what the geometry forces. Where pan scored 0.9981 against its nearest word and rope scored 0.9825 against its own, sheet reaches only 0.8635, and its list of neighbours alternates between the two topics. Its mean cosine to the cooking words is 0.3504 and to the sailing words 0.4318, which are nearly the same number, where an ordinary cooking word like dough scores 0.4015 to its own topic and 0.1481 to the other. The merged word can no longer say which topic it belongs to, and nothing in the fit reported that anything had gone wrong.

Part 5 showed that a table has no fixed axes, and this is the practical consequence. Two fits of the same texts that both work will not agree about coordinates, so no quantity built out of coordinates directly can be carried from one to the other, and any comparison between two tables has to be made out of quantities a turn leaves alone.

To measure that rather than assert it, the same twenty-four documents were fitted twice by a different technique, one that begins from random positions, changing nothing but the seed. Both fits found the shape that is there, putting two words of one topic nearer each other than two words of different topics, at 0.8333 against 0.7313 in the first fit and 0.8676 against 0.5593 in the second. On the coordinates they agree about nothing. The word sail in one fit has a cosine of 0.4954 to the same word in the other, and the two fits put a word’s first coordinate 0.3292 apart on average and 0.8760 apart at worst.

…

The same texts fitted twice from different starts. Both found the topics; neither agrees with the other about where anything is.

On this corpus the two fits do not even agree about the neighbour lists, sharing 28 of the 69 nearest-word places, and the second fit separated the topics more sharply than the first. That is an honest complication rather than the claim I set out to make, and its cause is that twenty-four documents do not pin down a table well enough for a random start to be washed out. The narrower point survives it, since even two fits that agreed perfectly about every neighbour would still be written in unrelated axes.

The last limit is the one that follows from the definition most directly and gets discussed least clearly. Nearness in one of these tables is defined by what kept company with what in the texts that were read. It is a summary of usage, and it is a summary of the usage in that particular collection, so every regularity of those texts, including the ones nobody intended, is in the answers.

The corpus here shows the mechanism at a size we can check. sail and boat score 0.3638, which is wrong about English and right about these twenty-four documents, where the heaviest company boat keeps is harbour at 6.0 and anchor at 3.0, and sail never once appears near either of them. Nothing in the fit was mistaken. The texts placed the two words in different company, the fit reported it, and a system reading the answer downstream has no way of knowing that the difference came from how these particular documents were composed.

Scale changes the size of the effect and not its nature. A much larger collection replaces the artefacts of a designed corpus with the habits of whoever wrote the collection, and a habit that shows up consistently in the writing shows up as a small angle between the words involved. What makes this awkward is that the resulting opinion is not held anywhere in particular.

It is distributed across every coordinate of every affected word, so there is no entry to correct, and Part 5 has already shown why editing coordinates directly is not a coherent operation.