Moving a Vocabulary

We need two corpora rather than one, and they should differ in what they are about while agreeing on the letters they are written in. So the first is the eighteen sentences the byte pair page already learns from, about reports and costs and analyses, and the second is eighteen sentences I wrote about bread and pastry.

Every character the second uses appears in the first, which was arranged on purpose, since a letter the old corpus never contained would be unreadable for a reason that has nothing to do with moving a vocabulary. What the two do not share is their words.

Fit the first corpus until it runs out of pairs it has seen twice and it gives 137 rows from 85 merges. Hand it the second corpus and it reads it in 428 pieces. Fit the second corpus on its own and it gives 125 rows from 79 merges, and reads itself in 276.

Half again as long is not a rounding error, and it is the ordinary situation rather than a contrived one. The first vocabulary spent its 85 merges on analysis and estimate and expected, and none of them on dough or butter or pastry, so every word of the second corpus is assembled out of pieces that were frequent somewhere else.

Learning the second vocabulary took no time worth measuring. Counting pairs over eighteen sentences 79 times is arithmetic a person could do by hand in an afternoon and a computer does before the request finishes. If that were the whole cost, nobody would ever keep a vocabulary that did not suit their text.

What costs something is the thing that reads the vocabulary. Whatever model sits behind the tokenizer holds one row of numbers per token, and those numbers are learned from the text rather than counted from it. Give every row 256 numbers, which is a width I have chosen here so that the count can be a number rather than an adjective, and the new vocabulary is a table of 32,000 numbers, every one of which would start from nothing.

And the table is only the entrance. Everything behind it was trained against the rows the old table produced, so a table redrawn from noise makes the model’s first layer unreadable to its second, and the training that follows is not a small correction. So the cost of changing a vocabulary is 79 merges on one side of the tokenizer and 32,000 numbers plus everything trained against them on the other, and it is that gap the method is aimed at.

Written down plainly there are three things one can do with a model whose vocabulary no longer suits its data, and it is worth setting them beside each other before any mechanism, because the method below is the third of them and its whole claim is about the second.

The third row is doing something that reads as too good until one sees where the work went. Nothing is being learned; the method only answers, for every token of the new vocabulary, which rows of the old table it corresponds to. What it cannot do is invent information the old table never held, and the rest of this page is about exactly where that limit falls.

A vocabulary hands a text to a model as a list of positions, and the model finds row 60 of its table by counting to 60. That works because within one vocabulary a position means one thing, and it stops working the moment there are two vocabularies, since the number 60 was assigned by whichever merge happened to be learned sixtieth.

So there is nothing to carry across by number, and the question the method has to answer is what a token of the new vocabulary corresponds to in the old one. The only thing the two vocabularies have in common is that both are made of strings, so whatever is going to travel has to travel by spelling.

For each token of the new vocabulary in turn, three rules are tried and the first that applies answers. The first is about the token that means “something was here and I cannot say what”, which is a label rather than text and whose opposite number in the old vocabulary is the token with the same job, whatever either of them is spelled. The second is the copy rule, which fires when the old vocabulary holds this exact spelling, marker and all. The third is everything else.

for each token t of the new vocabulary

  1.  t is the label for the unspellable        →   the old label
  2.  t is spelled in the old vocabulary too    →   that one row
  3.  otherwise                                 →   read the text of t with the old vocabulary

The third rule is the interesting one and it has two steps. Decoding turns the token into the text it stands for, and then the old vocabulary encodes that text and hands back however many pieces it needs. So a token that the old vocabulary never held is seeded from the rows of the pieces that spell it, which is a statement the old model can actually be asked to make.

It looks at first as though the second rule is a shortcut for the third, since a token the old vocabulary already holds would surely come back as itself when read again. It does not, and the reason is the whole of Part 4, so here it is enough to see it once on one piece.

The four-word corpus of the byte pair page teaches a piece made of an l and an o that never ends a word, because the words it comes from are low and lower and the l and the o are followed by a w. Both vocabularies below hold it. Under the copy rule it maps to itself, one row, exactly.

Under the third rule it would first become the two letters as plain text, and those two letters read as text are a whole word ending in o, which is a different thing and, on that corpus, an unspellable one.

What comes back from the third rule is a list of old positions, and the model half of the method fills the new row with the average of those old rows. That is the step the method is named for, since an average of several vectors is arithmetic rather than training, and it is why the whole thing costs a lookup per token.

new row for t  =  mean over the old rows the pieces of t point at

This page can measure which old rows, and it does not measure the averaging, because the table belongs to the model and there is no such table here. That is an honest gap rather than an omission of something small; whether the average of two rows is a good starting point for a piece is a question about the model, and I have not answered it on this page.

One case has to be handled before the averaging can even be attempted, which is a token with no old rows at all. An average over nothing is not a number, so such a token is reported as having nothing rather than quietly handed an empty list, and whoever builds the table has to find it a row from somewhere else.

Before the eighteen sentences, a corpus small enough to check by hand, which is the one the byte pair page works its merges on. Four words with their counts, low five times, lower twice, newest six times and widest three. Fit it twice, once stopping after three merges and once after ten, and we have two vocabularies of 15 and 22 rows over identical text.

That is the easiest move that exists, since the two agree about the corpus, the alphabet, the marking convention and the order the merges were learned in. Anything that goes wrong here will go wrong in a harder move as well, which is the reason to look at it first.

Take the ten-merge vocabulary as the old one and the three-merge vocabulary as the new one, which is the direction somebody takes who wants a smaller model rather than a better-fitting one. All 15 tokens are answered by the first two rules, 14 of them copied and one of them the label for the unspellable, and not one token has to be read again.

The reason is the ordering. Merges are learned one at a time and each one adds a row, so a fit that stops early holds exactly the first rows of a fit that goes on longer. Shrinking a vocabulary learned from the same corpus is therefore a pure copy, and none of the machinery of Part 4 is reached at all.

Turn the move around, so the three-merge vocabulary is the old one and the ten-merge vocabulary is the new one, and seven tokens are now strings the old vocabulary never held. Fourteen are still copied, and the seven that are not are seeded from 2.86 old rows each on average.

…

Read down the right-hand column of the growing direction and the seven readings are visible one at a time. Six of them are words or word endings and come back as the pieces that spell them. The seventh is the piece made of an e and a w, which comes back carrying an end-of-word marker it never had, and section 12 is about that.

The case the method was designed for is a new token that is a whole common word. The ten-merge vocabulary has a row for low with its marker, and the three-merge vocabulary does not, since it stopped before that merge. Decoded, the new token is the text low, and the old vocabulary spells that text in two pieces it does have.

Here is the finding this page is built around, and it is small enough to state in one sentence. A piece carries, in its marker, the fact that it did or did not end a word, and decoding is precisely the operation that throws that fact away, so a piece that only ever sat in the middle of a word comes back as a whole word and is read as a different thing entirely.

The seventh reading of the last section is the instance. The new vocabulary has a row for the two letters e and w with no marker, which is what the middle of newest looks like. Decoded it is the text ew. Handed that text, the old vocabulary sees a word two letters long, spells it as e followed by w with the marker on it, and hands back those two rows. The marker was never in the original piece and there is no text that could have kept it out.

…

the piece   ew            an e and a w, neither ending a word
its text    "ew"          the marker is gone, and no text can hold it
read again  e, w</w>       the w now says a word ended here

It is worth being clear about why this cannot be repaired inside the third rule. Decoding has to produce text, because text is the only thing the old vocabulary knows how to read, and a marker is not part of the text. One could imagine passing the position along beside the text, and then the old vocabulary would need an entry point that encodes a fragment known to sit inside a word, which is a different operation from encoding a text and not one any of these vocabularies offers.

The second loss is worse and comes from the same step. Take the old vocabulary one merge smaller, so that it stops after two merges and no longer holds the piece made of l and o. Now that piece has to be read again, and its text is a word two letters long ending in o.

No word of those four ends in o. Low ends in w, lower in r, newest and widest in t, so the symbol meaning “o at the end of a word” was never learned and cannot be. The old vocabulary answers with the row for l and then the row that means it could not say, and that second row is not a piece of the word at all.

the piece   lo            an l and an o, in the middle of low
its text    "lo"          a word ending in o
read again  l, [UNK]      and no word of this corpus ends in o

A new row seeded half from the row for l and half from the row that means “something else” is not a bad guess in the way a slightly wrong number is a bad guess. The second row is a summary of everything the old model ever failed to read, so averaging it in mixes the token with a category it has nothing to do with.

Both losses get much larger when the two vocabularies mark opposite ends. A byte pair vocabulary marks the piece that ends a word; a WordPiece vocabulary marks the pieces that continue one, leaving the first piece of a word unmarked. The two describe the same idea and no spelling is shared between them except by accident.

Move a ten-merge byte pair vocabulary into a WordPiece vocabulary of the same 22 rows over the same four words and the copy rule fires four times. Seventeen tokens take the third rule, and five of those come back as nothing but the stand-in, because they are single letters marked as continuations and their text is a one-letter word ending in a letter no word of the corpus ends in.

Two of the seventeen land on a plausible row, and it is worth seeing why. The continuation piece est decodes to the text est, which the byte pair vocabulary reads as the whole word est, and it happens to hold a row for exactly that spelling, so the mapping is right. The word-initial piece low lands on the row for low with its marker in the same way, and that one is right about the letters and wrong about the claim, since a word-initial piece is not a piece that ended a word.

There is one more shape, and it is the case where the third rule produces nothing rather than something wrong. A vocabulary built over byte values gives every byte a printable name, including the bytes that are spaces and tabs and line breaks, so it holds tokens whose text is whitespace.

Decode one of those and the text is a single space. Hand a space to a vocabulary whose first step is to find the words, and it finds no words, so it returns nothing at all. Fit a byte-level vocabulary of 266 rows over the four words and move the byte pair vocabulary into it, and exactly 10 of the 266 come back with nothing, which are the ten byte values that are whitespace.

Those ten are reported as having no rows rather than being handed an empty list, which matters because the model half divides by the number of rows. Ten rows out of 266 have to come from somewhere other than this method, and the method is at least able to say which ten.

Back to the eighteen sentences about reports and the eighteen about pastry. The old vocabulary is 137 rows and the new one is 125, and the move answers every one of the 125. Fifty-seven are copied because the old vocabulary holds their spellings, one is the label for the unspellable, and the remaining 67 are read again, from 2.63 old rows each on average.

The 57 look better than they are. Only 14 of them are longer than a single character, which is to say that most of what survives by spelling is the shared alphabet, the letters and the letters that end words. The 14 that are longer are the ones worth having, and they are the pieces two corpora of ordinary English cannot help sharing, the word the in both its forms, and, in, and a handful of two-letter runs.

The size of the new vocabulary is a free choice, so it is worth seeing what the choice does to the move rather than guessing. Fit the pastry corpus at nine sizes, move the report vocabulary into each, and count the two routes at each size.

…

The copied count climbs from 47 to 57 and then stops, while the count read again goes from 4 to 67. That shape follows from what a merge buys. The first rows a fit learns are its alphabet, which the two corpora share almost entirely; every row after that is a piece the pastry corpus found frequent, and the longer and more particular those pieces get, the less likely the report corpus is to have found the same string frequent.

The pieces from Part 4 are not a property of a toy corpus, and the counts here say how common they are. Of the 67 tokens read again, 35 never ended a word, and 25 of those come back from the old vocabulary with a marker on their last piece, claiming a word ended where the original piece claimed nothing of the kind.

Twelve of the 67 reach the label for the unspellable somewhere in their reading, and two of those reach nothing else. Those two are the letters k and x with the end-of-word marker, which the pastry corpus needs for rack and Mix and the report corpus never needs, since no word of eighteen sentences about analyses ends in either letter. Their new rows would be averages of one row, and that row means “something I could not read”.

The reason for all of this was length, so here is the length. The pastry corpus falls from 428 pieces to 276, and one sentence of it, held out of the corpus, falls from 34 pieces to 19.

…

One thing about those numbers has to be said plainly, because it is easy to read the page as claiming more than it does. The 276 is not something the move achieved. The new vocabulary reads the new corpus in exactly the pieces a vocabulary learned fresh on that corpus would, because it is a vocabulary learned fresh on that corpus. Moving the old rows into it changes nothing whatever about how text is cut.

What the move buys is the second column of section 3, the 32,000 numbers that would otherwise have started from noise. The sequence length is bought by learning a new vocabulary, which anybody can do; the move is about what it costs to actually use one.

The report corpus was the model’s home, and after the move it is not read well at all. Under the old vocabulary the eighteen report sentences cost 263 pieces; under the new one they cost 505, which is worse than the penalty the pastry corpus was paying in the other direction. The running sentence of this section goes from 25 pieces to 37.

It gets worse than length on that sentence. Under the old vocabulary one piece of it comes back as the stand-in, the z of Alvarez, and under the new one five do. The four that were added are the apostrophe, both hyphens, and the x of expect, since the pastry corpus uses none of the first three and uses an x only at the end of Mix. Switch the widget above to the sentence of the old domain to see the two readings side by side.

Counted at the width of 256 chosen in section 2, the new table is 32,000 numbers. Every row of it has at least one old row to average, so the move fills all 32,000, which is the number that would otherwise have been drawn from noise.

512 of those numbers, being the two rows seeded only from the label for the unspellable, are filled with something that carries nothing about the token it was given to. Two rows of 125 is 1.6 per cent of the table, and the method at least names which two rather than leaving them to be found later.

Section 17 measured the two routes across nine sizes, and the same nine fits say what each size costs in length, so the choice can be made with both columns in view rather than one.

Read across the rows and the trade is not the one it looks like. Going from 52 rows to 125 costs 63 more tokens that have to be read again, and buys 196 pieces off the corpus. Whether that is a good exchange depends on how much a rebuilt row costs the model, which is exactly the quantity this page cannot measure, so the honest reading of the table is that the right-hand column is the reason to take a larger vocabulary and the middle columns are the reason to be careful about it.

Everything measured above follows from one property of the third rule, and it is worth stating as a fact about the method rather than as a result. The only channel between the two vocabularies is text. A piece is not text; a piece is text together with a claim about where in a word it sat, and that claim is written in a marker which is not part of any string the piece stands for.

So the move is exactly as faithful as the copy rule can make it and no more. Where a spelling is shared, the marker travels with it and nothing is lost. Where it is not, the marker cannot travel, because the operation that produces the text for the old vocabulary to read is the operation that removes it. On the two eighteen-sentence corpora that cost 25 of the 125 new rows a false claim about position, and there is no choice of decoding that would have avoided it.

Look for the quantity being maximised and there is not one. Every other method in this section has an objective, whether it is the corpus getting shorter or a likelihood going up, and a fit can be compared with another fit by reading that number. A move has no such number. Given the two vocabularies the answer is determined, and there is no dial to turn and nothing to compare.

That absence is not a technicality. What one wants to know is whether the moved model works, and no arrangement of the counts above answers it. Fifty-seven copied of 125 sounds encouraging and 67 read again sounds worrying, and neither number is evidence, because a copied row might be a piece the new corpus uses for something else entirely and a rebuilt row might be a perfectly good average.

The only way to know is to run the model on the new corpus, and that is a measurement about a model rather than about a vocabulary.

The last point is the one the copy rule cannot see, and it is the reason the previous section is not merely cautious. A spelling shared by two vocabularies is copied exactly and its row arrives intact, carrying everything the old model learned about that piece. What the spelling does not carry is which words the piece turns up in, and on two corpora about different subjects that can be entirely different.

The second row is the sharpest of them. In the report corpus the piece is the tail of reran and than, and in the pastry corpus it is the whole word an, so a row that meant “the end of a longer word” is now being asked to mean an article. The row is copied exactly, nothing was lost in the move, and it is still describing something the new corpus does not do.

Gathered in one place, these are the inputs on which the method stops being defined rather than becoming approximate, together with what has to be decided in each case and what turns on the decision. Several of them are genuinely open, with defensible answers on either side, and those are the ones worth a reader’s attention, since they are the decisions anybody moving a vocabulary has to make and nothing in the method makes for them.