SentencePiece

We carry the same sentence as the other pages in this section, and beside it the same corpus of eighteen short sentences of ordinary English about reports and costs and analyses. The corpus holds 72 distinct words across 133 occurrences and does not contain the sentence, which is the arrangement we want, since a vocabulary is only interesting on text it was not built from.

Dr. Alvarez didn't expect the low-cost re-analysis.

The earlier examples first split on whitespace, then learned pieces inside each resulting chunk. A whitespace split leaves Dr., didn’t and low-cost intact. It does not decide whether a full stop ends a sentence or belongs to an abbreviation.

A richer pre-tokenizer may split punctuation, contractions or hyphenated expressions as well. Those are additional decisions. SentencePiece lets us train from the text stream without first supplying a linguistic word segmentation.

A whitespace split treats each uninterrupted run as one chunk. This is often a convenient starting point for English, although a chunk need not be a linguistic word. A Chinese sentence with no spaces becomes one long chunk. Calling that chunk a word would confuse a formatting boundary with a language boundary.

To make that concrete, this page carries five short sentences of Chinese beside the English corpus, written the way Chinese is written, with nothing between the words. Between them they use seven words, the words for research, life, origin, student, very, many and good, and several of those words turn up in more than one sentence. A whitespace split finds five chunks in them, one per line, and reports nothing unusual. The repetition a learner needs is there, and it is inside the chunks, where a rule about spaces cannot see it. Part 5 fits both schemes on these five lines and measures what each finds.

Character-based subword learning can still find repeated pieces inside those long chunks. It does not inherently require an annotated Chinese segmenter. The distinction is whether a separate word segmentation is required by the training pipeline. SentencePiece provides a framework that can learn directly from unsegmented text.

There is a second cost, and it is the one that shows up on our own English sentence. A scheme that cuts on spaces has to record somewhere that a word ended, or gluing the pieces back gives one long run of letters, and the usual place to record it is on the last character of each word. That makes the w that ends low a different symbol from the w inside lower, which is a real distinction and is bought at a real price.

Counted on our eighteen sentences the corpus uses 36 distinct characters, the space among them. Marking the end of each word turns the 35 that are not the space into 51 symbols, because sixteen letters appear both inside a word and at the end of one and are held apart, and the full stop only ever ends one. Sixteen rows are spent before a single merge is learned, and the smallest vocabulary that can exist at all is 52.

The holes are worse than the count. A letter that the corpus uses only inside words has no end-of-word symbol, so a word ending in it cannot be spelled. These eighteen sentences contain the letter z, in the word size, and no word in them ends in z. Alvarez does.

SentencePiece removes the first question by refusing to answer it. Every run of whitespace in the text is replaced by an ordinary character, one is put at the front, and from then on the text is a single stream of symbols with nothing special about any of them. The learner counts pairs over that stream exactly as before. Decoding joins the pieces and turns every such character back into a space.

Two things follow immediately and both are worth stating before any detail. Nothing in the procedure refers to a language, so the same code runs on English and on Chinese and gets a defensible answer for both. And nothing was thrown away on the way in, so the pieces carry the spaces along with the letters and putting them back is concatenation rather than guesswork.

It is worth saying plainly what SentencePiece is not. It is not a third way of choosing pieces alongside merging and shrinking. It is a treatment of the text that either of those can be run on, and section 13 shows both running on the same four words. The contribution is the step before the learner, which is why this page spends its first half there.

The character chosen is ▁, which Unicode calls lower one eighth block and numbers 9601. It was picked for two properties. It looks like a space, so a marked text is still readable, and it does not turn up in ordinary writing, so nothing is being overloaded. Any character meeting the second of those would serve, and section 21 measures what changes when a different one is used.

The mark goes in front of a word rather than behind it, which is the mirror image of the boundary marker in section 3 and is not merely a matter of taste. A mark in front says a space preceded this, and a space preceded the first word too, so one is prepended to the whole text. Marking the front means no letter is ever doubled, since a letter is a letter wherever it sits and the mark is a symbol of its own.

hello  world   →   ▁hello▁world

Compare the two alphabets on the same eighteen sentences. Marking the front leaves the 35 characters that are not the space untouched and adds one symbol, for 36 in all. Marking the end leaves 34 characters that only ever appear inside a word and adds 17 more that are a character with a boundary attached, for 51.

Fifteen rows is a small saving on a table of 137 and a large one on a table of 40, and the second is the case that matters. Below 52 rows the marked-at-the-end scheme cannot be fitted at all, because dropping a symbol would leave part of its own training corpus unspellable. The front-marked scheme runs from 37 upward.

Here is the whole transformation on our sentence, with nothing learned and nothing fitted. The sentence is 51 characters of which six are spaces, and it becomes a stream of 52 symbols, the six spaces replaced by six marks and one more put at the front. Cut at the marks it is seven units, one per word, each carrying its own mark.

…

Notice what did not have to be decided. The full stop after Dr. stayed attached to Dr. and the full stop after analysis stayed attached to analysis, and no rule looked at either of them, so the question of whether an abbreviation keeps its stop never came up. The apostrophe in didn’t is an ordinary character in an ordinary unit, and so is the hyphen in low-cost. Whether the pieces should divide at any of those places is left entirely to the counting that comes next.

Once the text is one stream, a merge could in principle join any adjacent pair, including a pair with a mark between the two halves. The framework does not allow that. Having marked the text it cuts it at every mark, so each unit is one mark followed by one word, and a piece is never allowed to reach out of its unit.

That second step is a genuine decision rather than a consequence of the first, and it is worth flagging now because the closing part measures what happens without it. Taken together the two steps amount to splitting on whitespace and putting a mark at the front of each piece, which sounds like it has quietly restored the thing the method set out to remove.

It has not, and the difference is precise. The mark is inside the unit, so it is a symbol the learner counts, merges and stores, and it comes back out at the other end. A separator is none of those things.

We need something small enough to check by hand, so for the next three sections we drop to four words with their counts, the example the original subword paper worked in its own text. The word low occurs five times, lower twice, newest six times and widest three. Marked, each is its characters with one mark in front, so low is four symbols rather than three and the sixteen word occurrences come to 95 pieces in all.

low  =  ▁  l  o  w

Now run the loop. Count every adjacent pair, weighted by how often the word occurs, take the largest count, join it everywhere, and repeat. Fifteen merges later every word is a single piece and the corpus is 16 pieces, one per occurrence. Drag the control and watch the mark wait.

…

The order the merges arrive in has a shape worth stopping on. The first two build the ending est out of e, s and t at a count of 9. The third and fourth build the word low out of l, o and w at a count of 7. Only then, at the fifth merge and still at a count of 7, does the mark join anything, and what it joins is the finished word.

That is not the counting deciding. At merge 2 the pairs l followed by o, o followed by w, and the mark followed by l all occur exactly 7 times, so the objective is indifferent between them and something outside the objective has to choose. The rule in force is that the earlier-sorting pair wins, and the mark is codepoint 9601 where l is 108, so any pair beginning with the mark sorts after every pair beginning with a letter and loses every tie it enters.

Across the whole run of fifteen merges a pair beginning with the mark was among the tied candidates eleven times, and it was taken three times, each of those being a step where it was the only candidate left at that count. So the mark attaches to a word only once nothing else can be built, which is why the vocabulary ends up holding ▁low and ▁newest rather than ▁l and ▁n. A mark that sorted before the letters would produce the opposite, and section 21 runs that fit.

Look at the finished table on the four words and something odd is there. It holds low, and it also holds ▁low, and those are two different rows with two different numbers. The same for newest and ▁newest, widest and ▁widest. A reader who opens such a vocabulary expecting a list of words finds it half full of things that are a word with a space stuck to the front, which is not a word in any ordinary sense.

It is the right answer, though. The two really are different things and the model reading them should know which it has. The low in low-cost has no space before it and the low in the low readings does, and a scheme that gave them one row could not put the space back afterwards. What looks like duplication is the boundary information, stored in the only place it can be stored once the space is a character.

On the larger corpus the mark stops waiting. The first merge joins h to e at a count of 32, and the second joins the mark to t at 25, so by the third merge the word the with a space in front of it is a single row. That happens because the is so common that the pair of a space and a t outnumbers almost everything else, which is exactly the behaviour the four words were too small to show.

…

Two numbers there are worth carrying away. Of the 99 merges the corpus supports, 42 extend a piece that already begins with the mark, so nearly half the learned table is about where words start. And 17 of the finished rows are a row the table already holds with a space in front of it,▁cost beside cost and ▁re beside re among them, so 17 rows out of 136 go on keeping the distinction section 11 described.

Everything so far has used the merging learner, which starts from single symbols and joins. The framework will run the other one just as happily, which starts from a large pool of candidate pieces and drops whichever is least missed until the table is the size asked for. Neither of them knows what a mark is; both take units already spelled in symbols with counts.

Asked for 30 rows on the four words, the merging learner stops at 27 when no pair occurs twice any more, and the shrinking one lands on 30 exactly. They share 23 rows. The seven the shrinking one holds alone are pieces of widest and newest such as id, ide and ewe, and the four the merging one holds alone are lo, low, newest and widest. And yet both spell the unseen word lowest identically, as ▁low followed by est.

Gluing the pieces back is three operations with nothing to decide. Concatenate the pieces, then replace every mark with a space, then drop the one at the very front that was put there when the text was marked. There is no rule about abbreviations, no rule about quotation marks, and nothing that has to remember where the spaces were, because the spaces are in the pieces.

▁the  ▁low  ▁cost   →   ▁the▁low▁cost   →   the low cost

Compare that with what a scheme built on a rule-based front end has to do. Its tokenizer deleted the whitespace and produced words, so its detokenizer has to decide where a space goes back in, and the decision is a language-specific rule again. A full stop takes no space before it in English, a French quotation mark does, and neither of those facts is in the pieces.

Now the claim the page turns on, checked rather than asserted. Both schemes were fitted on the same eighteen sentences until each ran out of pairs occurring twice, giving 136 rows one way and 137 the other. Both were handed the sentence they had never seen, and both had their pieces glued back together.

…

The front-marked fit returns the sentence exactly, character for character, spaces included. The other returns something else, and two things went wrong at once in it.

Dr. Alvare[UNK]didn't expect the low-cost re-analysis.

The letter z at the end of Alvarez is a symbol it never learned, for the reason section 3 gave, so a stand-in row comes back in its place. And the stand-in carries no boundary, so when the pieces are joined the space after Alvarez is gone and the next word runs into it. One missing symbol cost a letter and a word boundary.

The five words underneath the readings are where the two fits part company in detail. The word analysis is one piece under both, since the corpus contains it. Inside re-analysis. it is three pieces under the front-marked fit and part of two under the other, and the reason is section 11’s point read backwards. The row for analysis here is ▁analysis, and there is no space before analysis inside re-analysis, so that row cannot apply and the fit falls back to an, aly and sis.

It rests on one thing, which is that the transformation into marks is reversible. The moment anything is normalised on the way in, the round trip recovers the normalised text and not the text. The framework’s own defaults already normalise three things, and rather than call the round trip exact and leave those unsaid, here they are.

…

There are three. A run of whitespace of any length and of any kind becomes one mark, so two spaces come back as one and so does a tab, which is the first row above and the third. Whitespace at the two ends of the text is dropped entirely. And a text containing the mark character itself is treated as though it held a space there, which is why the last row went in with one space, one mark and one space and came back with three spaces.

The exactness and the language-independence are worth something; the question is what they are worth in the currency everything downstream actually pays in, which is the number of pieces a text becomes. So both schemes were fitted on the same corpus at fourteen sizes and asked for two numbers each, what the whole corpus costs and what the held-out sentence costs.

…

On the corpus the front-marked fit is shorter at every size from 52 rows up, 553 pieces against 648 at the bottom and 261 against 263 at the top. That is the fifteen alphabet rows coming back as fifteen extra merges at every size where neither fit has run out of pairs. On the held-out sentence it is shorter at 52 rows and 60, and then the curves cross and it is longer at every size above that, 34 pieces against 28 at 80 rows and 28 against 25 at the largest fit.

This is the place on the page where the method is beaten by the thing it replaces, so it is worth understanding rather than noting. The front-marked fit spends 42 of its 99 merges on pieces that begin with a space. Those rows are useful only at the start of a word. A word the corpus never contained is usually met somewhere other than at its start, and there they are no help at all.

The stem expect is the cleanest case. The corpus holds expected and not expect, so neither fit has a row for the shorter word. The marked-behind fit reads it as exp, ec and a marked t, three pieces built from interior merges that apply anywhere. The front-marked fit reads it as ▁e, x, p, e and ct, five pieces, because its early merges all went into building▁expected and ▁e and there is nothing in the table for the middle of a word beginning with ex.

So the trade is between where the rows are spent. Marking the end lets a piece be reused anywhere inside a word and pays for it with a doubled alphabet and unspellable endings. Marking the front keeps the alphabet small and spends the merges on word beginnings, which is excellent on text like the corpus and poorer on text that is not.

Now the case the whole method exists for. Take the five short Chinese sentences from section 2 and fit both schemes on them, each asked for 40 rows. Then read back a sentence those five never contain, made of words they do, which is student, research, life and origin run together.

…

The marked-behind scheme finds one word per line, so it has five very long words to merge inside and almost no repetition across them. It learns three merges, stops at 17 rows however many it was asked for, reads the five lines in 18 pieces and the new sentence in five. The front-marked scheme learns six merges, reaches 18 rows, reads the lines in 15 pieces and the sentence in three, and those three are student, then research and life together, then origin.

The clearest sign of the mismatch is where the first scheme’s rows went. Three of its 17 are a character it already holds, kept apart because that character once fell at the end of a line. There are no word boundaries in this writing for a boundary marker to record, so those three rows record the ends of lines, and a piece carrying one puts a space into the decoded text at a place no space belongs.

Section 8 flagged this and here it is. Once the space is an ordinary character, nothing in the method forbids a merge from joining a pair with a space between its halves. The counting rule says take the commonest adjacent pair and a pair straddling a space is adjacent like any other. So the question of whether a piece may cross a word boundary has to be answered from outside, and the framework answers it by cutting at every mark, which is a choice rather than a consequence.

What the other answer produces can be measured rather than guessed. Run the same counting on the eighteen sentences with nothing cut and the very first merge of all joins the letter e to the space that follows it, at a count of 39.

…

Thirteen of the first forty merges produce a piece holding a space somewhere other than at its front. By rank 3 the piece is the whole of the word the with a space behind it, by rank 17 it is a space, the, and another space, and by rank 20 it is the ending of one word joined to the whole of the next.

The loop is answering exactly the question it was asked, which is which pair is commonest, and early on the commonest pairs are letters sitting either side of a gap. Forty merges in, thirteen of them have gone on pieces that straddle a boundary.

Section 10 showed a tie decided by the mark’s codepoint, and a codepoint is an accident of which character was picked. So the mark is not neutral, and the way to find out how much it matters is to change it. Here are the same four words fitted twice, once under the usual mark at 9601, which sorts after every letter, and once under an underscore at 95, which sorts before the lowercase letters.

…

Both tables hold 27 rows and 18 of them agree once the mark is written the same way, so nine rows on each side exist under one mark and not the other. The merges tell the story. Under the late-sorting mark the third, fourth and fifth merges are l with o, lo with w, and then the mark with the finished low.

Under the early-sorting one they are the mark with l, that with o, and that with w, so the word is built outward from the space one letter at a time and the table fills with the mark followed by l, then by lo, then by low, rather than with lo and low.

What did not change is the unseen word, which both fits cut as the marked low followed by est. So the effect is real and it is not catastrophic, and the honest summary is that nine rows of the 27 depend on which character was picked to stand for a space, for a reason that is arithmetic about codepoints and has nothing to do with language.

The strongest thing the method can say is that no step in it refers to a language. That is true and it is checkable, and it is much weaker than saying the result is language-independent. A vocabulary is learned from a corpus and carries that corpus’s habits permanently, and the corpus is written in some language.

Handed the Chinese sentence of section 19, the fit learned from our eighteen English sentences answers a mark followed by eight stand-in rows, and glued back it is eight stand-ins and nothing else. Every one of the eight characters was unspellable, because the alphabet is whatever the corpus happened to use, and the procedure that produced that vocabulary consulted nothing about English at any point.

The fix, where one is wanted, is not in this method. Starting from the 256 byte values rather than from the characters a corpus used closes the hole, at the cost of longer sequences, and it composes with everything on this page rather than replacing any of it. The marking removed the rule about where English words end; it left the corpus in place, and the corpus is where the alphabet of 36 symbols came from.

Gathered in one place, these are the inputs and questions on which the method stops being defined rather than becoming approximate, with what has to be decided in each case and what turns on the decision. Several are genuinely open with defensible answers on either side, and those are the ones worth a reader’s attention, since they are the decisions anybody setting this up has to make and nothing in the mathematics makes them.