Splitting on Spaces

We carry one sentence through this page, the same one the rest of this section carries, because it is short enough to check by hand and awkward in four separate places.

Dr. Alvarez didn't expect the low-cost re-analysis.

To a machine that is 51 characters in a row, and nothing in those 51 characters is marked as a beginning or an end. Anything that wants to count words, look words up, or learn something about a word has to be told where one word stops and the next starts, and there is no way to read that off the text without a rule that somebody wrote. The question comes before every other question about text, and it is asked once, before any model exists.

It is worth separating that question from the one it is usually confused with. Deciding where the words are is a claim about the writing system, since English puts gaps between its words and Chinese does not. Deciding what a model’s units are is a claim about a corpus, since a vocabulary of forty thousand entries has to spell every text it will ever meet out of pieces it has already seen.

This page is entirely about the first, and the rule it describes is the shortest of the answers these pages cover, which is why it comes before the rest of them.

The rule is one sentence long. Take every maximal run of characters that are not white space, and each one is a word. Nothing else is looked at, there are no exceptions, and there is nothing to configure.

a word  =  a maximal run of characters, none of which is white space
the words  =  every such run, in the order they appear

The word maximal is carrying the weight. A rule that cut at each space and kept whatever fell between two cuts would produce an empty word wherever two spaces met, and an empty word is not a word of anything. Taking the runs instead means the spacing is skipped over rather than divided at, so a single space, a double space and a line break all do the same thing to the answer.

Put the sentence to the rule and seven pieces come out. The playground above is doing exactly this, and it is worth reading the seven before going any further, because every difficulty on the rest of the page is already visible in them.

Dr.            [0, 3)
Alvarez        [4, 11)
didn't         [12, 18)
expect         [19, 25)
the            [26, 29)
low-cost       [30, 38)
re-analysis.   [39, 51)

The seven pieces hold 45 of the sentence’s 51 characters between them, which is 88.2 per cent of it. The six characters that are in no piece at all are the six spaces, and that is true of every text rather than of this one, since the gaps between the runs are by construction made of exactly the characters the runs exclude.

The rule says white space rather than space, and the difference is not decoration. A great many characters are white space, and a rule that recognised only the one on the space bar would join two words across a no-break space, which no reader would, and would run a whole line of Japanese together at an ideographic space. I put ten characters between the letters a and b one at a time and counted the pieces that came back.

…

Seven of the ten end a word and three do not, and the three are the interesting ones. A hyphen is expected there, since a hyphen is visibly a mark rather than a gap. The other two are not. The character called the zero width space, which exists to offer a line-breaking opportunity inside a long word, is not white space at all by the test the rule applies, so the letters either side of it stay in one piece; the soft hyphen behaves the same way. The rule asks whether a character is white space and never what it is called, and for those two the name and the answer point in different directions.

A rule of this kind could report a list of strings and stop there, and a great many implementations do. That throws something away that the rule knew a moment earlier, which is where in the text each piece was standing. A piece is therefore the characters together with the half-open span they occupied, the offset it starts at and the offset one past where it stops.

a piece  =  ( the characters ,  [ start ,  end ) )
length   =  end − start

Half-open because it makes the arithmetic come out. The length of a piece is the difference of its two offsets with no correction, an empty stretch is a span whose two offsets agree, and one piece ending where the next begins is the same number written once rather than two numbers that must be kept a step apart. It is the convention every slice of a sequence uses, for the same reasons.

Three things downstream need the position and cannot recover it. Anything that draws the reader’s text with a piece highlighted has to know which characters to highlight. Anything comparing two rules on one sentence has to line their answers up, and only the offsets do that, since the strings themselves differ. And a rule for a script that writes no spaces has nothing to report except the boundaries it chose, so for those rules the span is the whole of what they have to say.

Adding the spans up says how much of a text survived into the pieces. On the running sentence the answer is 45 characters of 51, and the six that are missing are the six spaces. On writing that contains no spaces the answer is everything, and on writing that is nothing but spaces the answer is nothing.

The share kept has to be read carefully, since it says how much of the writing was spacing and says nothing about whether the answer was any good. It is 88.2 per cent on this sentence because English spends roughly a seventh of its characters on spaces, and it is exactly one on the Chinese line because that script spends none, which is section 11.

The fourth row is the reverse case, a text where the rule keeps none of the writing at all and is entirely correct to, since three spaces hold no run of anything else.

The sharpest test of any cutting rule is to cut a text and put it back together immediately, with nothing in between, and compare what comes out against what went in character for character. Anything the rule discarded shows up as a difference and nothing else does. Something has to do the gluing, and anything holding the pieces alone has only one option, which is a single space between each pair.

…

On the running sentence a single space is right in all six places, so the sentence comes back exactly and the rule looks lossless. On the same sentence with one doubled space and one line break it is right in four places out of six, and the text that comes back is the first one, which is not the text that went in.

Both texts give the identical seven pieces, so nothing a model reads could tell them apart, and no amount of training recovers the difference, since it was thrown away before the model was reached.

The spans are where it can be got back. Putting each piece back at the offsets it came from, and taking the source between one piece and the next, reproduces both texts exactly, because the offsets never stopped describing the original. That works only while the original text is still in hand, which is a real limitation, though it is the difference between a loss that can be repaired at the boundary and one that cannot be repaired at all.

A full stop, a comma, a quotation mark and a bracket are not white space, so a run that contains one carries it along. The first three rows below are that, and they are the failure the next four pages of this section exist to repair.

…

Read the first row and the piece is away., which is not a word anybody would write in a dictionary. Read the second and the piece is small,. The third is worse, since a quotation mark on each side gives "low-cost" as one piece, and a bracket on each side gives (twice), so a word wrapped in marks is a piece that shares no characters at either end with the bare word it contains.

The fourth row above is the same behaviour and a much harder problem. Dr. ends in a full stop that belongs to the word, so keeping it is correct, and re-analysis. ends in a full stop that belongs to the sentence, so keeping it is wrong. The rule does the same thing to both, and it does the same thing for the same reason.

Over the six sentences this page measures, seven of the 47 entries end in a full stop, and one of the seven is an abbreviation while six are the ends of sentences. Nothing in the seven runs of characters distinguishes them. The information that would is a fact about English usage rather than about the characters, which is why the rules that get this right carry a list of abbreviations and why every such list is a list for one language.

The fifth and sixth rows are different in kind, and calling them failures takes an argument rather than an example. didn’t comes out whole, and low-cost comes out whole, and for a good many purposes those are the answers wanted.

They are failures against a particular convention. The annotation rules a generation of English research was built on cut a contraction into the verb and the negation, so that the negation is one unit wherever it appears, and the Unicode word boundary rules break a hyphenated compound into its halves. Put the running sentence to that second rule and it comes back as nine pieces rather than seven, with low and cost separate and re and analysis separate. Whether seven or nine is right is a question about what a word is, and section 15 is where that question is faced rather than answered.

The last row is the one that is not English, and it is the worst case the rule has. Nine characters of Chinese contain no white space anywhere, so there is exactly one maximal run and the answer is one piece holding the whole line. A reader of Chinese sees several words there. The rule sees one.

What makes this the worst case rather than merely a bad one is that every reading a caller could take is reassuring. The pieces cover 9 characters of 9, so the share of the writing kept is exactly one, which is higher than the 88.2 per cent the English sentence scored. Gluing the single piece back gives the line character for character, so the round trip is exact. On both of the measures Part 2 built, the answer here is better than the answer on the sentence the rule was designed for.

The costs so far have been shown one piece at a time, which says nothing about how often they arise. The corpus here is six sentences from the same notebook the rest of this section shares, 375 characters in all, and I read all six under this rule and under the Unicode word boundary rules, which are the first repair the next page describes.

…

Nine of the 47 entries carry a mark of punctuation at one end or the other. Taking those marks off would leave 46 distinct entries rather than 47, so exactly one entry is a word the list already holds wearing a full stop, which is re-analysis. against re-analysis. That is a small number on six sentences and it is small for a reason worth knowing. A duplicate needs the same word to appear both at the end of a clause and inside one, and six sentences give most words only one chance.

The most surprising thing in that table is that both rules arrive at 47 entries. It is a coincidence of this corpus rather than a result, and the lists underneath show why. Only 35 of the entries are held by both, so the two rules disagree about 12 of them in each direction, and they disagree in opposite ways. This rule spends its twelve on words wearing punctuation and on compounds kept whole; the other spends its twelve on the bare forms of those same words and on the halves of the compounds.

Five sections of failures make it easy to forget that this rule is used deliberately, underneath most of the methods that criticise it, including the subword learners at the end of this section. It is worth saying what it has that they do not, since those properties are why it is still where a pipeline starts.

It reads no corpus, so there is no fitting step and no data to gather, and the answer it gives today is the answer it gave last year. It has no settings, so two people applying it to one sentence get the same seven pieces and cannot have configured it differently. It carries no list of exceptions, so it cannot be out of date, and it makes no claim about any particular language, so it cannot be wrong about a language it has never seen in the way that a list of English abbreviations is wrong about German.

And it never rewrites a character, so every piece is exactly the slice of the source its span names, and the writing can be recovered from the pieces and their offsets.

The comparison in section 12 also runs the other way, and it is more interesting in that direction, because the repair is usually described as strictly better. On the same six sentences the boundary rules give 69 pieces where this one gives 63, so a model reading that corpus would read about a tenth more positions, and the cost of reading text grows at least in step with the number of positions.

The sharper difference is what happens to the characters. On the running sentence the boundary rules keep 41 of the 51 characters, against 45 here. The ten they leave behind are the six spaces and, unlike this rule, both hyphens and both full stops, since a mark that is not part of a word is not returned as one.

Gluing their nine pieces back with single spaces gives a line with the abbreviation’s stop gone, the compound split in two and the sentence’s own full stop missing, so the loss is not confined to the spacing the way it is here.

It is tempting to read Part 3 as a list of mistakes, and it is not quite that. A mistake needs a correct answer to be measured against, and the question of where the words are does not have one, because a word is a unit somebody defined rather than a thing in the text.

Take low-cost. A dictionary lists it as one headword. A search index that must match a query for cost wants two. A typesetter breaking a line wants two with a joint between them. All three are right about their own purpose, and there is no fourth authority they could appeal to, since the writing itself records only that somebody typed a hyphen. The measurement in section 12 says the same thing arithmetically. Two careful rules put 35 entries in the same place and 12 in different places, and nothing decides the 12 except which convention was adopted first.

The honest description of this rule, then, is that it takes one convention, that a word is whatever a writer separated with a gap, and applies it to everything without exception. That convention is defensible and it has one property none of the others has, since a reader can check the rule’s answer against the page in front of them by looking at where the gaps are, without knowing the language or consulting a list.

The rule makes one assumption and never states it, which is that the writing in front of it separates its words with gaps. That assumption is true of English and of most European writing, and it is false of Chinese, Japanese, Thai, Khmer, Lao and Burmese, which put nothing between one word and the next. It was also false of the Greek and Latin the alphabet arrived in, as the opening of this page describes, so it is a fact about a habit that spread rather than about writing.

What follows is a statement about the method rather than about any particular text. Where the assumption holds the rule’s answer is determined by the writing, and where it fails the rule’s answer is determined by nothing at all, since a text with no white space in it has exactly one maximal run however many words a reader finds there. The question the rule is asked and the question it answers come apart completely, and the answer it gives, one piece per unspaced stretch, is arithmetically correct and useless.

There is no repair available inside the rule. Finding the words in a script that writes none requires either a list of the words of that language or a model trained on text somebody has already cut up, and both of those are information from outside the text. That is what the next section of this site is, and the four methods in it differ mainly in where that outside information comes from.

Put those together and one property covers all of them. This rule is total. Every text has an answer, the answer is always a list of runs, and there is no input on which the rule can decline to say anything, so nothing that goes wrong announces itself. The table gathers the cases where the answer is something other than what a caller wanted, with what the rule actually determines in each.