Unicode Word Boundaries

We carry one sentence through this page, the same one the rest of this section carries, because it is short enough to check by hand and awkward in four separate places.

Dr. Alvarez didn't expect the low-cost re-analysis.

Cutting that at every run of white space gives seven pieces, and they are reasonable pieces. The trouble is not the pieces. It is that the rule which produced them has an assumption inside it that it never states, which is that whoever wrote the text put a gap between one word and the next. That is a habit of a writing system rather than a property of language, and it is false of a great deal of the world’s writing.

Here is the same rule on eleven characters of Japanese, which hold what a reader of Japanese would call several words. There is no white space anywhere in them, so there is exactly one maximal run of non-space characters and the answer is one piece holding the whole sentence. The rule reports no difficulty, because from inside the rule there is none. It was asked where the gaps are and it correctly answered that there are none.

A committee cannot answer “what is a word” for every language, and the Unicode word boundary rules do not try. They answer a narrower question that turns out to be answerable. Between any two neighbouring characters of a text there is a position, and the rules say, for that position, whether a word is allowed to end there. Nothing more.

The definition therefore has two halves and no third. The first half is a table that puts every character in the world into one of a small number of classes, so that a rule never has to talk about a character and can talk about a kind of character. The second half is a numbered list of rules, each of which looks at the classes on either side of one position and says break or do not break.

The rules are tried in their published order, the first that applies decides, and the last one applies to everything.

a text of n characters  has  n − 1  positions between characters
each position  is  the pair of classes on either side of it
the answer  =  the first numbered rule that matches that pair

Two things follow immediately and are worth having before any detail. The rules never disagree with each other, because the order settles every conflict by construction, so there is no tie to break and no scoring. And the whole calculation is one pass along the text asking a fixed question at each position, so it costs what reading the text costs, which is what a rule that has to run while somebody is typing needs.

Putting the sentence to the boundary rules gives nine words where the space rule gave seven, and the playground above is doing exactly that. It is worth reading the nine against the seven before going further, because every difficulty on the rest of the page is already visible in the difference between them.

Dr             [0, 2)
Alvarez        [4, 11)
didn't         [12, 18)
expect         [19, 25)
the            [26, 29)
low            [30, 33)
cost           [34, 38)
re             [39, 41)
analysis       [42, 50)

The nine words hold 41 of the sentence’s 51 characters between them, where the seven pieces of the space rule held 45. The ten characters in no word at all are the six spaces, both hyphens and both full stops. Section 14 measures the same difference over all six sentences.

The whole algorithm is small enough to run by hand, and running it once is worth more than reading about it. Take can’t stop, ten characters, so nine positions to ask about.

c  a  n  '  t     s  t  o  p
0  1  2  3  4  5  6  7  8  9

Three segments come out of that, can’t across positions 0 to 5, the space on its own, and stop to the end. Notice that the space is a segment. The rules cut the text into stretches and every character of the text lands in exactly one of them, which is why the segments always reproduce the text when joined and why deciding which of them are words is a separate step. Section 12 is about that step.

Naming the classes is what turns the rules from a table of characters into something a person can read and argue with. There are eighteen of them. Most describe what a character is, so there is one for letters, one for digits, one for the spaces that separate words, one for the ends of lines.

The interesting ones describe what a character is allowed to do, so there is a class for marks that may stand between two letters, another for marks that may stand between two digits, and another for marks that may do either.

Eighteen sounds like a lot for something this small, and on ordinary English it is. I counted the class of every character in the six sentences this section shares, 375 characters in all, and seven classes account for the lot.

The eleven that never turn up are the point of the exercise rather than an inefficiency. They are there for Hebrew and katakana, for the marks that combine with the letter before them and the invisible formatting characters, for the halves of a flag and the ends of lines, and for three joining marks these six sentences happen never to use, a quotation mark, a colon and a low line.

A rule written for the seven classes that do turn up would look complete on this corpus and would be wrong on the first document that was not in English.

The first eighteen characters of the running sentence, each with the class it was given and a mark wherever a word ended. This is the whole machinery, and there is nothing behind it.

…

Read the run of letters from position 4 to position 10 and the reason the rules never look at those characters individually is visible. Seven characters, one class, and the same rule applied six times. Every one of the five boundaries the rules found sits where the class changes, and the apostrophe at position 16 is a change of class where no word ended, which is what the next section is about.

One honest note about how these classes were arrived at. The standard publishes them as a property in its own data files, and what is shown here is worked out instead from each character’s general category together with the character lists the standard prints in the annex. That is an approximation. It is why a character from a script none of the examples on this page reaches could in principle be placed in a different class than a complete implementation would place it, and it is a limitation of what you are looking at rather than of the method.

This is the experiment that makes the classes legible. I took twelve marks, put each between the letters a and b, then put the same mark between the digits 1 and 2, and read both by the rules.

…

The three rows where the two columns differ are the argument for having classes at all. A colon holds two letters together, so a:b is one word, and it separates two digits, so a time written as 12:30 comes back as two. The comma does exactly the reverse, separating a,b and holding 1,000 together, and the third row that differs is a middle dot, which behaves exactly as the colon does because it was put in the same class. Neither of those answers is a rule about the colon or about the comma. Each is one general rule about a class, applied to a character somebody put in that class.

The rows that look alike and are not are worth as much. A hyphen and a low line are both a mark between two letters, and the hyphen is in none of the joining classes so it separates, while the low line is in the class that joins whatever it finds on both sides, so snake_case_name is one word. A narrow no-break space is in that same joining class, so a thousands group written with it stays whole, which is a character that every rule reading white space calls a separator and this table calls glue.

One class is applied before any of the others and changes how they are read. Combining marks, the characters that attach to the letter in front of them, together with the invisible formatting characters, are hidden from every later rule, so a rule that asks about the character on the left gets the last character that was not one of these.

The reason is that one word can be written two ways. The word résumé can be six characters, each accented letter a character of its own, or eight, each accent a separate combining mark following its letter. Both are the same word and a reader cannot tell them apart on the page. Put both to the rules and both come back as one word, and they do so because the two marks in the second spelling were made invisible before the rule for two letters was asked anything.

There is a case where the hiding is too eager, and it is worth reporting rather than smoothing over. The character that joins two emoji into one picture is an invisible formatting character by category, and the standard gives it a class of its own precisely so that a joined sequence stays together. Treated as ordinary invisible formatting, as it is here, it is absorbed into the emoji before it and the sequence breaks after it, so two joined emoji come back as two segments rather than one.

That is a real difference from a complete implementation and it is on the classes rather than on the rules.

The failure the previous page measures is that a mark of punctuation is not white space, so a run of non-space characters carries it along, and a word that reaches the end of a clause becomes a different entry from the same word inside one. These rules put every mark in a class, and a mark in none of the joining classes is separated from the letters beside it, so the failure does not arise.

…

Read the two lists underneath the table. Every entry on the left is an entry on the right with something taken off it, away against away., 2019 against 2019,, low and cost against low-cost. The two rules agree about 35 of the 47 entries and disagree about 12 in each direction, and the 12 are the whole of what these rules are for.

Both rules come to 47 entries over these six sentences, and that equality is worth naming so it is not misread as a result. Separating the marks shrinks a table only where the same word turns up both inside a clause and at the end of one, so that its two spellings can collapse into a single entry, and on this corpus that happens exactly once. It is visible in the list on the right, which holds both re-analysis and re-analysis., and no other pair there differs only by a mark. Six sentences give most words one chance to appear at all, so there is almost nothing available to collapse.

This is the case the rules were written for, and it is also where they are most often described inaccurately. The claim is not that the rules segment Chinese. The annex says the opposite in as many words. What the rules do is give an answer that is honest about the writing rather than one that looks like a success.

…

Read the Japanese sentence first, because it is the informative one. Eleven characters, mixing three scripts, and the rules give seven words where the space rule gives one. The five katakana characters at the front are held together into one word, correctly, because katakana has a class of its own and a rule that joins it to itself.

Everything after them is one word per character, because the standard deliberately leaves the kanji and the hiragana out of the letter class. Some of that answer is right and some of it is a placeholder, and the rules make no distinction between the two.

Chinese and Thai are the placeholder alone. Nine Chinese characters become nine words where a reader sees four, and seven Thai characters become seven. Korean is the control and shows this is a decision rather than an inability, since Korean is written with spaces and the annex keeps its syllables in the letter class, so the rules find the same two words there that the space rule finds.

Two of the rows in that panel are not about scripts at all, and they are the sharpest demonstration that the rules are reading something other than the gaps. A French thousands group is written with a narrow no-break space, which is white space by every ordinary test, so the space rule cuts 1 000 sensors into three pieces and reports one number as two. The boundary rules put that character in the class that joins what it finds on both sides, so the number stays whole and the answer is two words.

The flags run the other way and are stranger. A flag is two characters, neither of them a letter, and there is a rule that counts them in pairs so that a flag is never cut in half. Two flags in a row therefore give two segments, correctly, where the space rule gives one piece holding both.

Neither segment is a word by the ordinary test, since a word has to hold a letter or a digit and a flag holds neither, so the count of words for that text is zero. The rules found the right boundary and then the word test threw both pieces away, which is the two halves of the method disagreeing rather than either being wrong.

The rules cut a text into segments that tile it, and then somebody has to say which segments are words. That second step is not the standard’s and there is more than one defensible answer, so it is worth seeing both. On the running sentence the rules find 19 segments. Nine of them hold a letter or a digit and are the nine words of section 3. The other ten are the six spaces, the two hyphens and the two full stops.

Keeping the marks costs 15 more pieces on the six sentences and buys back everything the segments held, so the text can be reproduced from the pieces and the spacing. Dropping them is the usual choice and it is the one measured everywhere else on this page. What matters is that the choice sits outside the rules, which is why two implementations of the same standard can hand you different answers without either having got the standard wrong.

The repair is not free and the first bill is length. Over the six sentences the boundary rules produce 69 words where cutting at spaces produces 63, which is about a tenth more, and the extra pieces are the halves of the compounds. Anything reading the text afterwards pays per piece, and for a model that compares every position with every other position it pays roughly with the square of that count.

The second bill is the writing itself. Adding up the spans says how much of a text survived into a word, and the boundary rules keep less of it than the space rule does, because a mark that is not part of a word is not returned as one.

Gluing the nine words back with single spaces gives Dr Alvarez didn’t expect the low cost re analysis, which is a line with the abbreviation’s stop gone, both compounds pulled apart and the sentence’s own full stop missing. The space rule’s seven pieces glue back exactly. So the choice between the two is a choice about which loss is cheaper, and a vocabulary is glad to be rid of the marks while anything that has to hand the reader back their own text is not.

The loss is recoverable while the source is still in hand, because every word carries the offsets it came from and the characters between one word and the next are still there to be read. It is not recoverable from a list of words, which is what most things downstream are given.

There are texts where the simpler rule wins outright, and they are not exotic. Three of them are in the panel in section 10. An address comes back as two words with the character that makes it an address gone, so 18 of its 19 characters survive and the thing itself does not. A file name comes back as three.

A text holding a single approving hand and nothing else comes back as no words at all, because the rules found the segment correctly and then the word test refused it for holding no letter and no digit.

What those three have in common is that they were not written as prose. The rules were designed for what a person reading a document would call a word, and a message identifier, a path and a picture are none of them prose. The space rule keeps them whole for the uninteresting reason that it keeps everything whole, which is the right answer here by accident rather than by design.

It is easy to read a standard with the word “word” in the title as having settled what a word is, and this one does not, on purpose. What it fixes is a set of positions at which a break is permitted. Which of the resulting stretches count as words, whether the marks are kept, and whether the rules should be adjusted for a particular language are all outside it, and the annex says as much by calling its own rules a default and inviting tailoring.

The measurement in section 9 says the same thing without any appeal to the document. Two careful rules put 35 of 47 entries in the same place and 12 in different places, and nothing in the writing decides the 12. Whether low-cost is one unit or two depends on whether a dictionary, a search index or a typesetter is asking, and all three are right about their own purpose. A standard can make every implementation give the same answer, and this one did, which is why a double-click selects the same characters in two different programs. It cannot make that answer correct, because there is nothing here for it to be correct about.

The rules see classes of characters and nothing else, so any question whose answer lives in a language rather than in the writing is outside what they can decide. The full stop is the clearest case, and it already appears twice in the running sentence doing two different jobs. A sentence carrying an abbreviation written with two stops shows all three readings at once.

The U.S. sensors arrived.

the stop inside U.S.       kept, a letter stands on each side
the stop closing U.S.      dropped, a space follows it
the stop ending the line   dropped, nothing follows it

That is not a defect to be patched. The information that would separate the two is a list of the abbreviations of a particular language, and the moment a rule carries such a list it is a rule for that language and has given up the property that brought us here. Worse, the case has no correct answer even with the list, since a sentence ending in an abbreviation is written with one stop doing both jobs and no division of the characters assigns it to both.

The most unusual thing about this standard, read next to most standards, is that it names its own limits in the document. The scripts written without spaces are not handled badly by accident. Their letters are deliberately left out of the class that holds letters together, so the catch-all applies on both sides of every one of them, and the annex states that finding words in those scripts requires a dictionary or a model and that these rules are not a substitute.

What follows is a statement about the method rather than about any text. Where the writing separates its words with characters the rules can see, whether that is a space in English or a change of class in a Japanese sentence, the answer is determined by the writing. Where it does not, the answer is determined by nothing, and cutting at every character is a refusal written as an answer.

The nine words the Chinese line came back as are nine correct applications of a rule that says break when nothing else applies, and there were four words there.

Everything above suggests the rules are a convention, and they are. It would be a mistake to hear that as a criticism. A convention that everybody follows has properties a better rule that nobody follows does not, and they are worth naming, since they are the actual reason to use this rule rather than to invent one.

Two independent programs following the annex select the same thing when a reader double-clicks, so a document behaves the same way in both. The rules are published with a version, so an answer can be reproduced years later by saying which version produced it. They cover every script the standard covers rather than the one their author reads, so a program does not fail when it meets a language nobody tested it on.

And because the rules are stated on classes rather than on characters, a script added to the standard later gets sensible behaviour without any program being rewritten.

None of those is a claim that a word ends where the annex says it ends. They are claims about agreement, and agreement is what a person actually needs from the piece of a system that decides where words are, since the decision is going to be made identically millions of times and the cost of two parts of one pipeline making it differently is far larger than the cost of either being slightly wrong.

The cases below are gathered from the whole page. The right column says what the method does not determine and why, and it quotes a number where the number is a fact about the method rather than about any particular text.