Bytes and Characters

Every page in this section carries the same sentence, and this one carries beside it the eighteen sentences the merging page is built from. They are ordinary English about reports and costs and analyses, they were written so that every character the sentence uses turns up somewhere in them, and they do not contain the sentence. That last part is the whole arrangement, since a vocabulary is only interesting on writing it was not built from.

Dr. Alvarez didn't expect the low-cost re-analysis.

Merging pairs over those eighteen sentences until no pair is left that occurs twice gives a table of 137 entries after 85 merges, and it reads the sentence above as 25 numbers. That is a good vocabulary doing exactly what it was fitted to do, and 25 is roughly half of what either reading on this page will need for the same sentence. We are going to spend this Part on what it cannot do, and then the rest of the page on the answer that gives up the 25 in exchange.

Encode the sentence against the merged table, then read the numbers straight back and compare what returns against what went in, character for character. It does not return. The surname loses its last letter and the space after it, and the next word runs into what is left.

Dr. Alvarez didn't expect the low-cost re-analysis.
Dr. Alvare[UNK]didn't expect the low-cost re-analysis.

The reason is worth having in full, because it is not the reason it looks like. The corpus does contain the letter z, in the word size, so the letter is not missing from the alphabet. What is missing is the letter z at the end of a word, which is a different symbol in a scheme that marks where words end, and no word in those eighteen sentences finishes with one. So one of the 25 numbers is the stand-in entry, meaning “something was here”, and a stand-in carries no word boundary because it carries nothing at all. Losing the letter costs one character; losing the boundary means everything after it is read as part of the word in front, which is why the surname and the contraction come back joined.

Now put something harder to the same table. The first text below is our sentence, the second is the same sentence with two Scandinavian letters in it, and the third is one short sentence of Greek. What comes back after the numbers are read again is underneath each.

…

The third block is where the method stops working rather than works badly. Twenty-one characters of Greek become 18 numbers, seventeen of which are the same stand-in, and what returns is seventeen copies of that stand-in with the spaces gone and a full stop at the end. A model shown this is not shown a degraded sentence. It is shown the fact that something it could not read was there, seventeen times over, and nothing in those numbers distinguishes the sentence from any other Greek sentence with the same count of letters.

There are two directions out of this. One is to keep learning a vocabulary and to make it a better one, wider, fitted to more languages, refitted when it ages. That helps, and it never finishes, because whatever corpus it was fitted to is finite and the set of things people write is not. The other is to stop learning a vocabulary at all and to read the text as the units it is already made of, so that the question of an unfamiliar piece has no way of arising.

The second direction has two versions and they are not the same one. Reading a text as its characters gives one number per character, and reading it as its bytes gives one number per byte. On the English sentence above those are the same length and it is tempting to treat them as one idea. They are not, and Part 4 is about exactly where they come apart.

The first version takes the characters themselves as the units. Gather every distinct character the corpus contains, spaces and punctuation included, put them in a fixed order, and the table is finished. There is no counting of pairs, no merging, no size to choose and nothing to stop early. Fitting is one pass that collects a set.

On the eighteen sentences that set has 36 members, so the table has 37 entries once one is reserved to stand in for anything else. Compare that with the 137 the merging produced from the same text, and with the 256 the next section arrives at. A character table is the smallest thing here that can still spell the corpus it came from.

The second version goes one level down and stops needing a corpus. A text stored on a computer is a run of bytes, a byte holds one of exactly 256 values, and so a table with one entry per byte value spells every text that can be stored, in any script, whether or not anybody has ever written it. Nothing is fitted. The table is complete before the first text is read and it is the same table for everyone.

a byte value  =  a whole number from 0 to 255
the table     =  all 256 of them, in order
the number    =  the byte value itself

The number an entry goes in as is the byte value, so there is nothing to look up in either direction. What does need a decision is how to write the entries down, since a table is read by people and 68 of the 256 values have nothing visible to stand for them, the control characters and the space among them.

The usual answer, and the one used here, lets the other 188 stand for themselves and lends those 68 a shape from a block of letters no byte value occupies, which is why the space shows up as Ġ in the playground above. The choice is arbitrary in every respect except that it is fixed, and it does not affect a single number.

Here is the whole of it on our sentence, which is 51 characters long and happens to be 51 bytes long as well. The first four numbers under each reading are below, and they are the same four numbers for a different reason in each case.

as bytes        D    r    .    ␣
                68  114   46   32

as characters   D    r    .    ␣
                 7   28    4    1

Read either run of numbers back and the sentence returns character for character, including both full stops, the apostrophe, the hyphens and every space. Neither reading reached a stand-in, and the merged table on the same sentence reached one and lost a space.

A reader who has followed section 6 will ask the obvious question. If a byte table can be complete because there are only 256 byte values, why not a character table that is complete because Unicode has a fixed number of characters? The number a piece goes in as would simply be its codepoint, no corpus, no stand-in, one number per character rather than one per byte.

Allocating a learned row for every possible Unicode code point would make a very large embedding table, including many rows that text never uses. Compare that with one row per byte value at the same width.

code-point table = 1,114,112 × 768 = 855,638,016 values
byte table = 256 × 768 = 196,608 values
storage ratio = 1,114,112 / 256 = 4,352

The smaller table is the attraction of byte inputs. Later sections measure their cost in sequence length.

The guarantee is worth stating carefully, because it is narrower than it sounds and it is genuinely absolute inside its own boundary. Every text that can be stored is a run of bytes; every byte holds one of 256 values; the table holds all 256. So for any text at all, in any script, in any language, every piece the reading produces is in the table, and the lookup cannot fail.

text  →  bytes  →  numbers  →  bytes  →  text

Notice what has happened to the argument. Coverage stopped being a property of the corpus and became a property of the scheme. A character table covers whatever its corpus covered and no more, so its guarantee is a statement about eighteen sentences; the byte table’s guarantee is a statement about arithmetic, and reading a million more texts would not improve it because there is nothing left to improve.

The clearest way to see the difference is to let one language arrive at a time. Start with the eighteen English sentences, add one short sentence of French, then German, and so on through twelve languages, counting after each how many distinct characters everything so far has used and how many distinct byte values it has touched.

…

The character count starts at 36 and ends at 117, and it is still climbing steeply at the last sentence, with Greek adding 13 and Russian another 13 and the Chinese sentence adding 4 more from six characters of text. Nothing about that curve suggests it is about to flatten, since Han alone has tens of thousands of characters and these texts have met six of them. The byte table is 256 at every point on the chart, including at the left edge before any of the twelve arrived.

The green line is the honest qualification. Only 103 byte values are ever touched by all thirty texts together, so 153 entries of the byte table have never been used and are being carried anyway. That is what the guarantee costs when it is paid in entries rather than in coverage.

It is easy to over-read what has been bought, so here is the boundary drawn plainly. The guarantee says every text can be turned into numbers and turned back again. It says nothing whatever about those numbers being useful to a model that has never met writing of that kind before.

The Chinese sentence in Part 5 is the sharpest instance. The byte reading turns it into 18 numbers and gives it back exactly, and a model trained only on English has seen those particular byte values in almost no context and knows nothing whatever about what they spell. Representable and learnable are separate questions, and only the first has been settled here.

Everything so far has quietly used one sentence in which the two readings agree, and it is time to stop. The encoding that turns characters into bytes, which is UTF-8 everywhere on this page, does not use the same number of bytes for every character. It uses one byte for the characters of basic English, two for most accented Latin letters and for Greek, Cyrillic, Hebrew and Arabic, three for the Indic scripts and for Chinese, Japanese and Korean, and four for the rest, emoji included.

That is why the running sentence came to 51 either way. It is written entirely in characters that take one byte each, so the two readings cut it in exactly the same places and differ only in which number each piece goes in as. An English sentence cannot show the difference between the two methods on this page, and any measurement taken only on English will report that there is none.

Put two more texts beside the running sentence, the sentence itself with a unit of length written into it that carries two Scandinavian letters, and one short sentence of Greek, and the disagreement appears immediately. The accented text is 69 characters and 71 bytes, since two of its characters take two bytes each. The Greek sentence is 21 characters and 38 bytes, nearly twice as many.

Both columns move at once and they move for different reasons. The byte count rises because those characters need more than one byte, which is a fact about the encoding and would be the same for any corpus. The last column rises because the corpus never met those characters, which is a fact about the eighteen sentences and would fall to zero if they had. Only the first of those two goes away with more data.

The sharpest test of any of this is to encode a text, decode the numbers immediately with no model in between, and compare what returns against what went in. Anything the reading discarded shows up, and nothing else does. Here are the same three texts under both readings.

…

The byte reading is exact three times out of three and would be exact on any text put to it. The character reading is exact on the first and loses two characters on the second and seventeen on the third, and what it gives back in place of them is the stand-in’s own spelling, so the two Scandinavian letters come back as a word with two square-bracketed markers embedded in it.

The failure is the same shape as the merged vocabulary’s in Part 1, and it is smaller only because a character table is surprised less often than a table of merged pieces.

There is one place where the byte reading can produce something the character reading cannot, and it is the price of the same design that made the guarantee possible. A character takes between one and four bytes, so a run of byte numbers can stop part way through a character, and such a run does not stand for any text at all. It is not an unfamiliar piece and it is not a refusal. It is a run of numbers with no reading.

é          =  195  169
195 alone  =  �
169 alone  =  �

What comes back in place of a broken position is the mark above, which means that a character stood here and could not be read. The two halves of the letter fail for different reasons. The 195 opens a two-byte character and means nothing until the number after it arrives, and the 169 finishes a character and means nothing without the number that opened it.

Of the 256 numbers in the table, exactly 128 stand for a character on their own, the values 0 to 127, which are the one-byte characters of basic English. The other 128, the values 128 to 255, are never a character alone. They occur only inside a character of two, three or four bytes, so a run that stops just after one of them may have stopped in the middle of a letter.

That is not an edge case reached by malformed input. It is what a model generating numbers one at a time walks into every time it is interrupted, and the widget below shows how often. The Greek sentence has 38 numbers and 17 of its 38 stopping points land in the middle of a letter; the English sentence has 24 numbers and none of them do.

…

How narrow this is can be measured directly. Drawing runs of numbers uniformly at random, 546 of 2000 runs of length two are text, 209 of 2000 at length four, 24 at length eight, and none at all of the two thousand runs of length sixteen. Almost every run of byte numbers is not writing, so a model reading bytes spends part of its capacity learning which runs are, before it can learn anything about what they say.

Now the bill. The merged vocabulary from Part 1 reads the running sentence as 25 numbers and the byte reading takes 51, a little over twice as many. Across the whole eighteen-sentence corpus the merged reading takes 263 and the byte reading takes 763, which is 2.90 times as many on writing that is entirely English.

763 / 263    ≈  2.90
2.90 × 2.90  ≈  8.4

A reading is judged on two numbers together, how many entries its table holds and how many numbers a text becomes, and it is worth noticing that the three readings on this page do not order the same way on both. The byte table is the largest of the three at 256 entries and ties for the longest sequences. The character table is the smallest at 37 and gives sequences of the same length. The merged table is in between on entries and shortest on sequences.

Which is to say that on this corpus, and on English generally, the byte reading is worse than the character reading on both numbers at once. It carries nearly seven times the entries for exactly the same sequence length and the same exact round trip. Anybody measuring the two on English alone should conclude that the byte reading is a strictly worse choice, and on English alone they would be right.

So take one short sentence with one meaning, the report arrived late, and write it in twelve languages. The ratio between the two readings on each is the multiplier the byte reading charges for that writing system, and it is not a small spread.

…

English pays exactly 1.00, since every character it uses takes one byte. French, German and Spanish pay between 1.03 and 1.04, because only one character in each sentence is accented, and Polish pays 1.15. Greek pays 1.81, Russian 1.85 and Arabic 1.84, which is two bytes a letter diluted by the spaces and the full stop, which take one each. Japanese and Chinese pay exactly 3.00, since every character in both sentences takes three bytes and neither sentence contains a space.

Η έκθεση έφτασε αργά.

letters               17 × 2  =  34 bytes
spaces and stop        4 × 1  =   4 bytes

bytes                 34 + 4  =  38
bytes per character  38 / 21  ≈  1.81

I expected the per-character ratios to settle it and they do not, and the reason is worth more than the ratios were. A model is charged for the numbers a text becomes, not for the numbers per character, and the same meaning takes a different number of characters in different writing systems. Measure the twelve sentences by what they actually cost and the ordering changes.

The Chinese sentence is the cheapest of the twelve. It pays the worst multiplier on the page and still costs 18 numbers against English’s 24, because it says the same thing in six characters where English takes twenty-four. Hindi is the most expensive at 48, twice the English figure, and it gets there with fewer characters than English uses.

So the textbook statement that a byte reading charges non-Latin scripts three times as much did not hold when I measured the thing a model is actually charged for, and the language it is worst for on this list is not one of the three-byte ones.

Reading a text as its characters requires knowing what one character is, and that question has no answer the method supplies. The obvious reading is one codepoint, which is what every count on this page uses, and it disagrees with what a reader would point at in several ordinary cases.

The word café can be written two ways that look identical on the page. One spells the last letter as a single codepoint; the other spells it as a plain e followed by a codepoint meaning “put an acute accent on the thing before me”. Those are 4 characters and 5 characters, 5 bytes and 6 bytes, and they are different texts by every comparison a computer makes, so a model reads them as different words.

Emoji make it worse, since a tinted thumb is two codepoints, a family is five and a Welsh flag is seven, and a reader would call each of those one mark.

There are three defensible answers and each costs something. Counting codepoints is simple and splits marks a reader considers single. Counting what Unicode calls grapheme clusters matches a reader and needs a large body of tables that has to be shipped and updated. Rewriting every text into one preferred spelling before reading it makes the two cafés the same word and makes the round trip inexact, which is a real loss for anything that has to reproduce its input.

The byte reading avoids the question entirely, and pays for that by having no notion of a character at all.

The deeper limit here is no degenerate input at all. It is the method working exactly as designed. A byte is a unit of storage, it was never a unit of meaning, and nothing in either reading makes it one. A model reading bytes therefore has to learn that a particular run of them is a word before it can learn anything about the word, and it has to learn that afresh for every spelling, every inflection and every script.

The distance it must look across grows in the same proportion as the sequence. A paragraph that fitted in a thousand numbers under the merged vocabulary takes 2,900 under the byte reading, so anything the model has to relate across that paragraph is nearly three times further away than it was. That is the 2.90 from section 16 read as a distance rather than as a cost.

And there is nothing here that knows a piece is common. The whole of what a learned vocabulary buys is that frequent things become cheap, since the merging spends its entries on what recurs and leaves the rest to be spelled out. Both readings on this page charge for every character alike, so the three-letter word the costs three numbers every time it appears and would still cost three if it were the only word in the language.

Giving that up is what buys the coverage guarantee, and putting it back is the job of whatever groups the bytes into larger units afterwards.

Gathered in one place, these are the inputs on which the method stops being defined rather than becoming approximate, together with what has to be decided in each case and what turns on the decision. The ones with a genuine choice attached are the ones worth a reader’s attention, since nothing in the mathematics makes them and somebody has to.