Penn Treebank Rules

Both rules on the two pages before this one were answering a question about writing. Splitting on spaces asks where the gaps are, and the boundary rules ask, at each position between two characters, whether a word is allowed to end there. Neither asks anything about English, and that is a strength when the text might be in Thai.

These rules were written by somebody with a different problem. A person was about to sit down and mark, for every unit in a sentence, what part of speech it was and which units it grouped with. That person needs one row per thing the grammar has a name for. We carry the same sentence the other two pages carry, because it already breaks in three separate places.

Dr. Alvarez didn't expect the low-cost re-analysis.

Read it as an annotator. The stop after Dr belongs to the abbreviation and is part of the word being labelled, while the stop at the end belongs to the sentence and is a separate thing that has its own tag. And didn’t has to be two rows, since the verb takes one tag and the negation takes another, and a parser that saw them fused would have to learn a separate tag for every verb in English with a negation stuck to it. None of that is visible in the spaces, and none of it is visible in the characters either.

Here is the whole method applied once. Nine pieces, each shown with the half-open span of the sentence it stands for, so the first number is where it starts and the second is one past where it stops.

Dr.            [0, 3)
Alvarez        [4, 11)
did            [12, 15)
n't            [15, 18)
expect         [19, 25)
the            [26, 29)
low-cost       [30, 38)
re-analysis    [39, 50)
.              [50, 51)

Every one of those spans is honest, in the sense that reading the sentence between the two numbers gives back the piece printed beside them, character for character. That is worth saying out loud now because it stops being true in Part 4, and the sentence that breaks it is this same sentence inside a pair of quotation marks.

The three rules give 7, 9 and 9 pieces on that sentence, and the two nines are not the same nine. Reading them together is the quickest way to see what each rule thinks a word is.

The disagreements run in both directions, which is what makes them informative. The boundary rules split low-cost into two and these rules do not, since a hyphenated compound is one thing to a grammar and gets one tag. These rules split didn’t into two and the boundary rules do not, since an apostrophe between two letters is glue in the standard’s table and a morpheme boundary to a linguist. Neither rule is reading the other’s question wrongly.

It is worth being exact about how little there is here, because the rest of the page turns on it. The whole method is one pass over each run of non-space characters, asking three kinds of question.

There is no dictionary of English, no list of abbreviations, no frequency count and nothing that was learned from a corpus. The cost is one pass over the text and a fixed amount of work per run of characters, which is what the rest of this page is spending. Where each of the two earlier rules is a single clause, this is a dozen or so, and every one of them is a repair to a particular way splitting on spaces went wrong on English newspaper prose.

This is the clause that has no equivalent in either earlier rule, and it is the one that most clearly comes from the annotation rather than from the writing. There is no space inside didn’t and no mark that either earlier rule treats as a separator, and the rules cut it anyway, into did and n’t.

The reason is that an annotator labelling that sentence has to write down two facts, that there is a past tense auxiliary verb and that it is negated. Those are two tags, so they need two rows. The negation is written n’t and it is one morpheme, so the cut goes in front of the n rather than in front of the apostrophe, and the same treatment covers eight endings in all.

It's        ->  It    's
they're     ->  they  're
we've       ->  we    've
I'll        ->  I     'll
he'd        ->  he    'd
I'm         ->  I     'm
isn't       ->  is    n't
the dogs'   ->  the   dogs   '

Over the six sentences this section shares, exactly one piece comes from this clause, since the notebook holds one contraction. Over the wider corpus further down the page, 26 of its 266 pieces are one of these endings or one half of a fixed form, which is an upper bound rather than a count, because four of the spellings a fixed form produces are ordinary English words that also turn up on their own.

If the negation is always n’t, then whatever is left in front of it is whatever is left. For most verbs that is a word. For three of them it is not, and the rules produce a piece that no dictionary of English holds.

did  n't        the host is a word
is   n't        the host is a word
ca   n't        the host is what remains of can
wo   n't        the host is what remains of will
sha  n't        the host is what remains of shall

English spelt those three contractions by changing the verb as well as adding the negation, so there is no division of the characters that gives both morphemes their own letters. The rules take the position that the negation is the fixed part and the host absorbs the irregularity, which is why can’t comes back as ca and n’t, and the cut was placed there deliberately rather than falling out of the pattern by accident.

Some contractions leave no mark at all. cannot is two words to a grammar and six letters with nothing between them, so no rule reading the characters can find the join. The treebank rules handle those by naming them, ten forms in all, each with the place inside it where the cut goes.

…

The first nine rows are the rules working. The last six are the ones to read carefully, because in every one of them the answer is a single piece and nothing anywhere reports that a list was consulted and came up empty. The same apostrophe that gets didn’t cut in two leaves y’all whole, and the curved apostrophe that a word processor inserts by default leaves the very same contraction whole as well.

The failure that the splitting on spaces page measures is that a mark of punctuation is not white space, so a run of non-space characters carries it along and away. becomes a different entry from away. These rules repair it the direct way, by naming the characters that are a piece on their own wherever they stand.

; @ # $ % & ? !        a piece on its own, always
( ) [ ] { } < >        a piece on its own, always
" and a pair of '      a piece on its own, and rewritten
-- and ...             a piece on its own, taken whole
, and :                a piece on its own, unless a digit follows
.                      a piece on its own, only at the end

The first two lines are worth noticing for what they do not say. There is no condition on them at all, so a mark from that set is separated whatever surrounds it, and a@b.com comes back as three pieces while AT&T comes back as three as well. On newspaper prose that is right almost every time, since those characters really are punctuation there. On an address it is a piece of writing pulled apart into its punctuation and its letters.

A comma and a colon are the two marks that do a second job inside a number, and the rules keep them there with a condition a single character wide. A comma is separated unless the very next character is a digit.

1,000     ->  1,000        a digit follows the comma
3:30      ->  3:30         a digit follows the colon
yes, no   ->  yes  ,  no   a space follows the comma
note: yes ->  note : yes   a space follows the colon

It is a cheap rule and it is not the rule anybody would write from the definition. What a person means is that the comma is inside a number, and what the rules test is that a digit comes next, which is neither necessary nor sufficient in general. It happens to be right on every number in the corpus below, and it is right for the reason that English writes thousands groups with no space around the comma.

The same shape of condition, one character of context, is what the boundary rules use for their own version of this case, though they reach it from the other side by putting the comma in a class that joins two digits. Two different methods arrive at the same behaviour on 1,000, and they disagree on 3:30, which these rules keep whole and the standard cuts into two.

The full stop is the hard one, and it is hard for a reason that no amount of care about characters removes. In Dr. the stop is part of the word and should stay; at the end of the sentence it is a mark of its own and should come off; and the two are the same character.

The rules settle it without a list of abbreviations, by using a fact about where the text came from. The script ran on one sentence at a time, after somebody or something else had already decided where the sentences were, so a stop that is not the last thing in the text cannot be ending a sentence and must therefore be an abbreviation’s.

the last character of the text,
ignoring trailing spaces, closing brackets and closing quotes,
and not itself preceded by another stop
        is  a piece of its own
every other stop  stays where it is

Read that as an assumption rather than as a rule, because it is one. It says the text is exactly one sentence. Where that holds the answer is right on both stops of the running sentence at once, which no rule reading only the characters around each stop could manage. Where it does not hold, nothing raises an objection.

The first failure is the one anybody meets on their first paragraph. Give the rules two sentences in one string and the first sentence’s stop is not the last character of the text, so it is read as an abbreviation’s and stays attached.

…

The measurement underneath the probes is the useful one. Our six sentences give 72 pieces when they arrive one at a time and 67 when they arrive joined, and the five pieces that went missing are five full stops that stayed stuck to the word in front of them. Six stops came back on their own in the first case and one in the second, and the corpus grew five entries that are a word with a stop on it, including re-analysis. sitting beside the re-analysis that is already there.

Notice which of the six pieces in the amber list is legitimate. Only Dr. is an abbreviation; the other five are sentences whose own punctuation was swallowed. The rules cannot tell the difference and neither can their output, since the two cases produce the same shape of piece.

The second failure runs the other way and is the one the rules cannot repair even in principle. A sentence that ends in an abbreviation is written with one stop doing both jobs, so the final-stop rule fires, and the abbreviation loses the stop that belongs to it.

She consulted Dr.
        ->  She    consulted    Dr    .

The analysis was run by Alvarez et al.
        ->  The  analysis  was  run  by  Alvarez  et  al  .

A list of abbreviations would fix the first failure and not this one. Knowing that al. is an abbreviation tells us the stop belongs to it; it does not tell us where the sentence’s own stop went, because there is not a second one. The two marks were written as one, and no division of the characters into pieces can assign that character to two pieces at once.

Everything so far has been a decision about where to cut. This clause is different in kind, and it is what makes these rules unlike the two before them. A quotation mark is not only separated from the word beside it, it is replaced by something that is not in the text.

an opening "    becomes  two backquotes
a closing "    becomes  two apostrophes

The purpose is to record a distinction the writing was already making and the character was not. A reader of she said “wait” can see which mark opens and which closes; the plain typewriter quotation mark is the same character both times, so a parser reading the corpus would have to work it out again. The rules work it out once, from whether the mark begins the text or follows a space or an opening bracket, and write the answer into the piece.

The spelling chosen was the one a typist of the period used for directional quotes on a machine that had no directional quotes, which is why an opening quotation mark in treebank text is two backquotes and a closing one is two apostrophes. That is a convention of a particular decade’s typing, and it is the reason a modern reader meeting treebank output for the first time usually thinks something has gone wrong.

There is a second rewriting, switched off by default and on in the treebank itself, and its reason is worth telling because it is not linguistic at all. A parsed sentence in the corpus is written as a tree, and a tree is written with round brackets around each group. A round bracket that is part of the sentence therefore cannot be left as a round bracket, or anything reading the file cannot tell the two apart.

The last row is the tell. If this were a rule about brackets it would cover all of them. It covers exactly the six that the storage format needed escaping, and leaves the two it did not. A choice about how the annotation was written down reached back and changed what a word is, and every parser trained on that corpus has been expecting the escaped spelling ever since.

Both rewritings raise the same question, which is what a piece is now supposed to be. A piece has two halves, the text it holds and the stretch of source it came from, and until this point those two have agreed. They no longer do, and the pieces keep both rather than choosing.

…

Read the third line of the two amber boxes. Each of them holds two characters and points at a stretch of the sentence that is one character wide, so the length of a piece and the length of what it stands for have come apart. The span says where the piece came from, and the text says what it became, and a rule that normalises anything at all has to be allowed to say two different things there.

How much of a corpus is affected is a fair question and the answer is small, which is what makes the case interesting rather than alarming. Over the 22 sentences in the panel above, chosen so that every clause on this page is exercised at least once, 14 of 266 pieces are not the source their span covers.

Five per cent sounds like a small enough share to ignore and it is not, because what the rewriting costs does not shrink with it. A corpus where one piece in twenty is not a copy of the writing is a corpus that cannot be reassembled from its pieces, and it is a corpus in which searching the pieces for a quotation mark returns nothing at all, since none of them holds that character any more.

The bill everything downstream pays is length, and these rules are the longest of the three on both corpora. Anything reading the text afterwards pays per piece, and a model that compares every position with every other position pays roughly with the square of the count.

…

On the six sentences the three rules give 72, 63 and 69 pieces, and the gap is modest because that notebook has one contraction and few marks in it. On the wider corpus, which is newspaper prose with quotations, money, percentages and brackets in it, the same three rules give 266, 155 and 159. That is 72 per cent more pieces than splitting on spaces produces from identical writing, and every one of them is paid for on every pass over the corpus.

Reading that difference by what the extra pieces are made of shows where it went. Of the 266 pieces from those 22 sentences, 86 hold no letter and no digit anywhere in them, so roughly one piece in three is a mark of punctuation standing on its own.

Whether that is a cost or a benefit depends entirely on what is reading them. To the annotator it was a benefit and the whole point, since a comma has a part of speech in the treebank and needs a row. To a model counting how often words turn up together it is a third of every window spent on marks that carry almost no information about the sentence around them. The rules make one choice and everything downstream lives with it.

The other component is smaller than it looks. At most 26 of the 266 pieces come from the contraction and clitic clauses, and four of the spellings counted there are ordinary words that also appear on their own, so the true figure is lower. Splitting contractions is the clause that carries the most linguistics, and it accounts for at most 26 of the 111 pieces by which these rules exceed splitting on spaces, against 86 from separating the marks.

There is one column of that table where these rules are ahead and it is easy to miss, because the number is identical to the space rule’s. Over the six sentences the treebank rules and the space rule both keep 318 of the 375 characters, and the boundary rules keep 303.

318 is exactly the number of characters in those six sentences that are not spaces, so the treebank rules kept every single one of them. That is a real difference from the boundary rules, which found the marks correctly and then discarded them as not being words, and it is what makes these rules usable where the original text has to be handed back. A hyphen, a comma and a full stop all survive as pieces here and none of them survives there.

Pointed at writing that is not newspaper prose, these rules come apart in ways that splitting on spaces does not, and the panel in the playground has the cases. An address is separated at the character that makes it an address, so a@b.com becomes three pieces where the space rule gives one. A company name written with an ampersand becomes three. A percentage becomes two.

What those share is that the separated character really is punctuation in prose and really is not punctuation there. The rules were written for a corpus of newspaper articles and they read every text as though it were one, which on an address is the same failure the earlier pages describe from the other direction, an assumption about the writing that the rule never states.

There is a case running the other way that is worth reporting beside it. A file name such as re-analysis_2019.csv comes back whole from these rules and in three pieces from the boundary rules, because a hyphen is nothing to these rules and a separator to the standard. Neither rule is better at non-prose in general; each is accidentally right about a different kind of it.

Everything on this page has been a decision, and every one of those decisions was answerable only because the person making it knew which corpus was being annotated, in which language, under which house style. Take that away and most of the clauses have no content.

The hyphen is the cleanest case. These rules leave low-cost whole and the standard cuts it in two, and nothing in the writing decides between them. What decided it here is that the annotation gave one tag to a compound modifier, which is a claim about the grammar being used to describe English, and a different descriptive framework would have wanted two rows and got a different rule. The rules are the shadow of a particular linguistic analysis, and they are exact only relative to it.

That is not the criticism it sounds like, because it is the same thing that makes them useful. Everything trained on the treebank agrees about what a unit is, and agreement is what a person actually needs from the part of a pipeline that decides where the words are, since the decision is made identically millions of times and two parts of one system disagreeing about it costs far more than either being slightly odd.

Two of the clauses are lists, ten whole forms and eight endings, and a list is finite in a way a rule about characters is not. The eight endings cover the English clitics as they were spelled in the corpus. The ten forms cover the spoken contractions somebody thought of.

What matters is the shape of the failure rather than its size. A form that is on the list is cut; a form that is not is returned whole; and those two answers are the same answer. A caller receives one piece and has no way to ask whether that is because the text was examined and found not to need cutting or because no clause covers it at all. Six of the fifteen probes in section 7 fall into the second case and are indistinguishable from the first.

Adding entries does not change this and cannot. Whatever is on the list, there is a next form, and the writing gives no signal that would let a rule notice it has met one. Any method that decides a boundary by consulting a finite list of forms inherits the same silence, and the only escape is a method that infers the boundary from a corpus rather than from a list, which is what the rest of this section of the site is about.

Several clauses depend on how the text was typed rather than on what it says, and both of the assumptions they make were true of a corpus keyed in the late nineteen eighties and are false of a good deal of writing now.

The apostrophe is one character to these rules and there are two in common use, since the curved one is what a word processor inserts by default. The same contraction typed the modern way is returned whole. Quotation marks have the same problem twice over, since the rules read the straight typewriter mark and rewrite it into a typewriter convention, and curved quotation marks are neither read nor produced.

And the final-stop clause assumes a step before it that finds the sentences, which was true in a pipeline built around this script and is not true of a string arriving from anywhere else.

There is a subtler one worth naming. The rules as published are a pipeline of substitutions rather than a specification, so on writing they did not anticipate, a doubled comma or a run of three backquotes, the answer falls out of how one substitution’s replacement lands in the next one’s search. Nobody decided those cases, and two careful readings of the same rules can therefore differ on them without either being unfaithful.

The rewriting has a consequence that outlived every other choice on this page, and it is the reason later methods gave it up. Once a piece can be something other than the writing it came from, a list of pieces is no longer a division of a text.

the pieces joined with one space each
        ``  Dr.  Alvarez  did  n't  ...  ,  ''  she  wrote  .

the source
        "Dr. Alvarez didn't expect the low-cost re-analysis," she wrote.

Two separate things went wrong there. The spacing is gone, which is true of every rule on these three pages and is recoverable if one knows the language’s conventions, and the quotation marks are gone, which is not recoverable at all from the pieces, since two apostrophes might have been a closing quotation mark or might have been two apostrophes. The pieces plus the spans plus the original text do recover everything; the pieces alone recover nothing exactly.

Every method that came afterwards had to give this up in order to be usable for anything that hands the reader back their own text. That is why the methods later in this section are described as reversible or not reversible, and why the ones that keep the space attached to the word after it are keeping it for this reason rather than out of tidiness.

The cases below are gathered from the whole page. The right column says what the method does not determine and why, and it quotes a number where the number is a fact about the method rather than about any particular text.