Moses Rules

The three rules on the pages before this one were each answering a question and none of them was asked what happened next. Splitting on spaces asks where the gaps are, the boundary rules ask which pairs of characters a word may break between, and the annotation rules ask which units a person marking up grammar needs a row for. Every one of those ends when the pieces come out.

These rules were written by somebody with the pieces coming out of one end of a system and going back in at the other. A translation system was trained on pieces, decoded into pieces, and then had to hand a reader a sentence, so whatever was done on the way in had to be undoable on the way out. We carry the same sentence the other pages carry, because it breaks in the three places that matter here.

Dr. Alvarez didn't expect the low-cost re-analysis.

Read it as somebody building a phrase table. The stop after Dr has to stay, because Dr. and Dr would otherwise be two rows for one title. The stop at the end has to come off, for the same reason from the other side, since re-analysis. and re-analysis are one word written twice. And the apostrophe in didn’t hides a negation that the target language will spell as a word of its own, so it has to be visible to whatever aligns the two sides.

Here is the whole method applied once. Nine pieces, each shown with the half-open span of the sentence it stands for, so the first number is where it starts and the second is one past where it stops.

Dr.            [0, 3)
Alvarez        [4, 11)
didn           [12, 16)
't             [16, 18)
expect         [19, 25)
the            [26, 29)
low-cost       [30, 38)
re-analysis    [39, 50)
.              [50, 51)

Read the spans down the right-hand side and they run continuously through the sentence, skipping only the six spaces. Every piece is the slice of the sentence its two numbers name, character for character, and unlike the previous page that remains true in every part of this one.

The annotation rules give nine pieces on that sentence too, which is why the two are so often taken for one method. They are not the same nine. Seven of the pieces are shared, and the two that are not are the halves of the contraction.

The cut in front of the n is a claim about English, that the negation is a morpheme spelt n’t. The cut at the apostrophe is a claim about nothing at all; it is one rule applied wherever an apostrophe stands between two letters, with no list and no exception. Both give back the sentence when the pieces are joined, so reversibility does not choose between them. What chose was that the second needs nothing known about English to state, and the same script had to run over eleven languages of proceedings while the first would have needed a linguist for each of them.

It is worth being exact about how little there is here. The whole method is one pass over each run of characters that has no spacing in it, followed by a second pass that settles the final stop, and the questions asked are these.

There is no dictionary of English, no frequency count and nothing learned from a corpus. Two switches sit beside the clauses and are off unless a caller asks, one that cuts a compound at its hyphen and leaves a marker where the hyphen was, and one that rewrites a handful of characters a file format would otherwise swallow. Both are in Part 4, because both are rewritings and both are undoable, which is the argument of that part.

A full stop marks the end of a sentence and it marks an abbreviation, and it is the same character both times. In our running sentence both jobs are being done, once after Dr and once at the end, and a rule that cuts every stop off gets the second right and the first wrong while a rule that cuts none off does the reverse.

The annotation rules settle this with a fact about their pipeline rather than about English. They ran on one sentence at a time, so the last stop in the text is the sentence’s and every other stop belongs to whatever it is touching. That is exact when the text really is one sentence and it is not a claim about words at all.

These rules could not use it. Proceedings arrive as paragraphs, the sentence boundaries were another problem entirely, and the same script had to run over eleven languages of them. So the clause here asks about the word instead, and asking about the word is where a list becomes unavoidable, because Dr. and away. are the same shape. Nothing in the characters separates a title from an ordinary word with a stop after it.

A word ending in exactly one stop keeps that stop when any of four things is true, and the list is only the second of them. Read them as four separate attempts to recognise an abbreviation, of which three cost nothing to state.

the part before the stop holds another stop and a letter
        U.S.   e.g.   i.e.
the part before the stop is on the named list
        Dr.    Prof.   Ltd.   Nov.   J.
the part before the stop is on the figures list and a figure follows
        No. 5   Art. 3
the next word begins with a lowercase letter
        et al. ran   approx. forty

The first clause catches every initialism at once and needs no vocabulary, since a word with an interior stop and a letter in it is not an ordinary English word. The fourth is the interesting one, and it is a piece of reasoning about writing rather than about language, that an English sentence begins with a capital, so a lowercase word after a stop means the stop did not end a sentence. It costs one character of lookahead and it rescues abbreviations that nobody ever listed.

The third clause exists because No. is an abbreviation in No. 5 and an ordinary word in No. Nothing arrived. Those two are the same five characters and the difference is entirely in what comes next, so the entry carries a condition rather than an unconditional promise.

The way to see those clauses working is to give the rules the same abbreviation twice with different writing after it. Ten stops are below, every one of them somewhere in the middle of its text rather than at the end, so every one of them has an answer that can be stated before any rule is run.

…

Read the three pairs. No. keeps its stop before a figure and loses it before a sentence, which is the third clause doing exactly what it was written for. approx. is on no list at all and keeps its stop before forty and loses it before 40, which is the fourth clause rescuing a word nobody thought of and then failing on the same word for a reason that has nothing to do with the word. And et al. survives before a lowercase verb and is cut in front of a capital.

Those ten stops can be scored, since each of them has a right answer. Eight belong to the word in front of them and two end a sentence. These rules place seven of the ten where they belong, the annotation rules place eight, and a written-out table of exceptions places four.

The middle number is the one to be careful with. The annotation rules answered kept on all ten, because none of the ten stops is the last character of its text and their clause keeps every stop that is not. So their eight is exactly the number of cases whose answer happens to be kept, reached without distinguishing anything, and a set of ten with the proportions reversed would have given them two. These rules answered kept five times and split five times, so their seven is a score a differently balanced set would not move nearly so far.

All three of the misses above are the same failure. Gov. is a title and is not on the list, and the word after it is a surname beginning with a capital, so no clause saves it and the rules answer Gov and a stop. The annotation rules keep it whole, and they are right by accident, since they would keep small. whole in the same position.

Gov. Reyes asked for the figures.
        ->  Gov  .  Reyes  asked  for  the  figures  .

Dr. Alvarez signed the re-analysis.
        ->  Dr.  Alvarez  signed  the  re-analysis  .

Two titles, written identically, one on the list and one not, and the difference in the output is a spurious piece and a spurious row. That is the exact failure the whole method was built to prevent, since Gov now sits in the table beside every other truncated word and the stop it lost sits beside every sentence-final stop.

A list that carries a method and a list nobody hits are different objects, so it is worth counting. The English list here holds 55 entries, of which 27 are named words, 26 are the single capitals that catch an initial in a name, and 2 keep their stop only when a figure follows.

Counting over the six-sentence notebook would say nothing, since it holds one abbreviation, so the corpus below is eighteen sentences of minuted proceedings written for this page, with titles, months, article numbers and company suffixes in them. It is the register the rules were made for and it is deliberately generous to them.

…

The question the list answers was asked 39 times over those eighteen sentences, which is how many runs of characters ended in exactly one stop, and 19 entries answered it. The other 36 were never consulted at all, so a third of the list is doing every piece of the work it does here and the remaining two-thirds is there for writing this corpus does not contain. Five real abbreviations turn up that no entry covers, which is five more than the unused entries would suggest.

The table in the middle of the panel is the finding I did not expect, and it corrects what section 6 implies. Twenty stops were kept over these eighteen sentences, 18 of them by the named list and 2 by the figures list, and the two clauses that read characters kept none at all. The interior-stop clause needs a word like U.S., which these sentences happen not to contain, and the lowercase clause needs an abbreviation followed by an ordinary lowercase word, which in eighteen separate sentences almost never happens because the abbreviations here are titles and the next word is a surname.

Step back from the full stop for a moment, because the clause that reads a list has a shape that turns up all over this section of the site and is worth naming once. A rule states what to do everywhere, a table names the places where the rule must not be followed, and the table is consulted first.

for each run of characters with no spacing in it
        if the run is named in the table
                do what the table says and stop
        otherwise
                apply the rules

The reason it is not a longer rule is that the exceptions are not describable as characters. A rule that ends a word before n’t produces ca from can’t, which is wanted, and would produce wo from won’t and sha from shan’t, and any attempt to fix one of those inside the pattern breaks the others. As a table it is one line each, checked before the pattern runs, and the pattern is untouched.

The same argument runs the other way for an abbreviation. Mr. is a capital, two lowercase letters and a stop, which is precisely the shape of Ran., so no reading of the characters keeps one and cuts the other. A named list is what is left over when the writing genuinely does not carry the distinction, and reaching for one is a report that the characters have been examined and found silent rather than a shortcut past a rule somebody could have written.

Set beside that, the apostrophe clause here is the interesting case, because it is the place where these rules have no list and the annotation rules do. Cutting at the apostrophe is one statement, applied to every apostrophe standing between two letters, and it covers didn’t, it’s, they’re and y’all alike.

didn't   ->  didn   't
can't    ->  can    't
won't    ->  won    't
1990's   ->  1990   's
students'->  students   '
l'analyse->  l   'analyse   under the English setting
l'analyse->  l'   analyse   under the French setting

Nothing there is listed and nothing is examined and left alone. The price is that the halves are not morphemes, so didn is not a word and the negation is spelt ’t here where a linguist writes n’t. The gain is that the same clause was written once and turned round for French and Italian by moving the cut to the other side of the mark, which is a setting rather than a second table.

A table of this shape is under one constraint that is easy to miss and is the reason the shape survives here at all. The pieces an entry lists must join back to exactly the word the entry names.

…

Without that constraint the obvious entries are tempting. can’t would go to can and not, which reads better than ca and n’t and gives the aligner two real words. What it costs is that a piece stops being a stretch of the source, so the sentence can no longer be reassembled from the pieces and their positions, and it costs it for whichever words happen to be listed rather than uniformly, which is worse than costing it everywhere.

Normalising a word to a different word is a real job and it is done elsewhere in a pipeline, before or after this step, where it can be recorded. The one place these rules do change a piece’s text is the subject of the next part, and they do it in a way that can be read backwards.

The list of abbreviations is English. A French text handed to these rules gets the English abbreviations checked against it, and the setting that turns the apostrophe cut around does not turn the list around, because there is nothing to turn it to unless somebody has written the French file.

That is the honest description of the method rather than a complaint about it. The original shipped one file per language and the files were contributed by whoever needed that language, so a language is supported exactly when somebody has sat down and listed its abbreviations. On this page every measurement is English, and the French and Italian settings in the playground change only where the apostrophe cut falls.

English setting   Dr.  Alvarez  didn  't  expect  the  low-cost  re-analysis  .
French setting    Dr.  Alvarez  didn'  t  expect  the  low-cost  re-analysis  .

Here is the constraint stated plainly. A parser reading a treebank never has to produce the original paragraph, so the annotation rules were free to write things into a piece that were not in the writing. A translation system produces a sentence for somebody to read, so whatever it does on the way in has to be undone on the way out.

The test is simple and it is worth stating before the numbers. Take the pieces, throw away the spans, join them in order with nothing between them, and ask whether what comes out is everything in the source that is not spacing.

…

On the quoted sentence both rule lists give fourteen pieces and they part completely on that test. These rules rewrite nothing and the joined pieces are the sentence. The annotation rules rewrite two pieces, an opening quotation mark into two backquotes and a closing one into two apostrophes, and the joined pieces are not the sentence and cannot be made into it. Over the eighteen minuted sentences the same test passes eighteen times here and seventeen times there.

The sharpest way to see why that rewriting cannot be undone is to write a text in the spelling it produces. Take a sentence with quotation marks in it and a second sentence typed with two backquotes and two apostrophes, which is how a typist of the period produced directional quotes on a machine that had none.

she said "wait" and left
she said ``wait'' and left

Under the annotation rules those two come back as the identical seven pieces. Nothing in the output distinguishes a sentence whose quotation marks were rewritten from a sentence that was typed that way, so anything wanting the original has to keep the original, and the pieces alone are not a division of a text any more. Under these rules the first gives seven pieces and the second gives nine, because a backquote is not a quotation mark and each stands alone, and the two texts are told apart.

There are two things these rules will change about a piece’s text, and both are off unless asked for. The first replaces eight characters with names, because a phrase table is a text file whose fields are separated by vertical bars and whose decoder reads angle brackets as markup, so a vertical bar in a training sentence would have been read as a field boundary and corrupted its row.

&  ->  &        |  ->  |
<  ->  &lt;         >  ->  &gt;
'  ->  &apos;       "  ->  &quot;
[  ->  &#91;        ]  ->  &#93;

Read the mapping in the other direction and every arrow still points at exactly one thing, which is the whole difference from the quotation-mark rewriting of the previous section. Nothing else produces ", so the script that puts the text back together knows what it was reading. With the switch on, our quoted sentence has three pieces whose text is no longer the writing, the two quotation marks and the apostrophe in the contraction, and putting the mapping back recovers the sentence exactly.

The second rewriting is the compound. Cutting low-cost into three lets low and cost share their statistics with every other use of those words, and the piece left in the middle is spelt @-@ rather than - so that the same script can tell a hyphen that joined a compound from a dash that stood between two words. That marker is one character wide in the source and three characters long as a piece, and it undoes to the hyphen it stands for.

Dr.  Alvarez  didn  't  expect  the
low  @-@  cost  re  @-@  analysis  .

It is easy to read all this as saying the text comes back, and it does not, quite. Joining the pieces in order recovers every character that is not spacing, and it recovers none of the spacing. Joining them with one space each gives the source for all eighteen minuted sentences when the text was only cut at the spaces, and for two of the eighteen under these rules.

The two that do come back are the sentences that happen to end in an abbreviation, so that every piece was already separated by a space in the writing. The other sixteen have a comma or a final stop standing alone, and a space in front of a comma is not English. Restoring that is the job of the second script that shipped alongside, and it is a set of conventions about a language rather than anything the pieces carry.

So the promise is narrower and more useful than it sounds. The pieces plus their spans plus the source recover everything exactly, and the pieces alone recover every character in order with the spacing to be decided. What the annotation rules give up is the second of those, and it is the one a system with no access to the source has.

The bill is length. Anything reading the text afterwards pays per piece, and a model that compares every position with every other position pays roughly with the square of the count, so a rule that separates marks is spending on every pass over the corpus for the rest of the pipeline’s life.

…

On the eighteen minuted sentences these rules give 228 pieces where splitting on spaces gives 192, which is 18.75 per cent more from identical writing, and the written-out table gives 243 because it also cuts every compound at its hyphen.

228 − 192   =  36 more pieces
36 / 192    =  18.75 per cent

The 36 are not spread evenly over the writing. Twenty of them are full stops standing alone, eight are commas and four are quotation marks, which is the 32 pieces holding no letter and no digit, and the rest are the second halves of the contractions and possessives. On the six-sentence notebook these rules give 72, splitting on spaces gives 63, the annotation rules give 72 as well and the table gives 84.

Every rule in that table keeps the same number of characters, 931 on the minutes and 318 on the notebook, which is everything in the writing that is not spacing. That is worth noticing because it is not true of the boundary rules on the earlier page, which find the marks correctly and then discard them. What separates the four here is where the cuts fall.

The motive was to make the surface vocabulary smaller, so it is fair to ask whether it did, and the measurement here is the one that surprised me. Over the eighteen minuted sentences, splitting on spaces produces 137 different pieces and these rules produce 136. Over the six-sentence notebook, splitting on spaces produces 47 and these rules produce 49, so at that size the method makes the vocabulary larger.

Nothing is wrong with either number. The mechanism is real and it is visible in the panel above as the words the crude rule spells more than one way, five of them on the minutes and one on the notebook, each an extra row that separating the mark collapses into one. What is missing at this size is repetition. A saving of five rows is paid for by the rows the marks and the contraction halves open, and the two come out level.

entries both rules hold                               106

only at spaces
  a marked run whose word has no other row            24    small.   Gov.
  a marked run whose word already has a row            5    re-analysis.   Okafor,
  a contraction                                        2    didn't   It's
  at spaces               106 + 24 + 5 + 2   =       137

only under these rules
  the words those 24 runs become                      24    small   Gov
  marks standing alone                                 3    .   ,   "
  pieces of a contraction or possessive                3    didn   't   's
  under these rules       106 + 24 + 3 + 3   =       136

On the minuted corpus the two rule lists produce 228 and 227 pieces and disagree on fourteen entries each way, which the panel above lists in full. Ten of those fourteen are one word with and without a full stop, and they are informative because they do not all run in the same direction.

Five of the ten go to the annotation rules and one goes to these, and four have no right answer at all and are the subject of the next part. That is a real defeat on a real corpus, and it comes about because the minutes are eighteen separate sentences, which is exactly the arrangement the positional rule was designed for. Run the same rules over the paragraph those eighteen sentences would form and the positional rule loses every internal stop at once, which the previous page measures.

Pointed at writing that is not prose, these rules come apart in the way every rule on these pages does, by separating a character that really is punctuation in a sentence and really is not one here. A company name written with an ampersand becomes three pieces, a percentage becomes two, and an address is cut at the character that makes it an address.

The last row is worth dwelling on, because the reason for it is arbitrary rather than principled. A hyphen is one of the characters these rules leave inside a word, so low-cost survives, and an underscore is not on that list, so the file name is cut in two at a character doing exactly the same job. Nothing about writing decides that. It is which characters somebody thought of when the clause was written, and the annotation rules happen to have thought of neither and so keep the whole thing.

Everything on this page has been a rule except one clause, and that clause decides whether Dr. is one piece or two. Take the list away and the clause has no content, because the three clauses beside it cannot recognise an abbreviation that has no interior stop and is followed by a capital, and that covers most titles at the head of a sentence.

So the method is a rule and a table, and what the method does to a text is not determined until somebody has written the table down. That has three consequences worth separating. The answer for a given text depends on which file was loaded, so two callers running the same rules on the same sentence can get different pieces without either being unfaithful.

The table is per language, so a language nobody has written a file for is not badly handled, it is outside what the method says. And a table is finite while the abbreviations of a language are open, so there is always a next word, and no amount of adding entries changes that.

The practical consequence is about reporting rather than about quality. Two numbers measured on the same corpus by two people who loaded different files are not comparable, and nothing in the output says so, since a word that lost its stop looks the same whether the file was short or the word was genuinely not an abbreviation. Every count on this page was taken with one file of 55 entries in front of the rules, and a file with Gov and Sen in it would move the score in section 8 and the piece counts in section 19 at the same time.

There is one case where no list helps and no rule helps, and it is worth stating as a fact about English writing rather than as a shortcoming of anything. A sentence that ends in an abbreviation is written with one full stop, and that stop is closing the abbreviation and closing the sentence at the same time.

The re-analysis was signed by Okafor Co.

        Co.  is what the abbreviation needs
        .    is what the sentence needs
        Co.  and  .   would invent a character

A division of a text into pieces gives every character to exactly one piece, so the two requirements cannot both be met. Knowing that Co. is an abbreviation does not help, because the question is not what the word is, and the position of the stop does not help, because both readings put it there. These rules answer Co. and lose the sentence’s stop; the annotation rules answer Co and a stop and lose the abbreviation’s. Neither is wrong, and a third method would have to be wrong in one of those two ways as well.

What is genuinely open here is which of the two losses a pipeline would rather have, and that is decided by what reads the pieces. A model counting sentences wants the stop; a table keyed on surface forms wants the abbreviation whole, since the alternative puts a truncated title in a row of its own. Four of the eighteen minuted sentences end this way, which is often enough that the choice has to be made rather than deferred.

The other place the method stops is not a case at all, it is a boundary around the whole method. Requiring that the pieces join back to the writing rules out a large class of things a translation pipeline actually wanted, and those things do not become impossible, they become somebody else’s step.

Lowercasing is the plainest example. A phrase table with The and the as separate rows is wasting evidence in exactly the way this method exists to prevent, and folding the case is a rewriting that cannot be read backwards, since the does not say whether it began a sentence. Expanding a contraction, normalising a spelling, mapping a number to a placeholder and stripping an accent are the same shape.

So a method under this constraint splits a pipeline in two. One half is a division of the writing, which is what these rules are, and the other half is a sequence of changes that a caller must record if it wants them back. The two changes this method does allow itself are exactly the two that arrive with an inverse written down beside them, a table of eight characters and one marker standing for a hyphen, and what admitted them was that the mapping can be read in either direction rather than that they were useful.

The cases below are gathered from the whole page. The right column says what the method does not determine and why, and it quotes a number where the number is a fact about the method rather than about any particular text.