Section 9 of 12 · 37 lessons

Natural Language Processing

The words we read are comprehensible to us, but a numerical model needs some way to process them. We will start by turning text into pieces with numerical IDs. Then we can ask how a model might learn relationships between those pieces and use the surrounding words to interpret them.

0 of 37 finished

Tokens and Basic Units

Before a model can work with text, we need to decide what pieces it will receive. A word, a character, and a byte each give us a different starting point.

  1. 01What a Token IsWe can read the word cat, but a numerical model needs a representation it can process. A tokenizer divides text into pieces and maps those pieces to IDs, giving us a starting point for working with language.
  2. 02Bytes and CharactersA visible character and the bytes used to store it are not always the same unit. That distinction affects how much text a model receives and whether its representation can handle unfamiliar writing.
  3. 03Splitting on SpacesSpaces give us an obvious first place to divide a sentence. We will try that rule, then look at what it does with punctuation, repeated whitespace, and writing that does not separate words with spaces.

Rules for Word Boundaries

Splitting text on spaces leaves us with questions about punctuation, contractions, and different writing systems. These tokenizers make explicit choices about where one piece ends and another begins.

  1. 04Unicode Word BoundariesRules written around English letters and spaces do not cover every writing system. Unicode character information gives us a broader basis for deciding where boundaries should fall, though it does not settle every linguistic question.
  2. 05Penn Treebank RulesShould can't remain one piece, or should the negation be separated? Penn Treebank tokenization gives English contractions and punctuation consistent treatment so annotated text can be processed using the same conventions.
  3. 06Moses RulesA translation system needs a consistent way to handle punctuation without breaking abbreviations and other language-specific forms. Moses tokenization combines general splitting rules with exceptions for those cases.
  4. 07The Pattern Language Models UseBefore learning smaller pieces, a tokenizer may need to separate runs of letters, numbers, and punctuation. A regular expression can describe those initial groups while retaining information about the spaces around them.

Scripts Without Spaces

Not every writing system places spaces between words. We need another source of information about the boundaries, whether that comes from a dictionary, a probability model, or labelled examples.

  1. 08Maximum MatchingWhen spaces do not tell us where words end, a dictionary gives us one possible guide. Maximum matching takes the longest word it recognises at each position, leaving us to examine when that local choice fails.
  2. 09The Word LatticeChoosing a word early can change which words remain available later in a sentence. A word lattice keeps the alternatives so we can compare complete segmentations before committing to one.
  3. 10Segmenting With a Hidden ModelWe can see the characters in a sentence without knowing which ones begin or end a word. A hidden-state model uses learned probabilities to infer a sequence of labels for those positions.
  4. 11Learning Boundaries From ExamplesIf people have already marked word boundaries in example text, we can use those labels to teach a model. The surrounding characters provide clues it can use when deciding where to split new text.

Predicting Token Sequences

Some words are more likely after a particular phrase than others. Counting short sequences gives us a simple way to make a next-token prediction and a reason to consider phrases we have never seen.

  1. 12N-GramsThe words immediately before a position give us clues about what might come next. An n-gram model uses counts of short sequences to make that prediction, with adjustments for sequences missing from its training data.

Learning a Subword Vocabulary

A vocabulary containing every possible word would be enormous, and it would still miss new words. Reusing smaller pieces helps, but we need a way to decide which pieces deserve an entry.

  1. 13Byte Pair EncodingCommon letter combinations need not be stored one character at a time. Byte pair encoding repeatedly joins frequent neighbouring pieces, building a vocabulary that can reuse parts across different words.
  2. 14WordPieceA tokenizer needs reusable word parts and a consistent way to find them in new text. We will look at the pair-scoring rule used here to build a WordPiece vocabulary, then its longest-match encoding rule.
  3. 15The Unigram Language ModelA word can have several valid divisions into smaller pieces. A unigram tokenizer assigns probabilities to those pieces so it can compare complete segmentations while learning which vocabulary entries to keep.
  4. 16SentencePieceRequiring text to be split into words first makes tokenization depend on another set of boundary rules. SentencePiece can learn pieces from a text stream that explicitly represents whitespace, reducing that dependence on predefined words.
  5. 17Greedy CoverageThere is only so much room in a vocabulary. Instead of choosing the most frequent pair to merge, we can ask which new piece covers the most text that our earlier choices have not already covered.
  6. 18Moving a VocabularyA vocabulary learned under one tokenization scheme may need to be used under another. We will follow what happens to its pieces, including boundaries that disappear and entries that become indistinguishable after the move.

Word Structure

Words such as walk, walked, and walking share more than a few letters. We will look at how recurring word parts can be learned from examples or described with explicit rules.

  1. 19MorfessorWords often share stems and endings, such as walk in walked and walking. Morfessor looks for reusable parts by balancing the cost of storing a vocabulary against the cost of describing the observed words.
  2. 20Finite-State MorphologyA word's spelling can reflect a stem, an ending, and a rule that changes how they join. Finite-state morphology describes those relationships explicitly so we can analyse forms and generate them from their components.

Fixed Tables and Shorter Sequences

A large vocabulary takes up space, while a long sequence takes work to process. Hashing and byte patches make different compromises when those costs become a problem.

  1. 21Hashing CharactersA separate entry for every possible character can make a table large. Hashing maps characters into a fixed number of buckets, at the cost of sometimes giving different characters the same destination.
  2. 22Patching Without a VocabularyBytes let us represent text without a learned word vocabulary, but they can produce long sequences. Grouping nearby bytes into patches gives later parts of a model fewer positions to process.

Codes for Numerical Inputs

The idea of assigning a piece an ID can also be applied to numerical data. We will replace continuous values with a limited set of codes and examine what information is lost.

  1. 23Codebook QuantisationWe may need to replace a continuous vector with a compact ID. A codebook provides a fixed set of representative vectors, letting us choose a nearby entry and measure how much the replacement changes the input.
  2. 24Finite Scalar QuantisationInstead of choosing a whole vector from a learned codebook, we can give each coordinate a few allowed levels. Combining those choices produces a discrete code with a different way of controlling the available representations.

Word Vectors from Counts

A token ID identifies a word without telling us how it relates to other words. The contexts in which words occur give us information we can use to build more useful numerical representations.

  1. 25A Vector for a WordA token's ID tells us which word it identifies, but not how that word relates to another. A vector gives us several numerical coordinates whose learned relationships can support useful comparisons.
  2. 26Distance and SimilarityOnce words have vectors, we still need to decide what makes two vectors similar. Their distance, direction, and length can give us different answers, depending on the comparison we choose.
  3. 27Latent Semantic AnalysisDocuments about related subjects may use some of the same words in different amounts. Latent semantic analysis starts with a table of word use and keeps a smaller set of shared patterns for representing words and documents.
  4. 28Pointwise Mutual InformationTwo common words can appear together often simply because both appear everywhere. Pointwise mutual information compares their observed association with what we would expect if their occurrences were independent.
  5. 29Random IndexingCounting every possible word-context pair can require a very large table. Random indexing gives contexts small random vectors and accumulates them, keeping the representation a fixed size as new examples arrive.

Word Vectors from Prediction

If a representation helps a model predict the words around it, it must preserve something useful about how that word is used. These methods learn word vectors from that requirement and related observations.

  1. 30The Embedding LayerA token ID needs to lead to the vector associated with that token. An embedding layer provides the lookup table, and training adjusts its rows so the retrieved vectors become useful for the task.
  2. 31Word2vecWords used in similar contexts often have something in common. Word2vec uses a word-prediction task to learn vectors that capture useful regularities in those surrounding words.
  3. 32FastTextA rare word may share useful parts with words we have seen many times. FastText includes character fragments in its word representations, allowing those related spellings to share information during learning.
  4. 33GloVeA table of word co-occurrences contains information about how words are used together. GloVe learns vectors whose comparisons account for patterns in those counts, rather than keeping the full table as the representation.

Representing Order and Context

The same word can play different roles depending on where it appears and what surrounds it. A model needs access to that information as well as the word itself.

  1. 34Positional EncodingThe same words in a different order can mean something different. A model that receives token vectors needs a way to know where those tokens occur, which positional information provides.
  2. 35AttentionA word's meaning in a sentence can depend on words some distance away. Attention gives each position a way to compare the available information and combine the parts most relevant to its current representation.

Representing a Whole Text

Searching or comparing documents often calls for one representation of each text. We can combine its word vectors or learn a document vector, with different consequences for what survives.

  1. 36Pooling a TextComparing whole sentences often requires one vector per sentence. Pooling combines their token vectors into a fixed-size representation, but the choice of combination determines which details we can still recover.
  2. 37Paragraph VectorsA document may have useful characteristics beyond an average of its word vectors. Paragraph vectors learn a document representation by asking it to help predict the words in that document.