Section 10 of 12 · 4 lessons

Large Language Models

A language model receives some text and produces more text. To understand how, we need to look at what it was trained to predict and how earlier words influence that prediction. We will follow those steps from a single prediction to a complete response.

0 of 4 finished

Building a Language Model

We have ways to represent tokens and let them use context. Now we need to put those components together and define a prediction task that provides the model with something to learn.

  1. 01Transformer BlocksAttention lets tokens exchange information, but that is only part of a transformer. We will put it together with per-token processing, normalisation, and residual connections, explaining the job of each component.
  2. 02Next-Token PredictionText already contains examples of what follows what, giving us a training task without asking someone to label every sentence. A next-token model learns to predict each continuation using only the text available before it.

Generating Text

Predicting the next token does not produce a whole answer at once. We need to choose a token, add it to the text, and make another prediction using the updated context.

  1. 03Autoregressive GenerationA next-token model predicts one step ahead. To produce a response, we choose a token, append it to the existing text, and ask the model to predict again with that new context.
  2. 04Sampling and TemperatureThe highest-scoring token is not the only token a model could produce. Sampling and temperature control how we choose from its predictions, changing the output without changing the learned weights.