Next-Token Prediction

A model learns from pairs, an input and the answer it should have given. For most tasks somebody has to write the answers. Text is unusual because it already contains them. Cover the last token of any stretch of text, and the covered token is the answer to the question of what comes next.

Before anything can be calculated, the text has to become numbers. This page’s vocabulary holds seven entries, <start>, the, cat, dog, sat, slept and <end>, numbered 0 to 6 in that order. The word the is token 1, cat is token 2, sat is token 4 and <end> is token 6.

A token ID identifies an entry in the vocabulary. An embedding gives that ID a vector; a transformer processes the vectors using available context. A final projection produces one logit, or unnormalized score, per vocabulary entry.

The training target at each position is the following token in the original text. The example uses explicit start and end tokens to show the boundaries. Real training pipelines also need policies for document boundaries, sequence packing, padding, and truncation.

Original sequence:  <start>  the  cat  sat  <end>
Input positions:    <start>  the  cat  sat
Target positions:   the      cat  sat  <end>

Read the two lower rows as columns and each column is one training example. At a given position the model may read every input token up to and including that column, and its target is the token written directly below.

Two things about this table matter for the rest of the lesson. A target never appears in its own input, so the model cannot answer by copying. And dog and slept are in the vocabulary without being in the sentence. The model still has to score them at every position, which is why each prediction is a row of seven scores and not a single yes or no about the target.

The target is the continuation that appeared in this example. Another example can have the same prefix and a different next token. Across data, the model is encouraged to assign probability according to the continuations it observes.

Suppose the data holds the cat sat in one example and the cat slept in another. After the cat, the first example says the answer was sat and the second says it was slept. A prediction that puts everything on sat is right about the first example and badly wrong about the second. A prediction that shares between sat and slept is partly right about both. Step 4 puts numbers on the two predictions, and the sharing one comes out well ahead. That is the sense in which the model is fitting the proportions in its data, and it is why the output is a distribution over the vocabulary and not one chosen token.

This is called self-supervised learning because the text supplies the target without a person labeling each position. It still uses a precise training target and a loss. Data selection, duplication, and coverage influence what the model learns.

The table in step 1 shows how little the supervision asks of anybody. It was produced by shifting the sentence one place. Nobody decided what the right continuation ought to be, which also means that whatever the text contains, mistakes and repetitions included, becomes a target.

Softmax exponentiates each logit and divides by the sum across the vocabulary. For the default table, the observed target has an assigned logit of one and all six alternatives have a logit of zero.

Target weight = exp(1)
Each alternative weight = exp(0) = 1

Total weight = exp(1) + 6
Target probability = exp(1) / (exp(1) + 6)

Token loss = −ln(Target probability)

Target probability = 2.7183 / (2.7183 + 6) ≈ 0.3118
Each alternative = 1 / 8.7183 ≈ 0.1147
Token loss = −ln(0.3118) ≈ 1.1654

The negative natural logarithm gives a small loss when the target is likely and a large loss when it is unlikely. This does not evaluate whether the sentence is true. It measures how well the distribution predicts the observed text.

Let p be a vocabulary probability, y the target ID, and N the number of scored positions. The average loss gives each scored position equal weight. The derivative below applies to a logit at one scored position.

L = −(1 / N) Σₜ ln pₜ(yₜ)

∂L / ∂logitₜ,ⱼ = (pₜ(j) − 1[j = yₜ]) / N

For the target, the derivative is negative unless its probability is already one. Gradient descent therefore increases that score locally. In a real network, backpropagation carries this derivative through the vocabulary projection, transformer blocks, and embeddings. Shared parameters couple the effects across examples.

Target logit:      (0.3118 − 1) / 4 ≈ −0.1721
Each alternative:  (0.1147 − 0) / 4 ≈ 0.0287

Sum over the row:  −0.1721 + (6 × 0.0287) ≈ 0

A large language model has a large learned parameter set and is trained on substantial language data. There is no universal parameter count at which a model becomes large. The next-token objective stays the same while model capacity, context length, data, and optimization become major design choices.

What stays the same is everything Part 2 calculated. Each position still gets one score per vocabulary entry, softmax still turns the scores into shares, and the loss is still the negative logarithm of the share given to the token that followed. What changes is where the scores come from. In the table on this page they were assigned, one number per cell, and each could be moved without disturbing the others. In a model they are computed from parameters, and the same parameters produce the scores at every position of every example.

Pretraining supplies broad statistical patterns. It does not automatically turn a continuation model into a reliable assistant. Supervised fine-tuning can teach response behavior; retrieval can provide source material at answer time. Neither makes evaluation unnecessary.

The SDK example keeps the scores explicit. It calculates the mean loss, individual token losses, probability rows, and exact logit derivatives without fitting a language model.

import numpy as np
from oop_ml.numpy.modern import TokenCrossEntropy

logits = np.array([[1.0, 0.0, 0.0]])
targets = np.array([0])
result = TokenCrossEntropy().measure(logits, targets)
print(result.value)
print(result.gradient)

During training, earlier input tokens come from the training sequence, so causal positions can be processed together. During generation, the model must use its own selected tokens as the growing context. That difference is the subject of the next lesson.

The first of those claims can be checked on the small decoder. One pass over the four input tokens returns four rows of scores. Running the decoder again on only the first three tokens gives a last row that agrees with the third row of the full pass, and the same holds for the other prefix lengths, to within rounding in the sixteenth decimal place. A position’s scores depend only on the tokens at or before it. So one pass over a training sequence does the work of four separate predictions, and the loss at every position can be calculated from that single pass.