Supervised Fine-Tuning

A pretrained language model has learned one thing, which is to continue text. Nothing in that training says a question should be followed by its answer, or that an answer should stop when it is complete. A question followed by another question is also a plausible continuation. What the model is missing is a demonstration of the continuation we want.

Supervised fine-tuning continues training from existing parameters using examples selected for a task or behavior. A prompt-response pair might demonstrate answering a question, extracting fields, summarizing a passage, or following a required output format.

The target response is labeled data: it specifies what the model should be encouraged to produce. More examples are not automatically better if their responses conflict, contain errors, or fail to represent the intended use.

This page follows one pair all the way through. The prompt is a start token and two words, and the response is one word and an end token.

Prompt:    <start>  the  cat
Response:  sat  <end>

Small as it is, the pair demonstrates two behaviors. The first is which word should follow the prompt. The second is that the response then stops. The end token is part of the target, so a model that produces sat and carries on has not reproduced the demonstrated response.

That is why the behavior has to be defined before training and not discovered afterward. Training pulls the model toward whatever the responses contain, and it cannot tell an intended pattern from an accidental one. If some responses end with the end token and others trail off, both are being taught. If two responses answer the same kind of prompt in different formats, the model is pulled toward each in turn. Deciding what a good response looks like, including its format and where it stops, is the part of fine-tuning that happens before any gradient is computed. The collection of pairs is where that decision is written down.

The prompt needs to remain available because it tells the model what response is appropriate. The causal attention mask prevents a response position from reading future response tokens. A separate loss mask decides which next-token predictions contribute to the training objective.

Response-only loss excludes prompt-token predictions while including the assistant response targets. Some training setups score additional positions; this is an explicit objective choice rather than a universal requirement. The checkbox exposes that choice.

The two masks are easy to confuse, because both are rows of zeros and ones laid over the same sequence. They answer different questions.

Why leave the prompt predictions ungraded? In use, the prompt is written by someone else and handed to the model. The model is never asked to produce it. Grading the predictions of the two prompt words, the and cat, would spend part of every update on teaching the model to guess the wording of prompts, which is not the behavior being defined. The response predictions are the ones the model will have to make by itself, so those are the ones this objective grades.

The prompt still does its work either way. The prediction of sat is made by a position that has read the whole prompt, and the prediction of the end token is made by a position that has read the prompt and sat. Removing the prompt would leave those predictions with nothing to respond to.

The toy prompt ends at cat, and the target response is sat followed by an end token. Remember that a next-token target is shifted one position beyond its input. The output at the cat position predicts the first response token.

Input positions:   <start>  the  cat  sat
Target positions:  the      cat  sat  <end>
Loss mask:         0        0    1    1

A prompt token can therefore appear at an input position whose output is scored. Masking every output merely because its input token belongs to the prompt would wrongly exclude the first response prediction.

Laying the four predictions out one per row makes the alignment easier to see. Each row is one position. It reads the prefix on the left and is asked for the target beside it.

The row that predicts sat is the one that goes wrong. Its input token, cat, is a prompt token, and its target, sat, is a response token. A mask built from the input row would mark that position as prompt and switch it off.

Mask by input token (wrong):   0  0  0  1
Mask by target token (right):  0  0  1  1

The wrong mask keeps only the prediction of the end token. The prediction it throws away is the decision about how the response begins, which is the first thing a fine-tuned model has to get right.

Let m indicate whether a target is scored, and p give the probability assigned to that target. Divide by the number of included positions, not the full sequence length.

L = −Σₜ [mₜ log pₜ(yₜ)] / Σₜ mₜ

For this response-only mask:
L = [−log p(sat | <start> the cat)
     −log p(<end> | <start> the cat sat)] / 2

The live example fills in the numbers. Its vocabulary has seven tokens. At every position it assigns the target a logit of 1 and each of the other six tokens a logit of zero, which is where its slider starts. Softmax turns those seven logits into probabilities, and the token loss is the negative logarithm of the target’s share.

p(target) = e¹ / (e¹ + 6) ≈ 0.312
Token loss = −log(0.312) ≈ 1.165

Response-only mean = (1.165 + 1.165) / 2 ≈ 1.165
All-targets mean   = (1.165 + 1.165 + 1.165 + 1.165) / 4 ≈ 1.165

The two means agree, and that is a property of this example and not of the mask. Every target was assigned the same score, so every token loss is the same and any mean of them is 1.165. The denominator still matters. Dividing the two included losses by the sequence length of four would report a different number.

(1.165 + 1.165) / 4 ≈ 0.583

That is half the true mean, and it would shrink again with every token added to the prompt, although the response had not changed. Dividing by the count of included targets keeps the loss a statement about the response alone.

Where the mask does show is in the derivatives, which are what training acts on. The derivative of the mean loss with respect to a scored target’s own logit is its probability minus one, divided by the number of included targets. For an excluded target it is zero.

Response-only:  (0.312 − 1) / 2 ≈ −0.344
All targets:    (0.312 − 1) / 4 ≈ −0.172

A negative derivative says that raising the logit lowers the loss, so descent raises it. Under response-only scoring the two response targets are each pulled twice as hard as under all-targets scoring, because the same mean is now shared between two positions and not four. The two prompt targets are not pulled at all. The whole update goes toward the response.

The derivative with respect to an excluded position’s output logits is zero. This does not mean prompt embeddings or earlier hidden states receive no gradient: scored response positions can depend on them through attention and other shared computation.

The example uses assigned logits and calculates the objective only. Updating a real model requires backpropagating this loss to the selected parameters, followed by optimizer steps.

The SDK accepts one boolean per target row and rejects an all-excluded batch because its mean would have no denominator.

import numpy as np
from oop_ml.numpy.modern import TokenCrossEntropy

logits = np.zeros((4, 7))
targets = np.array([1, 2, 4, 6])
scored = np.array([False, False, True, True])
result = TokenCrossEntropy().measure(logits, targets, scored)
print(result.value)
print(result.gradient)

All 28 logits in that snippet are zero, so every position spreads its probability evenly over the seven tokens. Both things it prints can be worked out before running it.

p(target) = 1 / 7 ≈ 0.143
Loss = −log(1 / 7) = log 7 ≈ 1.946

Derivative at a scored target   = ((1 / 7) − 1) / 2 ≈ −0.429
Derivative at each other token  = (1 / 7) / 2 ≈ 0.071

The printed gradient has one row for each of the four positions and one column for each of the seven tokens. The rows for the two prompt targets are all zero. In each of the two response rows the target’s entry is −0.429 and the other six entries are 0.071, so the row sums to zero. Read as an instruction to an optimizer, a response row says to raise the target’s logit and lower the other six by the same total amount. A prompt row says nothing.

Real training also needs an unambiguous chat or completion format, boundary tokens, padding handling, held-out examples, and an optimizer. These define the concrete learning task around the objective.

Fine-tuning changes parameters. Prompting supplies instructions or demonstrations as input for the current response. Retrieval supplies selected source passages as additional input. These approaches can be combined, but they modify different parts of the system.

The toy behavior shows how the routes differ. A prompt could ask for one more word and then a stop, and it would have to ask again on every request. Fine-tuning trains on pairs like the one above until the model does that without being asked, which costs a training run and removes the instruction from every later prompt. Retrieval does not fit this behavior at all, because a habit of formatting is not a fact that some source passage states. It suits answers that live in documents.

Supervised fine-tuning imitates demonstrated target responses. Preference training uses comparisons or feedback about outputs and introduces a different training objective. A few demonstrations of good behavior do not guarantee that behavior on unfamiliar inputs.

Imitation has a blind spot, and the loss in Part 2 shows where it is. Every term asks how much probability the model gave the one demonstrated token. A different response that would have served the reader equally well earns no credit, because it is not the response in the pair. Preference training exists for the cases where it is easier to say which of two outputs is better than to write the single best one.

Evaluate both the target task and capabilities the original model should retain. Repeated training on a narrow dataset can overfit or weaken other behavior. Full-parameter updates are one option; the next lesson describes a smaller trainable update.