Sampling and Temperature

Three different things are easy to run together here. A model produces scores. The scores are turned into probabilities. A selection rule turns the probabilities into one chosen token. The controls in this lesson act on the second step, and they are easier to follow once the three are held apart.

A logit can be any finite real number. Softmax compares all vocabulary logits to produce nonnegative shares that total one. The absolute size of a score is less important than its difference from other scores.

The live example holds four candidates and their scores fixed. The candidate sat has a logit of 2, slept has 1, ran has 0 and sang has −1. None of those is a probability. They do not total one, and one of them is negative. Softmax turns them into the shares in the last column.

Only the gaps between the scores matter. Add 1,000 to every logit and the four probabilities come out the same, because each candidate is exactly as far from the others as before.

Sampling uses these shares to make a random selection. One draw can select a relatively unlikely token. Over many independent draws from an unchanged distribution, selection frequencies approach the assigned probabilities. In text generation, the distribution normally changes after every appended token.

Temperature divides the logits before softmax. A low positive value magnifies score differences and makes the distribution more concentrated. A high value reduces their influence and makes the distribution more even.

Reading down a column shows what the control does. The favorite, sat, holds 0.8650 at a temperature of 0.5 and 0.3849 at 3. The least likely candidate, sang, moves the other way, from 0.0021 to 0.1416. The scores are the same in every row. What changes is how strongly the preference they express is acted on.

For finite logits, positive temperature preserves ranking. At very low temperature, sampling concentrates on the maximum-score choices. Greedy decoding is implemented separately here, so there is no division by zero.

Both statements are visible in the table. No row changes the order of the four candidates. And at 0.1 the favorite has everything to four decimal places, so sampling at that setting behaves like always taking the largest share, without a temperature of zero ever being needed.

Let T be a strictly positive temperature, and let l contain the logits. Subtracting the maximum before exponentiating leaves the distribution unchanged and avoids overflow from large positive scores. The example holds the following four logits fixed.

l = [2, 1, 0, −1]

At T = 1:
shifted scores = [0, −1, −2, −3]
weights = [1, exp(−1), exp(−2), exp(−3)]
probabilities = weights / sum(weights)

In general:
pᵢ = exp((lᵢ − max(l)) / T)
     / Σⱼ exp((lⱼ − max(l)) / T)

The table calculates the normalized probabilities. The same scores and vocabulary remain in place while the temperature changes.

T = 1:    shifted scores / T = [0, −1, −2, −3]
          weights ≈ [1, 0.3679, 0.1353, 0.0498],  sum ≈ 1.5530
          probabilities ≈ [0.6439, 0.2369, 0.0871, 0.0321]

T = 0.5:  shifted scores / T = [0, −2, −4, −6]
          weights ≈ [1, 0.1353, 0.0183, 0.0025],  sum ≈ 1.1561
          probabilities ≈ [0.8650, 0.1171, 0.0158, 0.0021]

T = 2:    shifted scores / T = [0, −0.5, −1, −1.5]
          weights ≈ [1, 0.6065, 0.3679, 0.2231],  sum ≈ 2.1975
          probabilities ≈ [0.4551, 0.2760, 0.1674, 0.1015]

Top-p, also called nucleus sampling, orders tokens by probability. Keep the smallest prefix whose cumulative probability reaches the threshold. Retain the token that crosses it, remove later tokens, and normalize again.

Using the assigned distribution from the guided illustration gives a simple calculation:

Sorted probabilities = [0.50, 0.30, 0.15, 0.05]
Top-p threshold = 0.80

First candidate:   cumulative probability = 0.50
Second candidate:  cumulative probability = 0.50 + 0.30 = 0.80
Retained shares = [0.50, 0.30]

Final distribution = [0.50 / 0.80, 0.30 / 0.80, 0, 0]
                   = [0.625, 0.375, 0, 0]

These illustrative shares differ from the live table's softmax values. They show the filtering arithmetic without rounding ambiguity. A fixed top-p threshold can retain many tokens for an uncertain distribution and very few for a concentrated one.

Retained shares = [0.6439, 0.2369],  total = 0.8808

Final distribution = [0.6439 / 0.8808, 0.2369 / 0.8808, 0, 0]
                   ≈ [0.7311, 0.2689, 0, 0]

A selection policy is a short list of decisions, and each one changes the distribution the sampler finally draws from. Stating them in order is what makes a result repeatable and makes two results comparable.

The SDK applies temperature first and top-p second. Its stable sort preserves vocabulary order when probabilities tie. Passing a random generator into sample keeps ownership of the random stream explicit.

The code below asks for a temperature of 0.8 and a top-p of 0.9 on the page’s four logits, and then draws once from a seeded generator. Followed by hand, it goes like this.

After temperature 0.8:  [0.7183, 0.2058, 0.0590, 0.0169]
Running total:          0.7183, then 0.9241
Retained:               sat and slept
Final distribution:     [0.7773, 0.2227, 0, 0]
Seeded draw:            slept

The order of the two controls changed the answer here. On the unscaled distribution a threshold of 0.9 is reached only at the third candidate, at 0.9679, so filtering first would have kept ran as well. The draw is worth a second look too. It selected slept, which held 0.2227 against 0.7773 for sat. The same policy with a different seed can return sat, and both are correct outputs of the policy.

import numpy as np
from oop_ml.numpy.modern import TokenDistribution

distribution = TokenDistribution.from_logits(
    np.array([2.0, 1.0, 0.0, -1.0]),
    temperature=0.8, top_p=0.9,
)
token_id = distribution.sample(np.random.default_rng(4))
print(distribution.probabilities)
print(token_id)

Lower temperature can reduce variation, but it cannot remove an error already favored by the model. Higher temperature can surface alternatives, including unsuitable ones. Neither should be treated as a direct correctness control.

A model's reported next-token probability, the probability after sampling adjustments, and the probability that an answer is factually correct are different quantities. Choose a policy by evaluating complete responses for the task.