Attention
A word's meaning in a sentence can depend on words some distance away. Attention gives each position a way to compare the available information and combine the parts most relevant to its current representation.
10 sections · 10 questions · 3 coding challenges
The arrow keys move between sections too.
Why Must a Translator Remember the Whole Sentence in One Summary?
BOTIE
Bahdanau, Cho and Bengio's 2014 work challenged this fixed-summary bottleneck in neural translation. Their decoder could use a weighted combination of source representations at each output step. The part of the source contributing most strongly could change as the translation progressed.
Attention gives us a way to ask for relevant information. We compare a current request with the available items, turn the comparison scores into weights, and combine the information those items carry. The query, key, and value formulation used in the 2017 Transformer makes these roles explicit. We will separate those roles before calculating the scores, so the matrix operations have a purpose we can follow.
Which Words Help Us Understand This One?
The word bank can refer to a river bank or a place holding money. The surrounding words help us interpret it. One fixed starting vector for bank does not identify which context applies.
Attention lets a token gather information from other tokens, with different contributions for different contexts.
Keep the same word and change its context
Read the two sentences. Bank is the same token, but river and money suggest different uses. We need its representation to incorporate relevant surrounding information.
Sentence one
- we
- sat
- on
- the
- Highlighted: river
- bank
Sentence two
- we
- kept
- Highlighted: money
- in
- the
- bank
The arithmetic implements comparison followed by combination
The next sections build queries, keys, and values from input vectors, calculate comparison scores, and show how their shares combine values.
The colour-vector experiment uses assigned projections so every calculation is inspectable. It is not a trained demonstration of language understanding.
Explore the full example
Start with one head and both checkboxes off. Compare the rows for the two red tokens, change one token, then block future tokens. Inspect the role vectors to see the numbers each head actually reads.
These colour names have assigned vectors. The four projection matrices are identities, so the example exposes the arithmetic without claiming to have learned language.
Calculating the example…
Part 1. From Context to a Weighted Combination
1. Give the three roles different jobs
A token’s starting vector is the same wherever the token appears. In the example on this page, red begins as [1, 0, 0, 0] at the first position and again at the last. Nothing in those four numbers says that a blue token sits between them. What is missing is a way for each token to take in information from the others, in amounts the tokens themselves decide.
Attention builds that from three vectors per token, and each has one job. The query belongs to the token doing the gathering. It describes what that token is looking for. The key belongs to a token that might be read. It is what a query is compared with. The value belongs to the same token as the key. It is the information that token hands over, in whatever amount the comparison decides.
A token starts with an input vector. Three separate sets of weights transform it into a query, a key, and a value. Separate weights matter because the features useful for comparing two tokens need not be the same features we want to pass forward.
For example, a model might learn a comparison useful for connecting a pronoun with an earlier noun. The value can then carry information from that noun. This is a possible learned use of the operation, not a role we assign to a particular head in advance.
Keep in mind
The colour example sets all three sets of weights so that they leave a vector unchanged. A token’s query, key and value are then all equal to its input vector, which lets us check every number by hand. A trained layer learns three different transformations.
2. Compare first, then decide the shares
The calculation for one gathering token has four steps. We will follow the first red token through them, and the worked example below shows the arithmetic.
First, compare. We use a dot product to compare a query with each key. That means multiplying corresponding entries and adding the products. The red query scores 1 against each red key and 0 against the blue key, so its row of scores is 1, 0, 1.
Second, scale. Each score is divided by the square root of the vector width. The width is 4 here, so the row becomes 0.5, 0, 0.5. Part 2 explains what the division is for.
Third, turn the scores into shares. A function called softmax raises e to the power of each score, then divides each result by the total for the row. Every share is therefore positive, and the row totals one. A larger score receives a larger share. A score of zero still receives a share, because e to the power of zero is 1 and not 0.
Fourth, combine. The model does not have to select a single token. It can combine information from several. Each share multiplies the corresponding value vector, and we add those contributions to get the result for the querying token.
One red token reads three colour vectors
In the default example, the role projections leave the assigned vectors unchanged. The first red query matches both red keys equally. Its comparison with the blue key is smaller.
red = [1, 0, 0, 0]
blue = [0, 1, 0, 0]
red · red = (1 × 1) + (0 × 0) + (0 × 0) + (0 × 0) = 1
red · blue = (1 × 0) + (0 × 1) + (0 × 0) + (0 × 0) = 0
Scaled scores = [1, 0, 1] / √4 = [0.5, 0, 0.5]
Shares = [exp(0.5), exp(0), exp(0.5)] / (2 × exp(0.5) + 1)
≈ [0.3837, 0.2327, 0.3837]
Result ≈ (0.3837 × red) + (0.2327 × blue) + (0.3837 × red)
≈ [0.7673, 0.2327, 0, 0]Both red occurrences contribute. There is no rule saying that the largest share must take everything. Displayed values are rounded; the calculation uses full precision.
The blue token reads the same three vectors
Now let the blue token gather. Its query matches its own key and neither red key, so its row of scores has a single 1, in the middle.
blue · red = 0
blue · blue = 1
Scaled scores = [0, 1, 0] / √4 = [0, 0.5, 0]
Shares = [exp(0), exp(0.5), exp(0)] / (exp(0.5) + 2)
≈ [0.2741, 0.4519, 0.2741]
Result ≈ (0.2741 × red) + (0.4519 × blue) + (0.2741 × red)
≈ [0.5481, 0.4519, 0, 0]Blue gives its largest share to itself, yet its result holds more red than blue. Each red token contributes 0.2741, and two of them outweigh blue’s own 0.4519. The result is a combination of everything the token read. It is not a copy of the best match.
Keep in mind
Look at what the two red tokens received. Each started as [1, 0, 0, 0] and each ends as about [0.7673, 0.2327, 0, 0]. Their vectors now record that a blue token is in the sequence, which the starting vector could not do. They are also still identical to each other, because nothing in the calculation used where a token sits. Part 3 returns to that.
Part 2. Calculating Attention
3. Write the sequence as rows
Part 1 followed one token at a time. A layer does the same work for every token at once, and writing the sequence as a matrix is what makes that possible. Let X contain one input vector per row, so three tokens of four numbers each give three rows and four columns. Q, K, and V hold the projected queries, keys, and values, one row per token. The SDK stores one output unit per row of each weight matrix, so its projections use the transposes shown below.
Q = X W_Qᵀ + b_Q
K = X W_Kᵀ + b_K
V = X W_Vᵀ + b_V
Scores = Q Kᵀ / √d_k
A = softmax(Scores), applied separately to each row
Context = A VRead the six lines in order. The first three build a query, a key and a value for every token. The product Q Kᵀ takes the dot product of every query with every key, so Scores has one row per gathering token and one column per token being read. Softmax turns each row into shares. The product A V then combines the value rows using those shares, which gives one result row per token.
The colour example as whole matrices
The example’s projections leave every vector unchanged, so Q, K and V all equal X. The two worked examples from Part 1 are the first two rows of each matrix below. The third row repeats the first because the third token is red again.
X = Q = K = V =
[1, 0, 0, 0] red
[0, 1, 0, 0] blue
[1, 0, 0, 0] red
Q Kᵀ =
[1, 0, 1]
[0, 1, 0]
[1, 0, 1]
Scores = Q Kᵀ / √4 =
[0.5, 0, 0.5]
[0, 0.5, 0 ]
[0.5, 0, 0.5]
A ≈
[0.3837, 0.2327, 0.3837]
[0.2741, 0.4519, 0.2741]
[0.3837, 0.2327, 0.3837]
Context = A V ≈
[0.7673, 0.2327, 0, 0]
[0.5481, 0.4519, 0, 0]
[0.7673, 0.2327, 0, 0]Each row of A totals one, because softmax was applied to that row by itself. A column is under no such rule. The middle column totals about 0.9173, which is how much of the blue value the three results hold between them.
The head width is denoted by d_k. Dividing by its square root controls the growth in dot-product variance as the width increases, under the usual assumption of roughly independent, centered coordinates with unit variance. It does not normalize every possible input distribution.
The division matters because softmax responds to the size of the scores as well as their order. Larger gaps between scores push the shares further apart. A dot product over more coordinates adds more products, so without the division a wider head would tend to produce larger scores and more one-sided rows of shares for no reason other than its width. The two-head example in the next section shows the same comparison divided by a different width.
4. Keep several combinations separate
One row of shares can express one way of reading the sequence. A token may need more than one. It might need to find the noun a pronoun refers to and, separately, the verb that governs it, and a single row of shares has to trade one off against the other.
One head produces one weighted combination per token. Multiple heads calculate separate combinations in smaller vector spaces. Their results are placed side by side, then an output projection mixes their coordinates.
head_i = softmax(Q_i K_iᵀ / √d_k) V_i
Output = concatenate(head_1, …, head_h) W_Oᵀ + b_OEach head has its own queries, keys and values, so each head has its own table of shares. The total width is divided among the heads. With a width of four, one head compares in four coordinates and two heads compare in two coordinates each.
The same three tokens with two heads
Set the head count to two and keep the tokens as red, blue, red. The first head reads the first two coordinates of every vector, and the second head reads the last two.
First head sees: red = [1, 0] blue = [0, 1]
Second head sees: red = [0, 0] blue = [0, 0]
First head, red row:
Scaled scores = [1, 0, 1] / √2 ≈ [0.7071, 0, 0.7071]
Shares ≈ [0.4011, 0.1978, 0.4011]
Second head, red row:
Scaled scores = [0, 0, 0]
Shares = [1/3, 1/3, 1/3]
Output for red ≈ [0.8022, 0.1978, 0, 0]The two heads reached different rows of shares from the same tokens. The first head still tells red from blue. Its shares are further apart than the one-head row of 0.3837, 0.2327, 0.3837, because the same scores were divided by √2 and not by √4. The second head sees nothing that separates the tokens, so it reads all three equally. Its values are all zero in this example, so its equal shares add nothing, and the output keeps the first head’s result in its first two coordinates.
Changing the head count in this example changes which coordinates share a comparison. Real models learn the projections; the example keeps identity projections so the effect of splitting the coordinates is visible.
›How the gradient passes through softmax
Increasing one share necessarily changes the others because the row must still total one. If G is the arriving derivative with respect to the shares, the derivative with respect to the scaled scores is:
dScores = A ⊙ (G − sum(G ⊙ A, over the row))The symbol ⊙ multiplies matching entries. The sum inside the bracket is the average of G over the row, weighted by the shares, so the bracket measures how far each share’s derivative sits above or below that average. If every entry of G in a row is the same number, the bracket is zero and nothing passes back to that row’s scores. No change to the scores can raise every share together when the row must total one.
The NumPy backward pass then differentiates both sides of the query-key comparison, the value combination, and all four projections. Numerical gradient tests check the weights, biases, and inputs with one or several heads and with both masking settings.
Check yourself
Questions on Parts 1 and 2
Question 1 of 5
The query, key and value projections could share one set of weights without changing what the layer can learn.
Part 3. Order, Causality, and the Larger Network
5. Prevent a token from reading the future
When training a next-token predictor, a position must not see the tokens it is meant to help predict. A causal mask excludes later positions. It sets their scores to negative infinity before softmax, which gives them zero contribution while keeping the allowed shares normalized.
This example allows each position to read itself. The first position therefore reads only itself when the mask is enabled. Padding masks are a separate concern and are not implemented by this layer.
The mask has to act on the scores and not on the finished shares. If we set a share to zero after softmax, the shares left in that row would total less than one, and the result would be a shrunken combination. Removing the score first means softmax divides by the total of the allowed entries only, so the allowed shares fill the whole row.
Blocking the future in the colour example
Tick Block future tokens with the tokens at red, blue, red. A blocked score is written −∞ below, and e raised to that power is zero.
Scaled scores with the mask =
[0.5, −∞, −∞ ]
[0, 0.5, −∞ ]
[0.5, 0, 0.5]
A ≈
[1, 0, 0 ]
[0.3775, 0.6225, 0 ]
[0.3837, 0.2327, 0.3837]
Context ≈
[1, 0, 0, 0]
[0.3775, 0.6225, 0, 0]
[0.7673, 0.2327, 0, 0]The first token may read only itself, so its one allowed share is 1 and its result is its own vector. The blue token has lost the red that follows it. Its two remaining shares still total one, which raises its share for itself from 0.4519 to 0.6225. The last token has no future to block, so its row is the one Part 1 calculated.
The two red tokens now receive different results, [1, 0, 0, 0] and about [0.7673, 0.2327, 0, 0]. The mask did not tell either token where it sits. It changed what each one was allowed to read.
6. Attention is one component of a transformer
Go back to the example with the mask off. The two red tokens received the same row of shares and the same result. That follows from the calculation and not from the choice of tokens. A score depends only on the two vectors being compared, and the result is a sum, which keeps no record of the order of its terms.
We can check by moving the tokens. Set them to blue, red, red, and each token gets the result it had before. Blue still receives about [0.5481, 0.4519, 0, 0] and each red still receives about [0.7673, 0.2327, 0, 0]. The results moved with their tokens and nothing else changed, so the layer gives a token the same result wherever in the sequence it is placed.
Unmasked attention by itself does not encode sequence order. Positional encoding supplies that information. Residual connections, normalization, and feed-forward transformations help build a larger network around attention.
The example can show the first of those. Tick Add positional encoding, and a different pattern of four numbers is added to the vector at each position before attention reads it. The two red tokens then enter as [1, 1, 0, 1] and about [1.9093, −0.4161, 0.02, 0.9998], which are no longer the same vector. Their rows of shares differ as well, about [0.3349, 0.4053, 0.2599] for the first and [0.2013, 0.1547, 0.6440] for the last.
Those rows are harder to read than the colour-only ones, and that is worth noticing. The projections in this example leave vectors unchanged, so the dot product compares the position numbers and the colour numbers together. A trained layer learns projections that decide how much of each kind of information a comparison uses.
The cost of the layer is set by the table of shares. Every token is compared with every token, so three tokens need nine shares per head and six tokens need thirty-six. Doubling the sequence multiplies the table by four.
Keep in mind
This layer implements self-attention, where queries, keys, and values come from the same sequence. Its explicit attention matrix grows quadratically with sequence length. The transformer-block lesson now combines it with normalization, residual connections, and a feed-forward network in a small decoder forward pass. That example does not implement cross-attention or full language-model training.
The scaled dot-product and multi-head construction is described in Attention Is All You Need. Attention shares show how values were combined in a layer; they do not, on their own, establish why a complete model made a decision.
Check yourself
Questions on Part 3
Question 1 of 5
How does the causal mask stop a position reading the tokens it is meant to help predict?
Try it with Botie
Build with Botie
Write and run Python