Positional Encoding

The same words in a different order can mean something different. A model that receives token vectors needs a way to know where those tokens occur, which positional information provides.

9 sections · 5 questions · 3 coding challenges

1 / 9

The arrow keys move between sections too.

How Would a Model Know Which Word Came First?

BOTIE

The same words can describe different events when we change their order. A collection of word representations alone does not distinguish who acted on whom. If our architecture no longer processes tokens one after another, we need another way to make their positions available.

The 2017 Transformer removed recurrence from its sequence-processing architecture and added positional information to token embeddings. Its paper considered learned positions and used a fixed pattern of sine and cosine values in the reported model.

Think of attaching an address to each token's representation. The content says which token is present; the positional information helps identify where it appears. Different position schemes provide different ways to compare addresses, including relative separations. Position is additional information for the model to use, not a guarantee that it will interpret every ordering correctly.

Sources for this history

Vaswani and colleagues (2017), Attention Is All You Need

The Same Words Can Say Something Different

The dog chased the cat and the cat chased the dog contain the same words but describe different events. A representation listing only which words occurred has lost information.

Position information gives a model access to where each token appears, alongside what token it is.

Hold the words fixed and change their order

The vocabulary entries are unchanged, but the actor and recipient change. Order is information the task may need.

First sentence

dog
chased
cat

Second sentence

cat
chased
dog

The same words in different positions.

1 of 4

Content and location become available together

The numerical example constructs the position vectors and shows their addition to token embeddings.

A model must still learn to use this information. Encoding position does not prescribe a grammatical interpretation.

Part 1. Keep Content and Position Available

1. Add information without adding coordinates

An attention layer compares every token with every other token at once, and nothing in that comparison says which token came first. Hand it the same tokens in another order and it answers the same rows in that other order, because all it ever saw was a set of vectors. The words of a sentence are not a set. Which token precedes which is information the input has to carry, and an embedding alone does not carry it, since a token looks up the same vector wherever it occurs.

What is missing is a vector that depends on where a token sits and on nothing else. We make that position vector the same width as the token vector, then add corresponding entries. This keeps the shape expected by the next layer. It also means the next layer sees a mixture of content and position, and must learn to use that mixture.

Adding a position vector is a design choice. A model could instead learn a table of position vectors, or encode relative positions in another way. We start with a fixed pattern because every value can be reproduced without training another table, and because a position the training data never reached still gets a vector, since the vector is a function of the position rather than an entry looked up in a table.

The same token at three positions

For a four-coordinate vector, the first sine-cosine pair changes faster than the second. At position zero, all the sine entries are zero and the cosine entries are one.

Token vector = [1, 0, 1, 0]
Position 0   = [0, 1, 0, 1]
Combined     = [1 + 0, 0 + 1, 1 + 0, 0 + 1]
             = [1, 1, 1, 1]

Position 1 ≈ [0.8415, 0.5403, 0.0100, 0.99995]
Combined   ≈ [1.8415, 0.5403, 1.0100, 0.99995]

Position 2 ≈ [0.9093, −0.4161, 0.0200, 0.9998]
Combined   ≈ [1.9093, −0.4161, 1.0200, 0.9998]

The token has not changed. The different sums indicate that its occurrences occupy different places in the sequence. Subtract any two combined rows and the token cancels, leaving the difference of two position vectors, which is the same whatever token stood at those two places. That is what lets a later layer read a distance off the mixture without knowing what the content was.

Part 2. Building a Position Vector

2. Use pairs that change at different rates

Let p be the position, starting at zero. Let d be the vector width and k identify a pair of coordinates. B controls the spread of frequencies. Each pair has its own rate, one over B raised to a power that grows with k, so the first pair turns by one radian per position and every later pair turns more slowly. With the default base used here, a width of four has two pairs.

B = 10000
angle(p, k) = p / B^(2k/d)
PE[p, 2k]   = sin(angle(p, k))
PE[p, 2k+1] = cos(angle(p, k))

For d = 4 at p = 1:
First pair angle  = 1 / 10000^0   = 1
Second pair angle = 1 / 10000^0.5 = 0.01

A single sine value repeats and can be ambiguous. Pairing sine and cosine gives the phase of one oscillation, which is a point on a circle, and moving forward by a fixed number of positions turns that point by a fixed angle wherever the move started. A layer above can therefore read a distance as a rotation, and the rotation for a given distance is the same everywhere in the sequence.

Using several pairs at different frequencies provides patterns over different distances. The first pair completes a turn every 2π positions, which is about 6.28, and the second needs a hundred times as many. The fast pair on its own would nearly confuse position six with position zero, since by then it has come most of the way round.

Position 0, first pair  = (0, 1)
Position 6, first pair  ≈ (−0.2794, 0.9602)

Position 0, second pair = (0, 1)
Position 6, second pair ≈ (0.0600, 0.9982)

The slow pair has barely moved, and that small movement is what keeps the two positions apart. Lowering the base speeds the slow pairs up. At a base of 2 the second pair’s angle at position one is 1 over the square root of 2, about 0.7071, and its entries become 0.6496 and 0.7602 instead of 0.0100 and 0.99995, which is what the frequency base control in the playground shows. None of this makes the finite-width encoding a guarantee of uniquely identifying arbitrarily distant positions.

3. Add the pattern to every sequence

The SDK constructs the pattern once, freezes it, and broadcasts it over the batch. Position zero has the same positional vector in every sequence, because position means where a token sits inside its own sequence and not which sequence it belongs to. If the vector width is odd, the last coordinate contains a sine value without its paired cosine.

output[batch, position, coordinate]
    = input[batch, position, coordinate] + pattern[position, coordinate]

The pattern has no trainable parameters. Adding a constant has derivative one with respect to the input, so the backward pass returns the arriving input gradient unchanged, and a training step leaves the layer exactly as it was.

Part 3. Order Is Different from Permission to Look

A positional encoding gives information about location. A causal mask restricts which locations can contribute. They solve different problems and can be used together. The attention playground lets you enable them separately, and the two controls change different things, since the mask changes which columns of a row may be used and the encoding changes what every row holds.

Two identical tokens, with and without positions

Take the attention lesson’s default example, the tokens red, blue, red with projections that leave every vector unchanged, and read the row of shares each red token produces over the three tokens.

Positions off, mask off
  first red = [0.3837, 0.2327, 0.3837]
  last red  = [0.3837, 0.2327, 0.3837]

Positions on, mask off
  first red = [0.3349, 0.4053, 0.2599]
  last red  = [0.2013, 0.1547, 0.6440]

Positions off, mask on
  first red = [1, 0, 0]
  last red  = [0.3837, 0.2327, 0.3837]

Without positions the two red tokens are the same input, so their rows are identical and nothing downstream can tell them apart. Adding the pattern makes them different inputs, and the rows differ with no training at all. The mask does something else. It leaves the last red’s row exactly as it was, because that token may read everything before it, and it empties the entries the first red is not allowed to use while the remaining shares still total one.

Keep in mind

Being able to calculate a position vector for a longer sequence does not guarantee that a trained model will work well at that length. Position information also does not impose grammatical understanding. It gives the model information from which useful order-dependent behavior can be learned.

The sinusoidal construction appears in Attention Is All You Need. The parameter called wavelength in this SDK supplies the frequency base B in the equations above.