Natural Language Processing

Tell two identical tokens apart

A journey through positional encoding with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

How Would a Model Know Which Word Came First?

Part 1 of 3

The same words can describe different events when we change their order. A collection of word representations alone does not distinguish who acted on whom. If our architecture no longer processes tokens one after another, we need another way to make their positions available.

Sources for this history