Return the prediction to the input

A journey through autoregressive generation with Botie.

Large Language Models · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

How Does One Prediction Become a Whole Passage?

Part 1 of 3

A model that predicts the next token produces only one distribution at a time. To write a passage, we need to decide what happens after that prediction. The token we select becomes part of the context for the following decision.

Sources for this history

Radford and colleagues (2019), Language Models Are Unsupervised Multitask Learners

← Previous lesson: Next-Token Prediction Read the full lesson and derivations