Give the unlikely token a chance

A journey through sampling and temperature with Botie.

Large Language Models · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Why Does Always Choosing the Favourite Produce Dull Text?

Part 1 of 3

A sentence can have several reasonable continuations. Always selecting the highest-scoring option sounds safe, but repeated local choices can produce repetitive or uninteresting passages. Sampling indiscriminately has the opposite problem: very unlikely tokens can derail the continuation.

Sources for this history

Holtzman and colleagues (2019, revised 2020), The Curious Case of Neural Text Degeneration

← Previous lesson: Autoregressive Generation Read the full lesson and derivations