Group the bytes before processing

A journey through patching without a vocabulary with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Could We Process Bytes Without Paying for Every Byte in the Largest Model?

Part 1 of 3

Bytes can represent unfamiliar text without assigning every word or fragment a vocabulary entry. The difficulty is sequence length. If an expensive global model processes every byte as a separate position, broad coverage can come with a substantial computational cost.

Sources for this history

Pagnoni and colleagues (2024), Byte Latent Transformer: Patches Scale Better Than Tokens

← Previous lesson: Hashing Characters Read the full lesson and derivations