Compare complete subword paths

A journey through the unigram language model with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Must a Word Have Only One Useful Segmentation?

Part 1 of 3

A vocabulary can spell the same word through several sequences of pieces. A fixed merge order commits to one route, but another route may use pieces that generalise better when the wording or domain changes.

Sources for this history

Kudo (2018), Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

← Previous lesson: WordPiece Read the full lesson and derivations