Natural Language Processing
Apply the merge the vocabulary learned
A journey through byte pair encoding with Botie.
Natural Language Processing · A journey with Botie
First visit? Learn the workshop controlsBotie's notebook
Could Repeated Fragments Solve the Rare-Word Problem?
Part 1 of 3
A whole-word vocabulary has a recurring problem: names, compounds, and unfamiliar word forms keep arriving. Storing every possibility is impractical. Spelling everything one character at a time avoids some missing-word problems but makes sequences longer.