Natural Language Processing

Make the space visible

A journey through sentencepiece with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Why Should Vocabulary Learning Require a Word Split First?

Part 1 of 3

Many subword systems begin by asking another tool to split text into words. That creates a dependency before vocabulary learning even starts, and the split may be difficult in a writing system that does not place spaces between words.

Sources for this history

Kudo and Richardson (2018), SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing

← Previous lesson: The Unigram Language Model Read the full lesson and derivations