Natural Language Processing
Give Botie a vocabulary
A journey through what a token is with Botie.
Natural Language Processing · A journey with Botie
First visit? Learn the workshop controlsBotie's notebook
How Did Text Become Something a Program Could Work With?
Part 1 of 3
As humans, we recognise letters and words in the text we read. A program needs explicit rules for what to treat as one item. Even before modern language models, a compiler reading a program needed to distinguish a name, a number, and a punctuation symbol instead of treating the input as an undifferentiated stream of characters.