Natural Language Processing
Keep the word, number and punctuation apart
A journey through the pattern language models use with Botie.
Natural Language Processing · A journey with Botie
First visit? Learn the workshop controlsBotie's notebook
Why Put Rules Before a Learned Vocabulary?
Part 1 of 3
A learned subword tokenizer can merge neighbouring units into larger pieces. Before it learns or applies those merges, we may want to decide which kinds of characters are allowed to be grouped together. Otherwise a frequent combination could cross a boundary we wanted to retain.