What happens to two spaces?

A journey through splitting on spaces with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Could the Spaces Do the First Part of the Work?

Part 1 of 3

A search system needs to count and index the items in a document. For much English text, spaces seem to offer a convenient first answer: the stretches between them often resemble the words a reader would look for.

Sources for this history

Manning, Raghavan and Schütze (2008), Introduction to Information Retrieval: Tokenization

← Previous lesson: Bytes and Characters Read the full lesson and derivations