Notice the punctuation in another script

A journey through unicode word boundaries with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

What Happens When Text Uses More Than One Writing System?

Part 1 of 3

A rule written only for unaccented English letters quickly runs into trouble. An accent can be encoded separately from its base letter, and punctuation does not behave the same way in every script. Even deciding what a user should select with a double-click requires more than searching for spaces.

Sources for this history

The Unicode Consortium, Unicode Text Segmentation (Standard Annex #29)
The Unicode Standard, Version 1.0 (1991)

← Previous lesson: Splitting on Spaces Read the full lesson and derivations