Keep the IDs and embeddings together

A journey through moving a vocabulary with Botie.

Natural Language Processing · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

What Happens to a Learned Word Vector When Its Address Changes?

Part 1 of 3

A pretrained model's embedding table belongs to a particular vocabulary. If we adapt the model to another language or domain, a new vocabulary may describe the text more efficiently. But an ID in the new vocabulary can name a different piece from the same ID in the old one.

Sources for this history

Minixhofer, Paischer and Rekabsaz (2022), WECHSEL: Effective Initialization of Subword Embeddings for Cross-Lingual Transfer of Monolingual Language Models

← Previous lesson: Greedy Coverage Read the full lesson and derivations