Add a narrow correction

A journey through low-rank adaptation with Botie.

Adapting and Evaluating Modern Models · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Must We Change Every Weight to Teach a Model a New Task?

Part 1 of 3

A large pretrained model already contains a useful representation. Updating every weight for each new task can require substantial training memory and a separate large set of parameters to store for every adaptation. We would like to preserve most of that shared work.

Sources for this history

Hu and colleagues (2021), LoRA: Low-Rank Adaptation of Large Language Models

← Previous lesson: Supervised Fine-Tuning Read the full lesson and derivations