Low-Rank Adaptation

Adapting every weight in a large model can be expensive. LoRA keeps the original weights fixed and learns a smaller correction through two narrow matrices, reducing the number of parameters that need updating.

10 sections · 10 questions · 4 coding challenges

1 / 10

The arrow keys move between sections too.

Must We Change Every Weight to Teach a Model a New Task?

BOTIE

A large pretrained model already contains a useful representation. Updating every weight for each new task can require substantial training memory and a separate large set of parameters to store for every adaptation. We would like to preserve most of that shared work.

Hu and colleagues introduced LoRA in 2021, motivated by the possibility that useful task-specific changes occupy a much smaller space than the full weight matrices. They froze pretrained weights and learned low-rank updates through smaller matrices.

The original transformation remains available. Two smaller learned transformations together provide an adjustment to it. Their restricted size limits the rank of the update, reducing the number of trainable parameters. This is an assumption about the changes needed for adaptation, not a claim that the original model itself has only a few useful directions. The rank controls the capacity of the adjustment.

Sources for this history

Hu and colleagues (2021), LoRA: Low-Rank Adaptation of Large Language Models

Do We Need to Change Every Weight to Teach a New Task?

A large model can contain far more weights than we can afford to update and store separately for every task. Yet adapting it may not require an independent change to every connection. Could we describe a useful change with fewer adjustable numbers?

Low-rank adaptation, or LoRA, keeps an existing weight matrix frozen and learns an additional transformation through two smaller matrices. The update travels through a narrow intermediate space. Its width determines the rank limit.

Keep the original projection available

A projection combines input coordinates into output coordinates using a weight matrix. The pretrained matrix already does useful work, so LoRA preserves it as a fixed route. Freezing it means the optimizer does not update its entries.

Input

Input vector

Base projection

Frozen weights

Base result

Original output

The base route keeps its pretrained parameters.

1 of 4

Calculate the update before treating rank as a tuning shortcut

The example displays the assembled weight change, its effect on an input, and both parameter counts.

The factors are chosen values. Training an adapter requires a task objective, gradients, and a decision about which model projections receive adapters.

Part 1. Freeze the Base and Describe a Correction

1. Separate what stays fixed from what learns

The previous lesson fine-tuned by letting every weight move. A projection that carries one layer’s values to the next has one weight for every pairing of an input coordinate with an output coordinate. A model built from many wide projections therefore has a great many weights, and a fully fine-tuned copy for each task has to store all of them again. The question in the title is whether a task’s change really needs that many numbers.

LoRA freezes a pretrained weight matrix and adds a trainable low-rank update. During task training, gradients update the adapter factors while the base matrix stays fixed. The full model still runs a forward and backward computation through the relevant paths.

It helps to name the two things separately. The base is the pretrained matrix. It is frozen, which means the optimizer never writes to it. The correction is a second matrix of the same shape that is added to the base. The adapted projection is their sum, and only the correction is learned.

Adapted projection = base     + correction
                     (frozen)   (learned)

The tiny projection on this page maps four coordinates to four, so it has 16 weights. Full fine-tuning would learn and store 16 changed numbers for a task. The adapter worked through below describes its correction with 8.

Reducing trainable parameters can reduce optimizer-state storage and make task-specific checkpoints smaller. It does not remove the need to load the base model or store the activations needed for training.

Both halves of that follow from what is frozen and what is not. Many optimizers keep running numbers of their own for each parameter they update, so fewer trainable parameters means less of that state, and a saved task holds only the adapter. But the adapter’s gradient is found by running the whole model forward and then backward. That needs the base weights in memory, and it needs the values each layer produced on the way through.

2. Make rank a limit on the possible update

The adapter maps an input through a narrow intermediate space and then back to the output width. If that intermediate space has one coordinate, all output corrections are scaled versions of one learned direction. More coordinates allow combinations of more directions.

Follow that with numbers. Take an adapter whose intermediate space has one coordinate. Its first factor reads only the first input coordinate. Its second factor writes one half of whatever it is handed into the second output coordinate. The base is the identity, which returns its input unchanged, so anything different in the output is the adapter’s doing.

Input	The one intermediate number	Correction added to the output
[1, 0, 0, 0]	1	[0, 0.5, 0, 0]
[0, 1, 0, 0]	0	[0, 0, 0, 0]
[3, 1, 0, 0]	3	[0, 1.5, 0, 0]
[−2, 0, 1, 4]	−2	[0, −1, 0, 0]

Four inputs through one rank-one adapter. These are the factors used in the code of Part 3.

Every correction in the last column is a multiple of [0, 0.5, 0, 0]. The input decides the multiple and nothing else. Four numbers went in, one number came through the middle, and one number cannot carry more than a single direction of change out the other side. That is what a rank of one means. With two intermediate coordinates the adapter could add any combination of two directions, and with r of them, r directions.

Rank constrains the update, not the rank of the frozen base matrix. A small-rank correction can be added to a full-rank base. Whether that restriction is sufficient depends on the task and the projections being adapted.

The example shows this as well. Its correction has a rank of one, and the identity it is added to has the full rank of four. Their sum still has a rank of four. A low-rank update adjusts a matrix. It does not flatten it.

Part 2. Calculate the Factored Update

3. Fix the matrix convention before multiplying

Our code uses row-vector inputs. W maps the input width to the output width, A maps into rank r, and B maps back out. The scalar α controls the adapter scale.

W: (d_in, d_out)
A: (d_in, r)
B: (r, d_out)

ΔW = (α / r) A B
y = x W + (α / r) (x A) B
  = x (W + ΔW)

The second expression shows the separate base and adapter routes. The third shows why this linear update can be merged into a copy of the base matrix for inference. Merging does not require another task-training step.

One input through both routes

Use the adapter from Part 1 on the input [1, 0, 0, 0], with α set to 1. The base W is the four-by-four identity, A is the column [1, 0, 0, 0] and B is the row [0, 0.5, 0, 0], so r is 1.

Base route:     x W = [1, 0, 0, 0]

Adapter route:  x A = (1 × 1) + (0 × 0) + (0 × 0) + (0 × 0) = 1
                (x A) B = 1 × [0, 0.5, 0, 0] = [0, 0.5, 0, 0]
                α / r = 1 / 1 = 1

Sum:            y = [1, 0, 0, 0] + (1 × [0, 0.5, 0, 0])
                  = [1, 0.5, 0, 0]

The merged form reaches the same output by doing the work in another order. Multiply the factors first, and ΔW is a four-by-four matrix whose only nonzero entry is 0.5, in the first row and second column. Add it to the identity, and that single matrix sends [1, 0, 0, 0] to [1, 0.5, 0, 0] in one multiplication.

The two forms suit different moments. During training the routes are kept apart, because A and B are the things being updated. Afterward a merged copy costs nothing extra to run, while a separate adapter lets one stored base serve several tasks by swapping a small pair of factors.

A common initialization makes one factor zero so the initial correction is zero while the other factor is nonzero. Setting both factors to zero would give both a zero first derivative through their product. The live example instead uses nonzero chosen factors so the correction is visible immediately.

›Why both factors cannot start at zero

The correction is a product, and the slope of a product with respect to one factor is the other factor. The gradient that reaches A is the loss’s gradient multiplied by B, and the gradient that reaches B is multiplied by x A. If both factors are zero, both gradients are zero, and neither factor ever leaves zero.

If only B starts at zero, the correction is still zero at the start, so the adapted model begins as exactly the pretrained one. But x A is not zero, so B receives a gradient on the first step. Once B has moved, A receives one too.

4. Count parameters instead of assuming a saving

The full projection has one parameter for each input-output pair. The adapter has one matrix on each side of the rank bottleneck.

Full projection: d_in × d_out
Adapter:         (d_in × r) + (r × d_out)

For a four-by-four projection:
Full update = 4 × 4 = 16
Rank one adapter = (4 × 1) + (1 × 4) = 8
Rank two adapter = (4 × 2) + (2 × 4) = 16
Rank three adapter = (4 × 3) + (3 × 4) = 24

The default rank-one case saves trainable parameters. Rank two breaks even on this tiny matrix; rank three uses more. Large projection widths allow useful low ranks to be a much smaller fraction of a full update.

The break-even point can be read off the two formulas. For a square projection of width d the full update has d × d parameters and the adapter has 2 × d × r, so the adapter is the smaller of the two only while r is less than half of d. At a width of four that leaves rank one and nothing else, which is what the counts above show. The same two formulas at larger widths give the picture the method was designed for.

Width d	Rank r	Full update, d × d	Adapter, 2 × d × r	Adapter as a share of full
4	1	16	8	50%
64	4	4,096	512	12.5%
256	4	65,536	2,048	3.1%
1,024	8	1,048,576	16,384	1.6%

Parameter counts for square projections. They are counts from the two formulas, and they say nothing about whether a given rank is enough for a task.

The adapter’s count grows in step with the width, and the full update’s grows with the width squared. So the wider the projection, the smaller the share a fixed rank takes.

Part 3. Connect the Operation to Fine-Tuning

5. Inspect an explicit adapter

The SDK takes a base matrix, both factors, and alpha. It returns the adapted output and exposes the assembled update and parameter count. The chosen factors in this snippet are for arithmetic, not a trained task.

import numpy as np
from oop_ml.numpy.modern import LowRankAdapter

adapter = LowRankAdapter(
    base=np.eye(4),
    down=np.array([[1.0], [0.0], [0.0], [0.0]]),
    up=np.array([[0.0, 0.5, 0.0, 0.0]]),
    alpha=1.0,
)
print(adapter.respond_to(np.array([[1.0, 0.0, 0.0, 0.0]])))
print(adapter.trainable_parameters)

The snippet builds the adapter traced in Part 2 and prints two things. The first is the adapted output [1, 0.5, 0, 0]. The second is the trainable count of 8, which is four numbers in each factor.

Change alpha to 2.0 and run it again. The output becomes [1, 1, 0, 0]. The correction has doubled from 0.5 to 1, it still lands on the second coordinate alone, and the count is still 8. Alpha sets how large the correction is along a direction the factors have already chosen. It adds nothing to what the adapter can express.

The live example runs the same operation on different chosen factors. They are drawn at random from a fixed seed, so the update has no convenient zeros, and the input is [1, 0, 0.5, −0.5]. The base is again the identity, so the change in each coordinate is the adapter’s contribution.

Coordinate	Input	Output at rank one	Change
1	1	1.0007	0.0007
2	0	−0.0355	−0.0355
3	0.5	0.4884	−0.0116
4	−0.5	−0.5039	−0.0039

The live example at rank one and an alpha of 1.

All four coordinates moved this time, and it is still one direction. The table of the assembled update in the live example shows why. Its second, third and fourth rows are the first row multiplied by −0.817, −0.536 and 1.074. Sixteen entries are on display, and eight numbers produced them. Raise the rank to two and the rows stop being multiples of one another, while the count reaches the 16 of a full update. At rank three it passes that with 24.

6. Choose and evaluate the adaptation

Nothing so far has been trained, so it is worth saying where this operation meets the previous lesson. Training runs the model forward with the adapted output y in place of the base output, computes the response loss, and the gradient of that loss travels back along both routes. On the base route it arrives at W, which is frozen, so nothing is written. On the adapter route it arrives at A and B, and those are updated.

A real setup chooses which projections receive adapters, a rank and scaling policy, the training examples, and the optimizer. The response loss can be the same supervised objective used for full fine-tuning. The difference is which parameters are allowed to change.

Each of those choices has a cost that the tiny example makes visible. Every projection given an adapter adds another pair of factors to train and store. A higher rank allows more directions and costs more parameters, and past the break-even point it costs more than the full update it was meant to replace. The scaling sets how large the correction is beside the frozen base.

LoRA is not quantization. Quantization changes how numerical values are represented or computed. The two techniques can be combined, but parameter rank and numeric precision solve different storage and computation problems.

Evaluate the adapted task and the behavior worth retaining. A smaller checkpoint is a resource property; it does not by itself prove that an adapter learned the intended behavior.

The paper proposes freezing pretrained weights and learning low-rank changes to selected model projections. LoRA: Low-Rank Adaptation of Large Language Models.

Continue with Reinforcement Learning.