Fine-Tuning Without Touching the Weights

You have a model whose behaviour you need to change on your task, and it has about three billion weights. A full fine-tune updates every one of them, which means every one needs its value, its gradient and whatever the optimiser keeps per weight, all resident on the GPU at once. A Llama 3.2 base of that size takes about 13 GB of GPU memory just to hold at 32-bit precision, before training starts. The card you actually have is the free one in a notebook. The full fine-tune does not fit, and it would not fit with twice the card either.

The way out is not to make the model smaller. It is to stop treating the model as the thing you edit. You leave every weight where it is and train a separate, much smaller object that is added to the model's output. That object is a diff, and this essay is about why a diff of a particular shape is so much cheaper than the weights it changes.

A weight update is a matrix, and a matrix can be factored

Take one weight matrix inside the model, call it W, with d_in columns and d_out rows. During a full fine-tune W becomes W + ΔW, where ΔW has exactly the same shape as W. That is where the cost comes from: the update has as many entries as the weight.

Low-rank adaptation, LoRA, replaces ΔW with a product of two thin matrices. A has r rows and d_in columns; B has d_out rows and r columns. Their product B·A has the shape of W, so it can be added to W, but its entries are not free: they are generated from (d_in + d_out) × r numbers rather than d_in × d_out. The number r is the adapter's rank, and it is chosen by you. The usual starting values are 8, 16 and 32.

Here is the whole mechanism as a forward pass. For an input vector x:

h = W·x + s · B·(A·x)

Read it term by term. x is the layer's input, a vector of d_in numbers. W·x is what the layer computed before any fine-tuning: the frozen weights applied to the input, giving d_out numbers. A·x squeezes the same input down to r numbers, and B·(A·x) stretches those r numbers back out to d_out, so the second term has the same shape as the first and can be added to it. s is a fixed scale, discussed below. h is the layer's output, the base model's answer plus a correction that passes through an r-wide bottleneck.

Four things follow from that one line.

Only A and B receive gradients. W is marked frozen, so the backward pass computes how the loss changes with A and with B and never allocates a gradient for W. The optimiser state, which for the Adam family is a running record per trainable parameter, is kept only for A and B too. The memory that scales with what you train, the gradients and the optimiser state, is now proportional to the diff rather than to the model. The frozen base still has to be resident, and the activations saved for the backward pass still grow with batch size and sequence length. LoRA removes the first cost and leaves the other two where they were.

Training starts exactly at the base model. The LoRA paper initialises A with random Gaussian entries and B with zeros, so B·A is the zero matrix at the first step and the first forward pass is the untouched model. Nothing about the base's behaviour changes until a non-zero gradient has moved B.

The scale s is set by a second hyperparameter, alpha. Every LoRA configuration carries two numbers: r, the rank you have just met, and alpha. Alpha does not appear in the formula as itself. The scale that multiplies the correction is alpha divided by r, so the same alpha means a smaller scale at a larger rank. The reason for the division is that B·(A·x) is a sum of r terms, one per column of B, and would tend to grow as r grows; dividing by r keeps the correction's size roughly independent of the rank, so that a learning rate tuned at one r still works at another. Two consequences are worth keeping in mind when you copy settings from somewhere else. With alpha = 16, an adapter of rank 8 is applied at scale 16/8 = 2, and the same adapter at rank 64 at scale 16/64 = 0.25, an eightfold difference from one changed number. And the common rule of thumb alpha = 2r keeps the scale at exactly 2 whatever r you pick: 64/32 and 512/256 both give 2. Descriptions that call alpha "a multiplier" are shorthand for all of this.

The diff has rank at most r. Write B·A out and it is the sum of r outer products: the first column of B times the first row of A, plus the second column times the second row, and so on. Each outer product is a rank-1 matrix, so their sum can express at most r independent directions of change. This is the bet LoRA makes: that the change your task needs is closer to a few directions than to an arbitrary matrix.

Worked example: a diff of rank 1 you can write out, then the real sizes

The numbers below are the book's own and the models are invented.

Start small enough to write every entry. Take a layer with four inputs and four outputs, so W is 4 × 4 with 16 frozen entries, and attach an adapter of rank r = 1. A is then 1 × 4 and B is 4 × 1. Give them values:

A = [1, 0, −1, 2],    B = [1, 2, 0, 1]^T

That is eight trainable numbers. Multiplying them gives the diff B·A, a 4 × 4 matrix whose row i is B_i times the row A:

Sixteen entries, produced from eight numbers, and every row is a multiple of the same row A. That is what "rank 1" means in practice: the diff can scale one pattern up or down per output, and nothing else. A rank-2 adapter would add a second row pattern with its own column of multipliers, and so on up to r.

Now run an input through it, to see the order of operations in the formula. Take x = (1, 1, 1, 1). A·x is one number: 1 + 0 − 1 + 2 = 2. B·(A·x) is B scaled by that number: (2, 4, 0, 2). Check it against the long way round, (B·A)·x, which sums each row of the table above: 2, 4, 0, 2. The same answer, but the short way never built the 4 × 4 matrix; it passed the input through a single number in the middle. That bottleneck is the adapter. The layer's output is then W·x, whatever the frozen weights give, plus s times (2, 4, 0, 2).

Now the real sizes, with the same arithmetic. Suppose one attention projection maps 512 inputs to 512 outputs and you attach an adapter of rank 8.

Scroll the table sideways to see all columns.

The adapter trains 8,192 numbers and can produce a 262,144-entry change: 8,192 / 262,144 = 3.125% of the entries it stands in for. The general count is (d_in + d_out) × r against d_in × d_out, and the saving grows with the layer's width because the first is linear in the width and the second is quadratic.

Scale up once more to a small model with twelve layers, each with four such projections: 48 matrices. The frozen weights in those projections are 48 × 262,144 = 12,582,912. The adapters are 48 × 8,192 = 393,216. Stored at four bytes each, the adapter file is 393,216 × 4 = 1,572,864 bytes, about 1.57 MB, and it is shipped as its own file. Merge it into W when you want a single artefact for serving, or keep it beside the base and swap it for another task's adapter. Either way the base model is never patched in place.

Checking the formula on a real model

Now a real model: Llama 3.2 with about three billion weights, 28 decoder layers and an inner width of 3072. A common adapter configuration puts rank-32 adapters on the four attention projections of every layer, called q, k, v and o, and the adapter file that results is 73.4 MB on disk. Does the formula give that?

Two of the projections keep the full width; the other two, k and v, are narrower on the output side, at 1024. Per layer:

Scroll the table sideways to see all columns.

Over 28 layers that is 655,360 × 28 = 18,350,080 parameters, and at four bytes each, 73,400,320 bytes: 73.4 MB, the file size. (If the narrow width were a round 1,000 instead of 1024 the total would be 73.2 MB, so the file size itself pins the width; it is one of the few things about a model you can check from the outside.)

A heavier configuration on the same model, rank 256 with the three MLP matrices added as targets, gives a 1.56 GB adapter. The MLP matrices widen 3072 out to 8192 and back. Same table, more rows:

Scroll the table sideways to see all columns.

Both files come out of one rule: for each target matrix, add its two widths and multiply by r; sum over the targets in a layer; multiply by the number of layers; multiply by four bytes. That is the whole cost model for an adapter, and you can run it on any configuration you are handed before you train anything.

What the rank cannot do

Rank is a ceiling, not a promise. If the change your task needs is spread across more independent directions than r, the adapter fits what it can and leaves the rest, and no number of training steps recovers it. Raising r raises the ceiling and the file size together, which is the trade behind the two configurations above: the heavier one was chosen when the training set grew from 20,000 rows to 800,000. There is no formula for the right r; target modules, r and alpha are found by trial against your evaluation metric.

A diff of rank r is not a full fine-tune with fewer parameters. It is a different function class. Two runs with different r or different alpha are two different experiments, and the alpha convention above means a setting copied from one codebase may not mean the same thing in another.

Finally, LoRA is only half of QLoRA. The other half, holding the frozen base in four-bit precision while the adapters stay in full precision, is why a 2.2 GB base plus a 73 MB adapter fits on the free card. How those four bits are chosen is the subject of a separate essay in Part IV. What matters here is that the quantisation is applied to the base and never to the diff.

import torch
import torch.nn as nn

torch.manual_seed(0)
d_in, d_out, r, alpha = 64, 64, 4, 8


class LoRALinear(nn.Module):
    """A frozen nn.Linear with a trainable low-rank diff beside it."""

    def __init__(self, base: nn.Linear, r: int, alpha: float):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                                   # freeze W (and its bias)
        self.A = nn.Parameter(0.01 * torch.randn(r, base.in_features))  # r x d_in, random start
        self.B = nn.Parameter(torch.zeros(base.out_features, r))        # d_out x r, zero start
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)  # W·x + s · B·(A·x)

    def merge(self):
        with torch.no_grad():
            self.base.weight += self.scale * (self.B @ self.A)        # fold the diff into W
        return self.base


base = nn.Linear(d_in, d_out)
layer = LoRALinear(base, r, alpha)

trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in layer.parameters() if not p.requires_grad)
print(f"trainable {trainable}, frozen {frozen}")

# The behaviour we want: the base layer plus a rank-1 change. The adapter has to learn the change.
u, v = torch.randn(d_out, 1), torch.randn(1, d_in)
with torch.no_grad():
    W_target = base.weight + u @ v
target = nn.Linear(d_in, d_out)
with torch.no_grad():
    target.weight.copy_(W_target)
    target.bias.copy_(base.bias)

opt = torch.optim.Adam([p for p in layer.parameters() if p.requires_grad], lr=1e-2)
for step in range(1, 501):
    x = torch.randn(32, d_in)
    loss = ((layer(x) - target(x)) ** 2).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    if step in (1, 10, 100, 500):
        print(f"step {step:3d}: loss {loss.item():.4f}   W.grad is None: {base.weight.grad is None}")

adapter = {k: v for k, v in layer.state_dict().items() if not k.startswith("base.")}
print("adapter file holds:", {k: tuple(v.shape) for k, v in adapter.items()},
      "=", sum(v.numel() for v in adapter.values()) * 4, "bytes")

x = torch.randn(8, d_in)
with torch.no_grad():
    before = layer(x)
    merged = layer.merge()
    print(f"max difference between adapter and merged outputs: {(merged(x) - before).abs().max().item():.2e}")

# illustrative, not executed; API as of the course's library versions
from peft import LoraConfig, get_peft_model

config = LoraConfig(r=32, lora_alpha=64, lora_dropout=0.1,
                    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
model = get_peft_model(base_model, config)
model.print_trainable_parameters()