LoRA (Low-Rank Adaptation)

Full fine-tuning a 7B model needs ~90 GB of memory — most of it gradients and optimizer states for weights you barely change. LoRA sidesteps all of it: freeze the original model and train two tiny matrices alongside it. You update ~0.1–1% of the parameters, use a fraction of the memory, and get comparable quality. It's why fine-tuning went from a big-lab luxury to something you can run on a single GPU.

💡 In one line: LoRA freezes the base model and trains small low-rank matrices instead — ~1% of the parameters, most of the quality.

The Core Insight

Fine-tuning learns a weight update ΔW for each big weight matrix W. LoRA's key observation: that update has low "intrinsic rank" — it doesn't need to be a full-sized matrix. So you can approximate it as the product of two skinny matrices:

ΔW ≈ B × A, where A and B are tiny.

Then: W_new = W_frozen + B×A — the original W never changes.

Why the Savings Are So Large

Take a 4096 × 4096 weight matrix:

Fewer trainable parameters means no optimizer states for the frozen weights — which is where full fine-tuning's memory actually goes.

The Workflow

• r (rank) — the size of the bottleneck. 8–16 is typical; higher = more capacity + more parameters.
• alpha — a scaling factor. A common convention is alpha = 2r.
• target_modules — which layers get adapters. Attention projections (q_proj, v_proj) are the classic choice; all linear layers gives more capacity.
• dropout — regularisation.

Rule of thumb: start at r=8, alpha=16; raise r only if the model underfits.

Adapters: Small and Swappable

A LoRA adapter is a few MB versus a multi-GB full model copy. That changes the deployment picture:

• One base model, many adapters — one per task or customer.
• Swap adapters at runtime without reloading the base.
• Ship and version adapters trivially.

Merge or Keep Separate?

• Keep separate — swap adapters dynamically; adds a tiny inference overhead.
• Merge (W + BA folded into W) — zero inference overhead, but you're back to a full model copy and lose swappability.

LoRA vs. Full Fine-Tuning

Why Forgetting Is Lower

The base weights never move — so the model's general knowledge is structurally protected. Your task-specific behaviour lives in the adapter, and you can always remove it to get the original model back.

Limitations

• Very large domain shifts may still want full fine-tuning.
• Rank r caps capacity — too low and it underfits.
• Unmerged adapters add a small latency cost.

Practical Notes

Use Hugging Face PEFT (LoraConfig, get_peft_model), or Axolotl / Unsloth for a faster path. LoRA is the default fine-tuning method in 2026 — reach for full FT only when you've proven you need it.

Summary

• LoRA freezes the base model and trains two small matrices (ΔW ≈ B×A).
• It trains ~0.1–1% of parameters — a ~99%+ reduction — with comparable quality.
• Key knobs: r (rank), alpha, and target_modules; start at r=8, alpha=16.
• Adapters are a few MB and swappable — one base, many tasks.
• Forgetting is lower because the base is frozen — and LoRA is now the default method. EOF echo created