QLoRA (Quantized LoRA)

LoRA slashed the optimizer memory — but you still had to load the whole base model in 16-bit. For a 70B model, that's ~140 GB before training even starts. QLoRA removes that last barrier: quantize the frozen base to 4-bit, then train LoRA adapters on top. The result was striking — a 65B model fine-tuned on a single 48 GB GPU, with quality matching 16-bit fine-tuning.

💡 In one line: QLoRA quantizes the frozen base model to 4-bit and trains LoRA adapters on top — fine-tuning huge models on a single GPU.

The Problem QLoRA Solves

With LoRA, the adapters are tiny — but the frozen base still sits in memory at 16-bit:

• 7B → ~14 GB just to hold the weights.
• 70B → ~140 GB — multiple A100s, before a single gradient.

Since the base is frozen anyway, why store it at full precision?

The Core Idea

Quantize the frozen base to 4-bit; keep the LoRA adapters in 16-bit.

• The base is compressed 4× (16-bit → 4-bit) and never updated.
• The adapters train in full precision, so learning quality holds.
• During compute, weights are dequantized on the fly, layer by layer.

The Flow

Whiteboard

Note what doesn't happen: no gradients ever reach the 4-bit base.

The Three Innovations

QLoRA is more than "LoRA but quantized" — it introduced three techniques:

• NF4 (4-bit NormalFloat) — a data type information-theoretically optimal for normally distributed weights (which neural network weights are). It preserves quality far better than naive int4.
• Double quantization — quantize the quantization constants too, saving a further ~0.37 bits per parameter (~3 GB on a 65B model).
• Paged optimizers — use NVIDIA unified memory to page optimizer states to CPU RAM during memory spikes, preventing OOM crashes on long sequences.

The Memory Numbers

Fine-tuning a 65B model:

For a 7B model, QLoRA fits comfortably on a free Colab T4 (~16 GB) — which is precisely why open-model fine-tuning exploded.

Does Quality Suffer?

Barely. The QLoRA paper showed 4-bit tuning matching 16-bit full fine-tuning performance — their Guanaco models reached ~99% of ChatGPT's level on the Vicuna benchmark, trained in 24 hours on one GPU.

The reason: quantization error lands on frozen weights the adapters then learn to compensate for. The adapters aren't quantized, so learning stays precise.

The Trade-off: Speed

QLoRA is slower than LoRA — roughly ~30–40% — because weights must be dequantized on every forward pass. You're trading compute time for memory. If the model fits with plain LoRA, use LoRA; if it doesn't, QLoRA makes it possible at all.

QLoRA vs. LoRA

Code Sketch

Then train as usual — only the adapters update.

Practical Notes

• Target all linear layers (not just attention) — the QLoRA paper found this matters more at 4-bit.
• Use bf16 compute dtype on modern GPUs.
• Merging an adapter back into a 4-bit base is lossy — dequantize first, or keep them separate.
• Tooling: bitsandbytes + Hugging Face PEFT, or Unsloth for a faster path.

Summary

• QLoRA = 4-bit frozen base + 16-bit LoRA adapters.
• Three innovations: NF4, double quantization, and paged optimizers.
• It cut 65B fine-tuning from >780 GB to <48 GB — one GPU.
• Quality is nearly identical to 16-bit; the cost is ~30–40% slower training.
• Use LoRA if it fits; use QLoRA when it doesn't.