Full Fine-Tuning
Full fine-tuning is the original, maximal form of tuning: update every weight in the model on your data. It gives the most adaptation possible — and costs the most in GPU memory, time, and risk. It's the baseline every other method (LoRA, QLoRA, PEFT) is measured against, and understanding why it's expensive is what explains all of them.
💡 In one line: Full fine-tuning updates all of a model's weights on your data — maximum adaptation, maximum cost.
What is Full Fine-Tuning?
You take a pretrained model, train it on your dataset, and update every parameter via backpropagation. Nothing is frozen — 100% of the weights change.
The Training Loop
Standard supervised learning.Â
Repeat for a few epochs, watching the validation loss.
Why It's So Expensive
The killer isn't the weights — it's everything else you must hold in memory:
| What | Memory (7B model, fp16) |
|---|---|
| Weights | ~14 GB |
| Gradients | ~14 GB |
| Optimizer states (Adam: momentum + variance) | ~56 GB |
| Activations | several GB |
| Total | ~90 GB+ |
That's multiple high-end GPUs for a 7B model — and a 70B model is out of reach for most teams. This is exactly the problem LoRA and QLoRA solve.
Hyperparameters That Matter
- Learning rate — small (≈ 1e-5 to 5e-5). Too high destroys pretrained knowledge.
- Epochs — usually 1–3; more invites overfitting.
- Batch size — limited by memory (use gradient accumulation).
- Warmup + decay — stabilise early training.
Catastrophic Forgetting (the Big Risk)
Because every weight moves, the model can lose general abilities while learning your task — it gets better at your data and worse at everything else. Mitigate with a low learning rate, fewer epochs, mixing in general data, or by using PEFT instead.
Full Fine-Tuning vs. LoRA
| Full FT | LoRA / PEFT | |
|---|---|---|
| Weights updated | 100% | ~0.1–1% |
| Memory | Very high | Low |
| Quality | Best possible | Usually comparable |
| Forgetting | Higher risk | Lower |
| Artifacts | A full model copy per task | A small adapter per task |
Because LoRA gets most of the quality for a fraction of the cost, full fine-tuning is now the exception, not the default.
When It's Actually Worth It
- You need the absolute best quality and have the compute.
- A large domain shift (a very different language, or highly specialised text).
- Lots of data (tens of thousands of examples).
- Small models (under ~1B), where the cost is manageable.
Otherwise: use LoRA/QLoRA.
Practical Notes
- Checkpoint often; watch validation loss for overfitting.
- You get a full model copy per task — storage and deployment add up.
- Frameworks: Hugging Face Transformers
Trainer, Axolotl, DeepSpeed/FSDP for multi-GPU.
Summary
- Full fine-tuning updates every weight — maximum adaptation.
- Memory is dominated by gradients and optimizer states, not just weights (~90 GB+ for 7B).
- Use a low learning rate and few epochs to limit catastrophic forgetting.
- LoRA/PEFT usually gets comparable quality at a fraction of the cost.
- Reserve full FT for big domain shifts, lots of data, small models, or when quality is everything.Â