Adapters

Before LoRA, there were adapters — the method that founded PEFT in 2019. The idea: leave the pretrained model frozen and insert small trainable modules between its layers. Train ~1–3% of the parameters, get ~96% of full fine-tuning's quality. LoRA later refined the approach, but adapters introduced the principle every PEFT method still follows: freeze the model, train something small.

💡 In one line: Adapters are small trainable modules inserted into a frozen model's layers — the original parameter-efficient fine-tuning method.

What Are Adapters?

An adapter is a small neural module inserted inside each transformer block. The base model is frozen; only the adapter layers train. Introduced by Houlsby et al. (2019), they showed you could match full fine-tuning while training a tiny fraction of the weights.

The Bottleneck Architecture

Each adapter is deliberately tiny, using a down-up structure:

1. Down-project — d dimensions → m (where m ≪ d).
2. Non-linearity — usually ReLU or GeLU.
3. Up-project — m → d dimensions.
4. Residual connection — add back to the input.

That bottleneck (m) is what keeps the parameter count small — it's the adapter's equivalent of LoRA's rank r.

Near-identity initialisation matters: the adapter starts as roughly a no-op, so inserting it doesn't damage the pretrained model — it learns its contribution from there.

Adapters vs. LoRA: Sequential vs. Parallel

This is the key architectural distinction:

• Adapters are sequential — they sit in the data path. Input flows through them, which adds inference latency.
• LoRA is parallel — it runs alongside the frozen weights and can be merged in, giving zero added latency.

That mergeability is why LoRA won as the default — but adapters remain the conceptual foundation.

The PEFT Family

"Adapter" is also used broadly for the whole family of insert-something-small methods:

They share one principle: freeze the base, train a small addition.

Choosing a PEFT Method

Composability: the Adapter Advantage

Because adapters are discrete modules, they can be combined:

• Stack them — several adapters in sequence.
• Fuse them — AdapterFusion learns to combine knowledge from multiple task adapters.
• Route them — pick an adapter per input.

This modularity remains adapters' strongest argument, and it inspired multi-LoRA serving (one base, many adapters swapped at runtime).

Why Adapters Still Matter

• They founded PEFT — the principle behind LoRA, QLoRA, and the rest.
• They're strongly modular and composable.
• They're well-studied for multi-task and cross-lingual transfer.

In practice, though, LoRA is the default in 2026 — mergeability and simplicity won.

Trade-offs

• Inference latency — they're in the data path and can't merge.
• More parameters than LoRA for similar quality.
• Architecture changes — modules must be inserted into the model.
• Bottleneck size trades capacity against cost.

Tooling

Hugging Face PEFT covers LoRA, prefix/prompt tuning, and IA³. AdapterHub / adapters is the dedicated library for classic bottleneck adapters, AdapterFusion, and adapter composition.

Summary

• Adapters insert small trainable modules into a frozen model — the original PEFT method.
• Their bottleneck (down → non-linear → up → residual) keeps parameters at ~1–3%.
• They're sequential (add latency, can't merge); LoRA is parallel (mergeable, zero overhead).
• The PEFT family — LoRA, prefix/prompt tuning, IA³, BitFit — all share: freeze the base, train a small addition.
• Adapters excel at modularity and composition (AdapterFusion), but LoRA is the practical default.