Introduction
Multi-head attention is the technique that runs several independent self-attention computations in parallel within a single transformer layer, each using its own separately learned Query, Key, and Value projections, allowing the model to capture multiple different types of relationships between tokens simultaneously rather than being limited to just one. Introduced alongside the original transformer architecture, multi-head attention is what gives self-attention much of its real expressive power in practice.
While the earlier Self-Attention and Query, Key, Value topics touched on multi-head attention briefly, this topic examines it in full detail — exactly how the splitting and recombining works, why multiple heads outperform a single large attention computation, and what different heads actually tend to learn.
Why Does Multi-Head Attention Matter?
Multi-head attention helps to:
- Capture multiple distinct types of relationships between tokens within the same layer
- Give the model more expressive representational capacity than a single attention computation
- Allow different heads to specialize in different patterns (e.g., syntax, long-range references, local context)
- Maintain full parallelizability while adding this richer representational power
- Provide a key architectural lever that contributes meaningfully to transformer capability
- Serve as a foundational component reused consistently across virtually every transformer variant
The Core Idea: Splitting Into Multiple Heads\
How Splitting Actually Works
Suppose a model uses:
- Embedding dimension: 768
- Number of attention heads: 12
Rather than computing one large 768-dimensional attention
calculation, the 768-dimensional Q, K, and V vectors are
each split into 12 smaller chunks of 64 dimensions:
768 / 12 heads = 64 dimensions per head
Each head then performs its own independent scaled
dot-product attention calculation using just its own
64-dimensional slice of Q, K, and V.Why Split Instead of Using One Large Attention Computation?
Using a single, full-sized attention computation forces the
model to blend all types of relevant relationships (grammar,
meaning, position, reference, etc.) into one averaged set of
attention weights per token pair.
By splitting into multiple smaller, independent heads, each
head is free to specialize — learning to focus on a different
kind of pattern without being forced to compromise or average
with what the other heads are learning.
This is analogous to having several different "experts" each
examine the same sentence from a different angle, rather than
one generalist trying to consider everything at once.Recombining the Heads
After each head produces its own output (a weighted combination
of Values, based on that head's own attention pattern), all
heads' outputs are concatenated back together into a single
vector of the original full dimension:
Head 1 output (64-dim) + Head 2 output (64-dim) + ...
+ Head 12 output (64-dim) = concatenated 768-dim vector
This concatenated vector is then passed through one final
learned linear layer, allowing the model to combine and
weight the different heads' contributions into a single,
unified representation.What Different Heads Tend to Learn (Illustrative Patterns)
| Head Type (Illustrative) | Tendency Observed in Research |
|---|---|
| Local/Syntactic Heads | Focus on nearby words, grammatical structure (e.g., adjective-noun pairs) |
| Positional Heads | Attend consistently to tokens at a fixed relative distance |
| Reference-Resolution Heads | Connect pronouns or references back to the nouns they refer to |
| Rare/Specialized Heads | Focus on unusual or task-specific patterns discovered during training |
(Note: interpretability research shows these patterns exist, but not every head learns a clean, easily-labeled pattern — some heads capture less interpretable or overlapping behavior.)
Single-Head vs Multi-Head Attention
| Aspect | Single-Head Attention | Multi-Head Attention |
|---|---|---|
| Relationship Types Captured | One averaged/blended pattern | Multiple, potentially specialized patterns simultaneously |
| Representational Flexibility | Lower | Higher |
| Computational Cost | Roughly comparable total cost when matched to equivalent total dimension | Distributed across smaller, parallel computations |
| Standard Practice in Modern Transformers | Rarely used alone | Nearly universal default choice |
Multi-Head Attention Across Model Types
| Model Type | Where Multi-Head Attention Applies |
|---|---|
| Encoder-Only (e.g., BERT) | Multi-head self-attention within the encoder |
| Decoder-Only (e.g., GPT, Claude, Llama) | Multi-head masked self-attention within the decoder |
| Encoder-Decoder (e.g., T5) | Multi-head self-attention in both encoder and decoder, plus multi-head cross-attention connecting them |
Key Properties of Multi-Head Attention
- Multi-head attention splits Q, K, and V into smaller sections, running independent attention computations per head.
- Each head learns its own separate projections, allowing it to potentially specialize in different relationship types.
- All heads' outputs are concatenated and passed through a final linear layer to produce a unified representation.
- Multi-head attention maintains full parallelizability while significantly increasing representational flexibility.
- This technique applies consistently across encoder-only, decoder-only, and encoder-decoder transformer designs.
Where Does Multi-Head Attention's Design Matter Most?
| Context | Relevance |
|---|---|
| Model Architecture Design | Number of heads is a key hyperparameter balancing capacity and efficiency |
| Interpretability Research | Studying individual heads to understand what patterns a model has learned |
| Efficient Attention Research | Techniques like grouped-query attention modify how heads share Key/Value computations |
| Model Scaling Decisions | Head count typically scales alongside overall model size and embedding dimension |
| Debugging Model Behavior | Attention head analysis can help explain specific model outputs |
Advantages
- Captures multiple distinct relationship types simultaneously within a single layer
- Provides significantly more representational flexibility than a single attention computation
- Remains fully parallelizable across heads, preserving transformer training efficiency
- Well-validated technique used consistently across virtually all major transformer architectures
- Enables valuable interpretability research into what individual heads have learned
Limitations
- Not every head learns a clean, easily interpretable pattern — some overlap or redundancy is common
- Adds architectural complexity (splitting and recombining) compared to a single attention computation
- Choosing an optimal number of heads involves real tradeoffs and typically requires experimentation
- More heads doesn't automatically guarantee proportionally better model performance
- Full multi-head attention still contributes to the overall quadratic computational cost of standard self-attention
Real-World Examples
| Application | Multi-Head Attention Relevance |
|---|---|
| GPT/Claude/Llama Architecture | Multi-head masked self-attention within every decoder layer |
| BERT Architecture | Multi-head bidirectional self-attention within every encoder layer |
| T5 Architecture | Multi-head self-attention plus multi-head cross-attention |
| Attention Visualization Research | Tools that display and analyze individual heads' learned patterns |
| Efficient Attention Variants (e.g., Grouped-Query Attention) | Modify standard multi-head attention for improved inference efficiency |
Best Practices
- Understand multi-head attention as enabling specialization, not simply adding raw computational power.
- Study the split-then-concatenate structure to understand exactly how head outputs combine.
- Consider head count as a meaningful architectural hyperparameter when reasoning about model design.
- Explore attention visualization tools if you want to directly observe learned head-level patterns.
- Recognize this technique's consistent presence across encoder-only, decoder-only, and encoder-decoder designs.
Interview Tip
A common interview question is:
"What is multi-head attention, and why is it more effective than using a single, full-sized attention computation?"
A strong answer is:
Multi-head attention splits the Query, Key, and Value vectors into several smaller sections, each processed by an independent attention "head" with its own learned projections, allowing different heads to specialize in different types of relationships — like grammar, positional patterns, or long-range references — rather than forcing one single attention computation to blend everything into a single averaged pattern. After each head computes its own output, they're concatenated together and passed through a final linear layer, combining these multiple specialized perspectives into one richer, unified representation, all while remaining fully parallelizable.
Explaining the specialization benefit, not just the mechanical splitting, makes your answer stronger.
Conclusion
Multi-head attention gives transformers the ability to capture multiple distinct types of relationships between tokens simultaneously, splitting self-attention into several specialized, independently learned computations that are recombined into a single, richer representation. Building directly on the Self-Attention and Query, Key, Value foundations, this mechanism is a key reason transformer-based models — from BERT to GPT to T5 — are able to represent language with the nuance and flexibility that powers their real-world capabilities.