Introduction
A residual connection — also known as a "skip connection" — is a simple but powerful architectural technique where a sub-layer's input is added directly to its output, allowing information (and gradients) to flow around that sub-layer as well as through it. In a transformer, residual connections wrap around both the self-attention and feed forward network sub-layers within every block, playing a critical role in making it possible to successfully train the very deep stacks of layers that modern LLMs rely on.
Without residual connections, training deep neural networks — including transformers with dozens or even over a hundred stacked layers — would be dramatically more difficult, since gradients tend to weaken as they propagate backward through many layers, echoing the vanishing gradient problem discussed earlier in the context of RNNs.
Why Do Residual Connections Matter?
Residual connections help to:
- Enable stable training of very deep networks, including transformers with many stacked layers
- Prevent gradients from vanishing as they propagate backward through many layers
- Preserve the original input information alongside each sub-layer's transformation
- Make it easier for each sub-layer to learn a smaller, incremental adjustment rather than a full transformation
- Support the overall depth and scale that gives modern LLMs much of their capability
- Work closely alongside layer normalization to stabilize transformer training
Where Residual Connections Fit in a Transformer Block
The Core Idea: Learning a Residual, Not a Full Transformation
Standard layer (without residual connection):
output = SubLayer(input)
Layer with a residual connection:
output = input + SubLayer(input)
Rather than forcing each sub-layer to learn a complete
transformation from scratch, the residual connection lets
it instead learn only the "residual" — the difference or
adjustment needed relative to the original input.
This is often a much easier optimization problem, especially
early in training, since a sub-layer can start close to
simply passing its input through unchanged and gradually
learn more meaningful adjustments over time.Why Residual Connections Solve the Vanishing Gradient Problem
During backpropagation, gradients must flow backward through
every layer of a deep network. In a standard deep stack, this
flow can shrink dramatically layer by layer, making it hard
for early layers to receive a meaningful training signal.
With a residual connection, the addition operation creates a
direct path for the gradient to flow backward, essentially
"skipping over" the sub-layer's transformation. This means
even if a sub-layer's own gradient signal weakens, the
residual path preserves a stronger, more direct gradient flow
all the way back to earlier layers.A Simple Illustration
Without residual connections:
Layer 1 output → Layer 2 output → Layer 3 output → ...
(each layer must fully "carry forward" all prior information)
With residual connections:
Layer 1 output = input + Layer1_transformation(input)
Layer 2 output = Layer1_output + Layer2_transformation(Layer1_output)
...
The original signal from early in the network remains directly
accessible throughout, rather than depending entirely on being
successfully passed through every intermediate transformation.Residual Connections in Both Transformer Sub-Layers
| Sub-Layer | Residual Connection Applied |
|---|---|
| Self-Attention | output = input + SelfAttention(input) |
| Feed Forward Network | output = input + FFN(input) |
Both major sub-layers within every transformer block are wrapped with their own residual connection, meaning a typical transformer block actually contains two residual connections — one around attention, one around the FFN.
Residual Connections vs No Residual Connections
| Aspect | With Residual Connections | Without Residual Connections |
|---|---|---|
| Gradient Flow in Deep Networks | Preserved via direct skip paths | Prone to vanishing as depth increases |
| Training Stability | Significantly more stable | Increasingly difficult as networks get deeper |
| Practical Network Depth Achievable | Enables very deep architectures (dozens+ of layers) | Limits practical depth before training breaks down |
| Optimization Difficulty | Sub-layers learn smaller, incremental adjustments | Sub-layers must learn full transformations from scratch |
Key Properties of Residual Connections
- A residual connection adds a sub-layer's input directly to its output, rather than replacing it entirely.
- This allows sub-layers to learn smaller, incremental adjustments rather than complete transformations.
- Residual connections create a direct gradient pathway, helping prevent vanishing gradients in deep networks.
- Every transformer block contains two residual connections — one around self-attention, one around the FFN.
- Residual connections work closely with layer normalization to stabilize training in deep transformer stacks.
Where Do Residual Connections Matter Most?
| Context | Why Residual Connections Matter |
|---|---|
| Very Deep Transformer Models | Essential for successfully training models with many stacked layers |
| Large-Scale LLM Training | Supports the depth and scale needed for strong model capability |
| Computer Vision (ResNets) | The same core concept, originally popularized in deep CNN architectures |
| Model Architecture Research | A foundational technique widely reused across many deep learning architectures |
| Training Stability Troubleshooting | Understanding residual paths helps diagnose deep network training issues |
Advantages
- Makes training very deep networks practically feasible, which wouldn't otherwise be possible
- Directly addresses the vanishing gradient problem through preserved gradient pathways
- Simplifies the optimization problem for each individual sub-layer
- Simple, computationally cheap technique (just an addition operation) with an outsized training benefit
- Widely validated technique, used far beyond transformers across many deep learning architectures
Limitations
- Doesn't eliminate all training challenges in extremely deep or large-scale networks on its own
- Requires careful combination with other techniques (like layer normalization) for full effectiveness
- Adds a small amount of additional computation, though generally negligible compared to its benefit
- Doesn't inherently improve model capability by itself — it primarily enables successful training at depth
- The optimal placement of residual connections relative to normalization is still an active area of architectural research
Real-World Examples
| Context | Residual Connection Relevance |
|---|---|
| GPT/Claude/Llama Architecture | Present in every transformer block throughout these models |
| ResNet (Computer Vision) | The architecture that first popularized residual connections for deep CNNs |
| Very Large-Scale LLM Training | A foundational enabler of successfully training models with many dozens of layers |
| Transformer Architecture Research | Frequently studied and refined component in newer architectural variants |
| Deep Learning Education | A commonly taught, foundational deep learning concept |
Best Practices
- Understand residual connections as enabling sub-layers to learn adjustments, not full replacements.
- Recognize their critical role in making very deep transformer architectures trainable at all.
- Study how residual connections and layer normalization work together, rather than in isolation.
- Appreciate that this simple addition operation has an outsized impact on training stability.
- Connect this concept back to the vanishing gradient problem covered earlier in RNN/LSTM limitations.
Interview Tip
A common interview question is:
"What is a residual connection, and why is it important for training deep transformer models?"
A strong answer is:
A residual connection adds a sub-layer's input directly to its output, so instead of learning a full transformation from scratch, the sub-layer only needs to learn the residual — the adjustment relative to the original input. This is important for training deep transformers because it creates a direct pathway for gradients to flow backward during backpropagation, preventing the vanishing gradient problem that would otherwise make it very difficult to train networks with many stacked layers, which is exactly what modern large language models require given their significant depth.
Explicitly connecting residual connections back to the vanishing gradient problem makes your answer stronger.
Conclusion
Residual connections provide a simple yet essential mechanism for enabling stable training of the very deep transformer stacks that modern large language models depend on, preserving gradient flow and letting each sub-layer learn incremental adjustments rather than complete transformations. With this concept in place, the final topic in this section — layer normalization — explores the other key stabilization technique that works alongside residual connections within every transformer block.