Introduction

A residual connection — also known as a "skip connection" — is a simple but powerful architectural technique where a sub-layer's input is added directly to its output, allowing information (and gradients) to flow around that sub-layer as well as through it. In a transformer, residual connections wrap around both the self-attention and feed forward network sub-layers within every block, playing a critical role in making it possible to successfully train the very deep stacks of layers that modern LLMs rely on.

Without residual connections, training deep neural networks — including transformers with dozens or even over a hundred stacked layers — would be dramatically more difficult, since gradients tend to weaken as they propagate backward through many layers, echoing the vanishing gradient problem discussed earlier in the context of RNNs.

Why Do Residual Connections Matter?

Residual connections help to:

• Enable stable training of very deep networks, including transformers with many stacked layers
• Prevent gradients from vanishing as they propagate backward through many layers
• Preserve the original input information alongside each sub-layer's transformation
• Make it easier for each sub-layer to learn a smaller, incremental adjustment rather than a full transformation
• Support the overall depth and scale that gives modern LLMs much of their capability
• Work closely alongside layer normalization to stabilize transformer training

Where Residual Connections Fit in a Transformer Block

Whiteboard

Loading diagram...

The Core Idea: Learning a Residual, Not a Full Transformation

Standard layer (without residual connection):
output = SubLayer(input)

Layer with a residual connection:
output = input + SubLayer(input)

Rather than forcing each sub-layer to learn a complete
transformation from scratch, the residual connection lets
it instead learn only the "residual" — the difference or
adjustment needed relative to the original input.

This is often a much easier optimization problem, especially
early in training, since a sub-layer can start close to
simply passing its input through unchanged and gradually
learn more meaningful adjustments over time.

Why Residual Connections Solve the Vanishing Gradient Problem

During backpropagation, gradients must flow backward through
every layer of a deep network. In a standard deep stack, this
flow can shrink dramatically layer by layer, making it hard
for early layers to receive a meaningful training signal.

With a residual connection, the addition operation creates a
direct path for the gradient to flow backward, essentially
"skipping over" the sub-layer's transformation. This means
even if a sub-layer's own gradient signal weakens, the
residual path preserves a stronger, more direct gradient flow
all the way back to earlier layers.

A Simple Illustration

Without residual connections:
Layer 1 output → Layer 2 output → Layer 3 output → ...
(each layer must fully "carry forward" all prior information)

With residual connections:
Layer 1 output = input + Layer1_transformation(input)
Layer 2 output = Layer1_output + Layer2_transformation(Layer1_output)
...

The original signal from early in the network remains directly
accessible throughout, rather than depending entirely on being
successfully passed through every intermediate transformation.

Residual Connections in Both Transformer Sub-Layers

Both major sub-layers within every transformer block are wrapped with their own residual connection, meaning a typical transformer block actually contains two residual connections — one around attention, one around the FFN.

Residual Connections vs No Residual Connections

Key Properties of Residual Connections

• A residual connection adds a sub-layer's input directly to its output, rather than replacing it entirely.
• This allows sub-layers to learn smaller, incremental adjustments rather than complete transformations.
• Residual connections create a direct gradient pathway, helping prevent vanishing gradients in deep networks.
• Every transformer block contains two residual connections — one around self-attention, one around the FFN.
• Residual connections work closely with layer normalization to stabilize training in deep transformer stacks.

Where Do Residual Connections Matter Most?

Advantages

• Makes training very deep networks practically feasible, which wouldn't otherwise be possible
• Directly addresses the vanishing gradient problem through preserved gradient pathways
• Simplifies the optimization problem for each individual sub-layer
• Simple, computationally cheap technique (just an addition operation) with an outsized training benefit
• Widely validated technique, used far beyond transformers across many deep learning architectures

Limitations

• Doesn't eliminate all training challenges in extremely deep or large-scale networks on its own
• Requires careful combination with other techniques (like layer normalization) for full effectiveness
• Adds a small amount of additional computation, though generally negligible compared to its benefit
• Doesn't inherently improve model capability by itself — it primarily enables successful training at depth
• The optimal placement of residual connections relative to normalization is still an active area of architectural research

Real-World Examples

Best Practices

• Understand residual connections as enabling sub-layers to learn adjustments, not full replacements.
• Recognize their critical role in making very deep transformer architectures trainable at all.
• Study how residual connections and layer normalization work together, rather than in isolation.
• Appreciate that this simple addition operation has an outsized impact on training stability.
• Connect this concept back to the vanishing gradient problem covered earlier in RNN/LSTM limitations.

Interview Tip

A common interview question is:

"What is a residual connection, and why is it important for training deep transformer models?"

A strong answer is:

A residual connection adds a sub-layer's input directly to its output, so instead of learning a full transformation from scratch, the sub-layer only needs to learn the residual — the adjustment relative to the original input. This is important for training deep transformers because it creates a direct pathway for gradients to flow backward during backpropagation, preventing the vanishing gradient problem that would otherwise make it very difficult to train networks with many stacked layers, which is exactly what modern large language models require given their significant depth.

Explicitly connecting residual connections back to the vanishing gradient problem makes your answer stronger.

Conclusion

Residual connections provide a simple yet essential mechanism for enabling stable training of the very deep transformer stacks that modern large language models depend on, preserving gradient flow and letting each sub-layer learn incremental adjustments rather than complete transformations. With this concept in place, the final topic in this section — layer normalization — explores the other key stabilization technique that works alongside residual connections within every transformer block.