Introduction
The Feed Forward Network (FFN) is one of the two core sub-layers within every transformer block, sitting right after the self-attention mechanism and applying an independent, position-wise transformation to each token's representation. While self-attention handles how tokens relate to and gather information from each other, the FFN is where much of the actual "processing" and pattern transformation of that information happens, applied identically and independently to every position in the sequence.
Despite receiving far less attention (pun intended) in popular discussions of transformers than self-attention itself, the FFN actually accounts for a substantial portion of a transformer's total parameters and computational work, making it a critical component in the overall architecture.
Why Does the Feed Forward Network Matter?
The FFN helps to:
- Apply additional non-linear transformation and processing to each token's representation
- Increase the model's overall capacity to learn complex patterns
- Operate independently on each token position, complementing attention's cross-token mixing
- Contribute a large share of a transformer's total learnable parameters
- Work together with self-attention to form the complete transformer block
- Provide the "thinking" step that follows attention's "gathering relevant information" step
Where the FFN Fits Within a Transformer Block
What the Feed Forward Network Actually Does
The FFN is a simple, two-layer neural network applied
independently to each token's vector representation:
FFN(x) = Activation(x · W1 + b1) · W2 + b2
1. First linear layer: expands the vector to a larger
intermediate dimension (often 4x the model's base dimension)
2. Activation function: introduces non-linearity (e.g., GELU,
commonly used in modern transformers)
3. Second linear layer: projects back down to the original dimension
Crucially, the exact same FFN (same weights) is applied to
every single token position independently — it doesn't mix
information between tokens the way self-attention does.A Simple Illustration of the Dimension Change
Model's base embedding dimension: 768
FFN's intermediate (hidden) dimension: 3072 (4x expansion)
Token vector (768) → Linear layer 1 → (3072) → Activation
→ Linear layer 2 → back to (768)
This "expand then compress" pattern allows the network to
learn richer intermediate transformations before condensing
the result back to the standard size used throughout the model.Self-Attention vs the Feed Forward Network
| Aspect | Self-Attention | Feed Forward Network |
|---|---|---|
| Primary Role | Mixes and relates information across different tokens | Transforms each token's representation independently |
| Operates On | The entire sequence at once (cross-token relationships) | Each token position separately (no cross-token mixing) |
| Parameters | Query, Key, Value projection weights | Two linear layers with an activation function between them |
| Analogy | "Gathering relevant context from elsewhere in the sequence" | "Processing and refining what's already been gathered" |
Why the FFN Contains So Many Parameters
Because the FFN expands to a much larger intermediate dimension
(commonly 4x) before compressing back down, and this happens
independently within every single transformer layer, the FFN
sub-layers collectively account for a substantial share —
often roughly two-thirds — of a transformer's total parameter count.
This is one reason parameter count (discussed in earlier topics)
scales so significantly with model depth and width.Common Activation Functions Used in FFNs
| Activation Function | Notes |
|---|---|
| ReLU | Used in earlier transformer implementations, simple and efficient |
| GELU (Gaussian Error Linear Unit) | Smoother than ReLU, widely used in many modern transformer-based LLMs |
| SwiGLU and other Gated Variants | Used in some newer, more advanced architectures for improved performance |
(These build directly on the activation function concepts covered in the earlier Activation Functions topic.)
Key Properties of the Feed Forward Network
- The FFN is a two-layer neural network with a non-linear activation function applied between the layers.
- It's applied independently and identically to every token position — no information is shared across positions within the FFN itself.
- The FFN typically expands to a larger intermediate dimension (often 4x) before compressing back to the model's base dimension.
- FFN sub-layers make up a substantial portion of a transformer's total parameter count.
- The FFN and self-attention sub-layers work together within each transformer block, each serving a distinct, complementary role.
Where Does the FFN's Role Matter Most?
| Context | Relevance of the FFN |
|---|---|
| Model Capacity and Scaling | A major contributor to total parameter count and representational capacity |
| Architecture Research | A frequent target for efficiency improvements (e.g., gated variants, sparsity) |
| Understanding Transformer Internals | Essential complement to understanding self-attention's role |
| Model Compression / Efficiency Work | FFN layers are commonly targeted for pruning or quantization due to their size |
| Mixture of Experts (MoE) Architectures | Often specifically replace/adapt the FFN sub-layer with multiple specialized "expert" FFNs |
Advantages
- Adds essential non-linear processing capacity beyond what attention alone provides
- Operates efficiently and independently per token, well-suited to parallel computation
- The expand-then-compress design allows richer intermediate representations
- Well-understood, simple structure that's straightforward to implement and scale
- Complements self-attention's cross-token mixing with position-wise transformation
Limitations
- Contributes significantly to a transformer's overall parameter count and compute cost
- Since it operates independently per token, it cannot itself capture relationships between tokens
- Larger intermediate dimensions increase memory and computational requirements
- A frequent target for optimization, indicating it's also a common source of inefficiency if not well-designed
- Choice of activation function and intermediate dimension size involves real architectural tradeoffs
Real-World Examples
| Context | FFN Relevance |
|---|---|
| GPT/Claude/Llama Architecture | FFN sub-layers present within every transformer block |
| Mixture of Experts Models | Replace the standard FFN with multiple specialized expert FFNs |
| Model Efficiency Research | FFN pruning and quantization techniques to reduce model size |
| Transformer Scaling Studies | FFN dimension is a key lever in balancing model size and capability |
| Custom Transformer Variants | Researchers experimenting with alternative FFN designs (e.g., gated activations) |
Best Practices
- Understand the FFN as the complementary "processing" step following attention's "gathering" step.
- Recognize the FFN's significant contribution to total parameter count when reasoning about model size.
- Study modern activation function choices (like GELU) used within FFN layers in current architectures.
- Consider FFN-focused optimization techniques (pruning, MoE) when working on efficiency-constrained deployments.
- Pair understanding of the FFN with self-attention to form a complete picture of a transformer block.
Interview Tip
A common interview question is:
"What role does the Feed Forward Network play within a transformer block, and how does it differ from self-attention?"
A strong answer is:
The Feed Forward Network applies a position-wise, non-linear transformation independently to each token's representation, typically expanding to a larger intermediate dimension before compressing back down, using an activation function like GELU in between two linear layers. This differs fundamentally from self-attention, which mixes information across different tokens in the sequence — the FFN, by contrast, processes each token's representation on its own, without any cross-token interaction. Together, self-attention handles gathering relevant context from across the sequence, while the FFN handles further transforming and refining that information for each token independently, and the FFN sub-layers actually make up a substantial share of a transformer's total parameters.
Clearly framing attention as "gathering" and FFN as "processing" makes your answer stronger and easy to remember.
Conclusion
The Feed Forward Network provides the essential position-wise processing step within every transformer block, complementing self-attention's cross-token information gathering with independent, non-linear transformation of each token's representation. With the FFN now covered, the next topics — residual connections and layer normalization — explore the supporting mechanisms that make training these stacked transformer blocks stable and effective at scale.