Introduction

The encoder and decoder are the two structural halves of the original transformer architecture, each built from stacked layers of self-attention and feedforward networks, but serving fundamentally different purposes: the encoder builds a rich understanding of an input sequence, while the decoder generates an output sequence, often while referencing that encoded understanding. Depending on the task, transformers can use both components together, or rely on just one of the two alone.

Understanding the distinct roles of encoders and decoders — and why most modern large language models have moved toward a decoder-only design — completes the architectural picture built up through tokens, embeddings, positional encoding, and self-attention in the previous topics.

Why Do Encoders and Decoders Matter?

Encoders and decoders help to:

• Divide the work of understanding input from the work of generating output
• Support different task types through different architectural configurations
• Enable powerful sequence-to-sequence applications like translation and summarization
• Explain why most modern LLMs use a simplified, decoder-only design
• Clarify how the original transformer paper's architecture is adapted for different purposes
• Provide the final structural piece connecting all previously covered transformer components

The Original Encoder-Decoder Transformer

Whiteboard

Loading diagram...

How the Encoder Works

The encoder's job: build a rich, contextual understanding
of the entire input sequence.

1. Input tokens converted to embeddings + positional encoding
2. Passed through multiple stacked encoder layers
3. Each layer applies self-attention (every token attends to
   every other input token) followed by a feedforward network
4. Output: a set of context-rich vectors representing the
   full input sequence, capturing relationships between all tokens

How the Decoder Works

The decoder's job: generate an output sequence, one token
at a time, using both the encoder's representation (if present)
and its own previously generated tokens.

1. Previously generated tokens converted to embeddings + positional encoding
2. Passed through multiple stacked decoder layers, each applying:
   a. Masked self-attention (only attends to earlier output tokens)
   b. Cross-attention to the encoder's output (in encoder-decoder designs)
   c. A feedforward network
3. Output: a probability distribution over the vocabulary
   for the next token to generate

Masked Self-Attention: A Key Decoder Feature

Unlike the encoder, which allows every input token to attend
to every other input token freely, the decoder uses "masked"
self-attention — each position can only attend to itself and
earlier positions, never future ones.

Why this matters: during generation, the model must predict
the next token without "cheating" by looking ahead at tokens
it hasn't generated yet. This masking enforces that constraint,
both during training and at inference time.

Cross-Attention: Connecting Encoder and Decoder

In the original encoder-decoder design, an additional attention mechanism — cross-attention — allows each decoder layer to also attend directly to the encoder's output, letting the generation process stay grounded in the original input's meaning throughout.

Decoder's masked self-attention → considers previously generated output
Decoder's cross-attention → considers the encoder's full input representation
Combined → generation that's both self-consistent and input-grounded

Three Transformer Configurations

Why Most Modern LLMs Use Decoder-Only Designs

Modern large language models are primarily trained on a simple,
general task: predict the next token given everything before it.

A decoder-only architecture — using masked self-attention over
the entire input-plus-generation sequence — turns out to be
sufficient and highly effective for this kind of general-purpose,
open-ended text generation, without needing a separate encoder
or cross-attention mechanism.

This simplification also makes training and scaling more
straightforward, contributing to why decoder-only designs
became the dominant choice for large-scale LLMs.

Encoder-Only vs Decoder-Only vs Encoder-Decoder

Key Properties of Encoders and Decoders

• The encoder uses full, bidirectional self-attention to build a rich understanding of the entire input.
• The decoder uses masked (causal) self-attention, only attending to earlier positions, to preserve valid generation order.
• Cross-attention connects decoder generation to the encoder's representation in full encoder-decoder designs.
• Most modern large language models use a decoder-only architecture, well-suited for general text generation.
• The choice between architectures depends heavily on whether a task requires understanding, generation, or both together.

Where Are Encoders and Decoders Used?

Advantages

• Separating encoding and decoding allows architectures to be tailored precisely to different task types
• Masked self-attention enables valid, coherent autoregressive generation
• Cross-attention allows encoder-decoder models to stay tightly grounded in input content
• Decoder-only simplicity has proven highly effective and scalable for general-purpose LLMs
• The modular design has enabled many specialized variants across different domains

Limitations

• Encoder-decoder models are more complex and computationally heavier than single-stack designs
• Decoder-only models, lacking a dedicated encoder, may handle certain structured input-heavy tasks less directly
• Masked self-attention inherently limits the decoder to generating tokens in strict left-to-right order
• Choosing the right configuration for a given task requires understanding these architectural tradeoffs
• Cross-attention adds computational overhead compared to simpler decoder-only designs

Real-World Examples

Best Practices

• Choose encoder-only architectures for tasks focused purely on understanding or embedding input, not generation.
• Choose decoder-only architectures for general-purpose, open-ended text generation tasks.
• Consider encoder-decoder architectures for tasks with a clear, structured input-to-output transformation, like translation.
• Understand masked self-attention as the mechanism enforcing valid, left-to-right generation order.
• Recognize why decoder-only simplicity has made it the dominant choice for today's large-scale LLMs.

Interview Tip

A common interview question is:

"What's the difference between the encoder and decoder in a transformer, and why do most modern LLMs use a decoder-only design?"

A strong answer is:

The encoder uses full, bidirectional self-attention to build a rich contextual understanding of an entire input sequence, while the decoder uses masked self-attention, only attending to earlier positions, to generate output one token at a time in valid order — with cross-attention connecting the two in the original encoder-decoder design. Most modern LLMs use a decoder-only architecture because their core training objective — predicting the next token given everything before it — doesn't require a separate encoding stage; this simplification makes training and scaling more straightforward, which is a major reason decoder-only designs became the dominant choice for large-scale generative language models.

Explaining both the architectural mechanism and the practical scaling motivation makes your answer stronger.

Conclusion

The encoder and decoder represent the two structural halves of the original transformer, with the encoder building understanding through full self-attention and the decoder generating output through masked self-attention and, in full encoder-decoder designs, cross-attention back to the encoder. With this final piece in place, the complete transformer architecture — from tokenization and embeddings through positional encoding, self-attention, and this encoder-decoder structure — provides the full technical foundation underlying virtually every modern large language model.