Introduction

GPT (Generative Pre-trained Transformer) is the architectural counterpart to BERT within the transformer family — where BERT is an encoder-only model built for understanding text bidirectionally, GPT is a decoder-only model built specifically for generating text, one token at a time, using only leftward (causal) context. This architectural choice is precisely what makes GPT-style models so well-suited to the open-ended, conversational text generation seen in tools like ChatGPT, Claude, and countless other modern LLMs.

While the "Popular Models" section covered GPT as OpenAI's specific product family, this topic focuses on GPT as an architectural pattern — the decoder-only transformer design that GPT popularized and that now underlies the vast majority of large language models, regardless of which company builds them.

Why Is the GPT (Decoder-Only) Architecture Significant?

The decoder-only architecture helps to:

• Provide a simpler, more scalable design compared to full encoder-decoder transformers
• Naturally support open-ended, autoregressive text generation
• Unify a huge range of tasks under a single, general next-token-prediction objective
• Eliminate the need for a separate encoder stage or cross-attention mechanism
• Become the dominant architectural choice for nearly all major modern LLMs
• Demonstrate how a single, general-purpose design can be scaled to extraordinary capability

GPT's Core Idea: Causal, Left-to-Right Generation

Whiteboard

Unlike BERT's bidirectional self-attention, GPT uses masked
(causal) self-attention — as covered in the Encoder & Decoder
topic — where each token position can only attend to itself
and earlier positions, never future ones.

This restriction isn't a limitation so much as a necessity:
since GPT generates text one token at a time, it must learn
to predict each token using only the information that would
actually be available at generation time — no "cheating" by
looking ahead at tokens that don't exist yet during inference.

How GPT Is Pre-Trained

Next Token Prediction (Autoregressive Language Modeling)

GPT is pre-trained using the single, general objective covered in depth in an earlier topic: given a sequence of tokens, predict the next one, repeated across massive amounts of text.

Input:  "The cat sat on the"
Target: predict "mat" using only the preceding tokens

This single, simple objective is repeated across billions of
examples, and — as explored in the Next Token Prediction topic —
this is sufficient to teach the model grammar, facts, reasoning
patterns, and style, purely as a byproduct of getting better
at this one core task.

From Pre-Training to Helpful Assistant

Stage 1 — Pre-training:
GPT-style models learn general next-token prediction over
massive, broad text datasets (as covered in Introduction of DL
and Introduction of LLM topics).

Stage 2 — Instruction Fine-Tuning / RLHF:
The pre-trained model is further refined to follow instructions
helpfully and align with human preferences, as covered in the
Fine-Tuning Basics and Reinforcement Learning topics.

This two-stage process transforms a raw next-token predictor
into the kind of helpful, conversational assistant seen in
products like ChatGPT and Claude.

GPT's Architecture in Context

GPT (Decoder-Only) vs BERT (Encoder-Only)

Why Decoder-Only Became the Dominant Design

As covered in the Encoder & Decoder topic, most modern LLMs
converged on decoder-only designs because:

1. Simplicity: no separate encoder stage or cross-attention needed
2. Generality: a single next-token-prediction objective handles
   an enormous range of downstream tasks without task-specific
   architecture changes
3. Scalability: the simpler design scales more straightforwardly
   to the massive parameter counts and datasets seen in modern LLMs

This is why GPT, Claude, Llama, DeepSeek, and virtually every
other major conversational LLM today shares this same
fundamental decoder-only architectural pattern.

Key Properties of the Decoder-Only (GPT-Style) Architecture

• GPT-style models use masked, causal self-attention, restricting each token to only earlier context.
• Pre-training relies on the single, general next-token-prediction objective.
• Decoder-only models don't require a separate encoder stage or cross-attention mechanism.
• This architectural simplicity has made decoder-only the dominant design for modern large language models.
• Instruction fine-tuning and RLHF transform a raw pre-trained GPT-style model into a helpful, aligned assistant.

Where Is the Decoder-Only Architecture Used?

Advantages

• Simpler architecture than encoder-decoder designs, easing training and scaling
• A single, general pre-training objective supports an enormous range of downstream tasks
• Naturally suited to the kind of open-ended, conversational generation users expect from modern AI
• Has proven to scale remarkably well with increased data, parameters, and compute
• Dominant, well-understood design with extensive tooling and research support

Limitations

• Causal attention prevents the model from directly leveraging future context during generation, unlike BERT's bidirectional view
• Best understood as fundamentally a next-token predictor, with the associated hallucination risks covered earlier
• Sequential, autoregressive generation is inherently slower than a single encoder pass over fixed input
• Lacks BERT-style bidirectional strengths for certain pure understanding/classification tasks
• Like all LLMs, subject to the reasoning limitations discussed in the LLM Behavior section

Real-World Examples

Best Practices

• Choose decoder-only architectures for open-ended generation, conversation, and completion tasks.
• Understand causal (masked) self-attention as the specific mechanism enabling valid autoregressive generation.
• Recognize instruction fine-tuning and RLHF as the steps that turn a raw pre-trained model into a helpful assistant.
• Compare against encoder-only designs like BERT when a task is purely about understanding rather than generating.
• Connect this architecture back to the Next Token Prediction and Scaling Laws topics for a complete picture of how these models are built and improved.

Interview Tip

A common interview question is:

"Why do virtually all modern conversational LLMs, including GPT, Claude, and Llama, use a decoder-only architecture rather than the original encoder-decoder design?"

A strong answer is:

Decoder-only models use masked, causal self-attention and are trained with a single, general next-token-prediction objective, which removes the need for a separate encoder stage or cross-attention mechanism required in the original encoder-decoder transformer design. This architectural simplicity turned out to scale extremely well — a single pre-training objective can support an enormous range of downstream tasks, from conversation to coding to reasoning, without task-specific architectural changes, which is a major reason decoder-only designs became the dominant choice for large-scale, general-purpose language models like GPT, Claude, and Llama.

Connecting simplicity directly to scalability makes your answer stronger.

Conclusion

GPT's decoder-only architecture, built around causal self-attention and next-token prediction, represents the design pattern underlying virtually every major modern large language model, from GPT and Claude to Llama and DeepSeek. Paired directly against BERT's encoder-only, bidirectional approach, this comparison completes a clear, concrete picture of how the same core transformer building blocks — self-attention, the FFN, residual connections, and layer normalization — can be configured in fundamentally different ways depending on whether a task calls for deep understanding or open-ended generation.