Introduction

T5 (Text-to-Text Transfer Transformer) is Google's encoder-decoder transformer model, introduced in 2019, notable for reframing virtually every NLP task — translation, summarization, classification, question answering — as a single, unified "text-to-text" problem: given some input text, produce some output text. Rather than requiring different architectures or output formats for different tasks, T5 treats everything as the same fundamental operation, just with different input and output text.

As the third major architectural pattern alongside BERT (encoder-only) and GPT (decoder-only), T5 completes the picture of how the same core transformer components — self-attention, the FFN, residual connections, and layer normalization — can be combined into the full encoder-decoder design originally introduced in "Attention Is All You Need," and applied specifically to tasks with a clear input-to-output transformation.

Why Is T5 Significant?

T5 helps to:

• Demonstrate a unified "text-to-text" framing that simplifies handling many different NLP tasks
• Provide a concrete, influential example of the full encoder-decoder transformer design
• Combine BERT-style bidirectional understanding (in its encoder) with GPT-style generation (in its decoder)
• Show how a single model architecture and training format can generalize across diverse tasks
• Serve as a foundation for later text-to-text and instruction-following model research
• Round out the three fundamental transformer configurations alongside BERT and GPT

T5's Core Idea: Everything Is Text-to-Text

Whiteboard

Instead of using different output formats for different tasks
(e.g., a numeric label for classification, a probability
distribution for translation), T5 reframes every task so that
both the input AND the output are plain text, with a short task
prefix indicating what operation should be performed.

This means the exact same model architecture, training process,
and even loss function can be used across a huge variety of
seemingly very different NLP tasks.

T5's Encoder-Decoder Architecture

Input text (with task prefix) → Encoder
  → Encoder builds a rich, bidirectional representation
    of the entire input (similar in spirit to BERT)

Encoder's output → Decoder (via cross-attention)
  → Decoder generates the output text one token at a time,
    using masked self-attention over its own previous outputs
    plus cross-attention to the encoder's representation
    (similar in spirit to GPT's generation process, but
    grounded in the encoder's understanding of the input)

How T5 Was Pre-Trained

Span Corruption (T5's Pre-Training Objective)

Random contiguous spans of text are masked out and replaced with special sentinel tokens; the model learns to predict the missing spans as text-to-text output.

Original: "The quick brown fox jumps over the lazy dog"
Corrupted Input:  "The quick <X> jumps over the <Y> dog"
Target Output:    "<X> brown fox <Y> lazy"

This is a text-to-text-friendly variation on the masked
language modeling idea introduced by BERT, adapted to fit
T5's encoder-decoder generation setup.

T5's Architecture in Context

BERT vs GPT vs T5: The Three Architectural Patterns

When Encoder-Decoder Makes Sense vs Decoder-Only

Encoder-Decoder (T5-style) tends to fit well when:
- There's a clear, distinct input and output (e.g., source
  sentence → translated sentence)
- The task benefits from deeply, bidirectionally understanding
  the full input before generating anything

Decoder-Only (GPT-style) tends to fit well when:
- The task is open-ended, conversational, or doesn't have a
  clean separation between "input" and "output"
- Simplicity and general-purpose flexibility across many
  unrelated tasks matters more than specialized input encoding

This is a major reason most general-purpose conversational
LLMs today are decoder-only, while encoder-decoder designs
remain common for specific structured tasks like translation.

Key Properties of T5

• T5 reframes every NLP task as a text-to-text problem, using a task-specific prefix to indicate the desired operation.
• It uses the full encoder-decoder transformer design, combining bidirectional encoding with autoregressive decoding.
• T5 was pre-trained using a span corruption objective, a text-to-text adaptation of masked language modeling.
• The unified format allows a single model, training process, and loss function to handle many different tasks.
• T5 was released in multiple sizes, allowing tradeoffs between capability and computational cost.

Where Is T5 (and Its Text-to-Text Approach) Used?

Advantages

• Unified text-to-text framing simplifies handling many different task types with one architecture
• Combines the strengths of bidirectional understanding (encoder) and generation (decoder)
• Well-suited to tasks with a clear, structured input-to-output transformation
• Demonstrates strong transfer learning across a wide variety of NLP benchmarks
• Multiple model sizes accommodate different capability and resource requirements

Limitations

• More architecturally complex and computationally heavier than a decoder-only design
• Cross-attention adds overhead compared to simpler decoder-only generation
• Less naturally suited to fully open-ended, unstructured conversational use cases
• Has been less prominent in mainstream discussion as decoder-only LLMs have become dominant for general-purpose AI
• Still subject to general LLM limitations like hallucination and reasoning constraints

Real-World Examples

Best Practices

• Choose encoder-decoder architectures like T5 for tasks with a clear, distinct input-to-output transformation.
• Consider the text-to-text framing when designing systems that need to handle multiple related NLP tasks uniformly.
• Compare against decoder-only models when a task is more open-ended or conversational in nature.
• Understand span corruption as T5's adaptation of masked language modeling for a text-to-text setup.
• Use T5 or similar encoder-decoder designs when leveraging cross-attention grounding to the input is especially valuable.

Interview Tip

A common interview question is:

"What is T5's 'text-to-text' approach, and how does its encoder-decoder architecture combine ideas from both BERT and GPT?"

A strong answer is:

T5 reframes every NLP task — translation, summarization, classification, and more — as a text-to-text problem, where both the input and output are plain text, with a task prefix indicating what operation to perform, allowing one unified architecture and training process to handle many different tasks. Architecturally, its encoder uses full, bidirectional self-attention similar to BERT to deeply understand the input, while its decoder uses masked self-attention plus cross-attention to the encoder's output, similar in spirit to GPT's generation process but grounded directly in that encoded input — combining bidirectional understanding with autoregressive generation in a single model.

Explicitly connecting T5's two halves back to BERT and GPT makes your answer stronger and ties the three architectures together.

Conclusion

T5's encoder-decoder architecture and unified text-to-text framing complete the trio of fundamental transformer configurations alongside BERT's encoder-only design and GPT's decoder-only design, each suited to a different balance of understanding versus generation. Together, these three architectural patterns — encoder-only, decoder-only, and encoder-decoder — demonstrate how the same core transformer building blocks can be assembled in different ways depending on whether a task calls for deep understanding, open-ended generation, or a structured transformation between the two.