Introduction

Self-attention is the core mechanism that allows a transformer to weigh the relevance of every token in a sequence against every other token, building a representation for each token that's informed by its full surrounding context. While earlier topics introduced self-attention at a high level as part of the broader transformer architecture, this topic takes a closer, more detailed look at exactly how it works mathematically — the Query, Key, and Value vectors, the scaled dot-product calculation, and how multiple attention heads combine to capture different types of relationships simultaneously.

Understanding self-attention at this level of detail is essential for truly grasping why transformers behave the way they do, since this single mechanism — repeated across every layer of every transformer block — is what allows models like GPT, Claude, and Llama to connect relevant information across an entire input, regardless of how far apart it appears.

Why Does Self-Attention Matter?

Self-attention helps to:

• Let every token directly consider every other token in a sequence, regardless of distance
• Build context-aware, contextual representations rather than fixed, static ones
• Enable full parallelization during training, unlike RNNs' sequential processing
• Capture multiple, different types of relationships simultaneously through multi-head attention
• Serve as the foundational mechanism underlying every transformer-based model
• Provide the specific technical answer to "how does a model know what's relevant?"

The Three Vectors: Query, Key, and Value

Whiteboard

Forevery token, three separate vectors are computed by

multiplying its embedding by three learned weight matrices:

Query (Q)  = Embedding × W_Q   → "What am I looking for?"
Key (K)    = Embedding × W_K   → "What do I contain/offer?"
Value (V)  = Embedding × W_V   → "What information do I provide?"

These three weight matrices (W_Q, W_K, W_V) are learned during
training — they're what allows the model to learn what kinds
of relationships matter for predicting language well.

The Scaled Dot-Product Attention Formula

Attention(Q, K, V) = softmax( (Q · K^T) / √d_k ) · V

Step by step:
1. Q · K^T: compute a similarity score between each token's
   Query and every token's Key (via dot product)
2. Divide by √d_k: scale down the scores to keep training
   numerically stable (d_k is the dimension of the Key vectors)
3. softmax: convert the scaled scores into probabilities
   that sum to 1 across all tokens
4. Multiply by V: use those probabilities as weights to
   combine all tokens' Value vectors into a single output

A Simple Step-by-Step Illustration

Sentence: "The animal didn't cross the street because it was tired"

When computing the new representation for the word "it":

1. "it"'s Query vector is compared against the Key vector
   of every other token in the sentence
2. This produces raw similarity scores — likely highest
   for "animal" (since "it" refers back to the animal)
3. Scores are scaled and passed through softmax, producing
   attention weights, e.g.:
   "animal" → 0.65, "street" → 0.08, "tired" → 0.05, ... (rest smaller)
4. "it"'s new representation becomes a weighted combination
   of all tokens' Value vectors, dominated by "animal"'s Value

Result: "it"'s representation now strongly reflects that it
refers to "the animal," resolved purely through attention weights
learned from data — no explicit grammar rules were programmed.

Why Scaling by √d_k Matters

As the dimensionality of Q and K vectors (d_k) grows larger,
the raw dot-product values tend to grow larger too, which can
push the softmax function into regions with extremely small
gradients — making training slow or unstable.

Dividing by √d_k keeps the scores in a more stable numerical
range, helping gradients flow effectively during training.
This is why it's called "scaled" dot-product attention.

Multi-Head Attention: Multiple Perspectives at Once

Instead of computing attention just once, transformers split
Q, K, and V into multiple smaller sets ("heads") and compute
attention independently within each one:

Head 1: might learn to focus on grammatical relationships
Head 2: might learn to focus on long-range references
Head 3: might learn to focus on local, nearby word patterns
... (often 8, 16, or more heads per layer)

Each head's output is then concatenated together and passed
through a final linear layer, combining all these different
"perspectives" into one richer, unified representation.

Self-Attention vs Cross-Attention

(Cross-attention was introduced in the Encoder & Decoder topic and used specifically in encoder-decoder models like T5.)

Full (Bidirectional) vs Masked (Causal) Self-Attention

(This distinction was covered in depth in the BERT and GPT architecture topics.)

Key Properties of Self-Attention

• Self-attention computes Query, Key, and Value vectors from the same input sequence using learned weight matrices.
• The scaled dot-product formula compares Queries against Keys, scales the result, and applies softmax to get attention weights.
• Multi-head attention runs several attention computations in parallel, capturing different relationship types simultaneously.
• Self-attention allows any two tokens to directly interact, regardless of their distance in the sequence.
• Whether attention is full (bidirectional) or masked (causal) depends on the model's architecture and intended task.

Where Does Self-Attention's Mechanism Matter Most?

Advantages

• Enables direct, parallel connections between any two tokens regardless of distance
• Fully parallelizable, unlike the sequential processing required by RNNs/LSTMs
• Multi-head design captures multiple types of relationships within the same layer
• Learned entirely from data, without requiring hand-coded linguistic rules
• Forms a flexible, general-purpose mechanism reused across language, vision, and other domains

Limitations

• Computational cost scales quadratically with sequence length in its standard form
• Requires substantial data to learn meaningful, useful attention patterns
• Attention weights, while interpretable to a degree, don't fully explain model reasoning
• Multiple heads can sometimes learn redundant or less interpretable patterns
• Long sequences require specialized, optimized attention variants to remain computationally practical

Real-World Examples

Best Practices

• Understand Query, Key, and Value as learned projections serving distinct roles: seeking, offering, and providing information.
• Study the scaled dot-product formula directly to build genuine intuition, not just a high-level analogy.
• Recognize multi-head attention as enabling multiple relationship types to be captured within a single layer.
• Distinguish self-attention from cross-attention, and full from masked attention, based on architecture and task.
• Consider the quadratic computational cost when reasoning about context window limits and long-sequence efficiency.

Interview Tip

A common interview question is:

"Can you explain the scaled dot-product attention formula and what each part of it does?"

A strong answer is:

The formula is Attention(Q, K, V) = softmax((Q · K^T) / √d_k) · V. First, Q · K^T computes a similarity score between each token's Query and every token's Key through a dot product, indicating how relevant each token is to the one currently being processed. Dividing by √d_k scales these scores to keep training numerically stable, since raw dot products can grow large with higher-dimensional vectors. The softmax function then converts these scaled scores into attention weights that sum to 1, and finally, multiplying by V produces a weighted combination of all tokens' Value vectors — giving each token a new representation informed by the most relevant context across the entire sequence.

Walking through each step of the formula in order makes your answer stronger and demonstrates real technical depth.

Conclusion

Self-attention, through its Query, Key, and Value vectors and the scaled dot-product formula, provides the precise mathematical mechanism that allows transformers to build rich, context-aware representations by directly relating every token to every other token in a sequence. Combined with multi-head attention's ability to capture multiple relationship types simultaneously, this mechanism — examined here in full technical detail — is what makes the high-level transformer architecture, encoder-decoder designs, and every model built on top of them actually work.