Attention And Transformers
The Attention Mechanism
Module 5, Chapter 5 closed with a reframing: instead of compressing an entire sequence into one fixed-size vector, let the model directly look at every position
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Module 5: Sequence Models Time to complete: ~25 minutes
Table of Contents
- The Problem, Restated
- Attention's Core Idea: a Weighted Lookup
- Query, Key, Value — the Central Vocabulary
- Computing Attention Scores
- Scaled Dot-Product Attention
- A Full Worked Example
- Attention as an Encoder-Decoder Add-On (Historically)
- Summary & Next Steps
1. The Problem, Restated
Module 5, Chapter 5 closed with a reframing: instead of compressing an entire sequence into one fixed-size vector, let the model directly look at every position in the input and decide, dynamically, which ones matter most for the task at hand. This chapter makes that idea mathematically precise.
2. Attention's Core Idea: a Weighted Lookup
At its heart, attention computes a weighted average of a set of vectors, where the weights themselves are learned and computed dynamically, based on how relevant each vector is to the current context.
An Everyday Analogy: Searching a Library
─────────────────────────────────────────
QUERY: what you're looking for — e.g. "books about
gardening"
KEYS: each book's catalog entry/summary — what the
library uses to judge RELEVANCE to your query
VALUES: the actual book CONTENT you'd receive if a
given book is judged relevant
Attention computes: for THIS query, how relevant is EACH
key? Then it returns a WEIGHTED COMBINATION of the values,
weighted by those relevance scores — mostly gardening books,
with a small contribution from anything tangentially related.
─────────────────────────────────────────
3. Query, Key, Value — the Central Vocabulary
Every attention computation involves exactly three roles, each derived from the input via a learned linear transformation (Module 2, Chapter 1's dense layers):
The Three Roles
─────────────────────────────────────────
Query (Q): "what am I looking for right now?" — derived
from the CURRENT position that needs
context (e.g. the word currently being
generated or processed)
Key (K): "what do I contain, for MATCHING purposes?"
— derived from EVERY position that could
be attended TO
Value (V): "what do I ACTUALLY contribute, once
selected?" — also derived from every
position, but represents the
CONTENT to be aggregated, not just
used for matching
─────────────────────────────────────────
import numpy as np
def create_qkv(x, W_q, W_k, W_v):
"""x: input vectors, shape (seq_len, d_model)"""
Q = x @ W_q # what to look FOR
K = x @ W_k # what to be MATCHED against
V = x @ W_v # what to actually RETURN
return Q, K, VQ, K, and V all start from the SAME input in self-attention (Chapter 2) — but each is projected through its own learned weight matrix, letting the model learn different, specialized representations for "searching," "matching," and "content" purposes.
4. Computing Attention Scores
The Three-Step Attention Computation
─────────────────────────────────────────
Step 1: SCORE compute how well the query matches
EVERY key (typically via a dot product
— Section 5 explains why)
Step 2: NORMALIZE apply softmax (Module 2, Ch.2) across
all scores, converting them into a
valid probability distribution
that sums to 1
Step 3: AGGREGATE compute a WEIGHTED SUM of all
values, using the normalized
scores as weights
─────────────────────────────────────────
5. Scaled Dot-Product Attention
The specific formula used by Transformers (introduced in "Attention Is All You Need") is called scaled dot-product attention.
Attention(Q, K, V) = softmax( (Q · Kᵀ) / √d_k ) · V
import numpy as np
def softmax(x, axis=-1):
exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
return exp_x / np.sum(exp_x, axis=axis, keepdims=True)
def scaled_dot_product_attention(Q, K, V):
d_k = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d_k) # Step 1 (Section 4) — SCALED to avoid extreme values
attention_weights = softmax(scores) # Step 2 (Section 4)
output = attention_weights @ V # Step 3 (Section 4)
return output, attention_weightsWhy Divide by √d_k (the Scaling)
─────────────────────────────────────────
Dot products between HIGH-dimensional vectors tend to
produce LARGE values purely due to dimensionality, even for
UNRELATED vectors — large raw scores push softmax into its
saturated region (Module 2, Ch.2's discussion of extreme
sigmoid/softmax inputs), producing near-zero gradients for
most positions and near-one-hot output weights. Dividing by
√d_k keeps score magnitudes in a reasonable, stable range
REGARDLESS of the dimensionality chosen, which matters a
great deal for training stability at the LARGE d_k values
used in real Transformers (often 64 or more).
─────────────────────────────────────────
6. A Full Worked Example
import numpy as np
np.random.seed(42)
# A tiny "sequence" of 3 tokens, each represented by a 4-dimensional vector
x = np.random.randn(3, 4)
d_model, d_k = 4, 4
W_q = np.random.randn(d_model, d_k) * 0.5
W_k = np.random.randn(d_model, d_k) * 0.5
W_v = np.random.randn(d_model, d_k) * 0.5
Q, K, V = x @ W_q, x @ W_k, x @ W_v # Section 3
output, weights = scaled_dot_product_attention(Q, K, V)
print("Attention weights (each ROW sums to 1 — one row per QUERY position):")
print(np.round(weights, 3))
print("\nOutput (weighted combination of VALUES, per position):")
print(np.round(output, 3))Interpreting the Output
─────────────────────────────────────────
Each ROW of `weights` shows how much ONE query position
attends to EVERY key position (including possibly itself)
— a HIGH weight means "this position's VALUE strongly
contributes to my output." The resulting `output` is a NEW
representation for each position, now INFORMED by relevant
context from anywhere else in the sequence — regardless of
distance, directly solving Module 5, Chapter 5's long-range
dependency problem.
─────────────────────────────────────────
7. Attention as an Encoder-Decoder Add-On (Historically)
Attention was originally introduced (2014-2015) as an addition to the RNN-based encoder-decoder models from Module 5, Chapter 4 — the decoder, at each step, would compute attention over all of the encoder's hidden states (rather than relying only on one fixed context vector), directly fixing the fixed-size bottleneck. It was only in 2017 that "Attention Is All You Need" showed attention could replace recurrence entirely, removing RNNs from the architecture altogether — this is the leap covered in the rest of this module.
8. Summary & Next Steps
Key Takeaways
- Attention computes a weighted average of "value" vectors, where weights are determined dynamically by comparing a "query" against a set of "keys."
- Scaled dot-product attention computes scores via a dot product between queries and keys, scales by √d_k for numerical stability, applies softmax, then uses the result to weight the values.
- The √d_k scaling factor prevents large-dimensional dot products from pushing softmax into a saturated, low-gradient region.
- Attention was originally an add-on to RNN encoder-decoder models, fixing the fixed-size context bottleneck, before "Attention Is All You Need" showed it could replace recurrence entirely.
Concept Check
- In the library search analogy, what do the query, key, and value each correspond to?
- Why does scaled dot-product attention divide the raw scores by √d_k before applying softmax?
- How does attention directly solve the long-range dependency problem that plagued RNN-based sequence models?
Next Chapter
→ Chapter 2: Self-Attention & Multi-Head Attention
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index