Attention And Transformers
Positional Encoding
Self-attention (Chapter 2) computes relevance between every pair of positions using only their content (via Q and K) — critically, nothing in the attention comp
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 2: Self-Attention & Multi-Head Attention Time to complete: ~20 minutes
Table of Contents
- The Problem: Attention Has No Sense of Order
- Why Not Just Number the Positions?
- Sinusoidal Positional Encoding
- Why Sine and Cosine, Specifically
- Adding Positional Encoding to Embeddings
- Learned Positional Embeddings — the Alternative
- Positional Encoding in PyTorch
- Summary & Next Steps
1. The Problem: Attention Has No Sense of Order
Self-attention (Chapter 2) computes relevance between every pair of positions using only their content (via Q and K) — critically, nothing in the attention computation itself depends on where a position sits in the sequence.
Why This Is a Genuine Problem
─────────────────────────────────────────
"The dog bit the man" and "The man bit the dog" contain the
EXACT SAME set of words — if attention only looks at
content, and never at ORDER, these two sentences would
produce IDENTICAL self-attention outputs (just permuted),
despite having completely different meanings.
This directly echoes Module 5, Chapter 1's opening point
about why order matters for sequences — and reveals that
removing recurrence (Chapter 2) came at the cost of ALSO
removing the NATURAL sense of order an RNN's step-by-step
processing provided for free.
─────────────────────────────────────────
2. Why Not Just Number the Positions?
The simplest fix — appending the raw position index (0, 1, 2, 3, ...) to each token's representation — turns out to work poorly in practice: raw position numbers can grow arbitrarily large for long sequences, and provide no natural way for the model to generalize to sequence lengths not seen during training.
3. Sinusoidal Positional Encoding
The original Transformer paper's solution: encode each position using a combination of sine and cosine waves at different frequencies, producing a fixed-size vector (matching the model's embedding dimension) that uniquely represents each position.
The Formula
─────────────────────────────────────────
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
pos: the position in the sequence (0, 1, 2, ...)
i: the dimension index within the encoding vector
d_model: the model's embedding dimension (e.g. 512)
─────────────────────────────────────────
import numpy as np
def sinusoidal_positional_encoding(seq_len, d_model):
positions = np.arange(seq_len)[:, np.newaxis] # shape (seq_len, 1)
dimensions = np.arange(d_model)[np.newaxis, :] # shape (1, d_model)
angle_rates = 1 / np.power(10000, (2 * (dimensions // 2)) / d_model)
angles = positions * angle_rates
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(angles[:, 0::2]) # EVEN dimensions get sine
pe[:, 1::2] = np.cos(angles[:, 1::2]) # ODD dimensions get cosine
return pe
pe = sinusoidal_positional_encoding(seq_len=10, d_model=16)
print(f"Positional encoding shape: {pe.shape}") # (10, 16)
print(f"Encoding for position 0:\n{pe[0]}")
print(f"Encoding for position 5:\n{pe[5]}")4. Why Sine and Cosine, Specifically
Two Key Properties That Make This Work
─────────────────────────────────────────
1. UNIQUE per position: each position gets a DISTINCT
encoding vector, since
different frequencies (from
the 10000^(2i/d_model) term)
produce different sine/cosine
patterns at each position
2. Encodes RELATIVE distance for any FIXED offset k, the
naturally: encoding at position
(pos + k) can be
expressed as a LINEAR
function of the encoding
at position pos — this
lets the model learn to
attend based on RELATIVE
position (e.g. "the word
3 positions back"),
which generalizes better
than absolute position
alone
3. Extends to UNSEEN lengths: the formula works for ANY
position, including
sequence lengths LONGER
than anything seen
during training — no
retraining required to
handle a longer input
─────────────────────────────────────────
5. Adding Positional Encoding to Embeddings
Positional encoding is added directly (element-wise) to each token's embedding, before the first attention layer — this single sum lets subsequent layers access both content (from the embedding) and position (from the encoding) simultaneously.
def add_positional_encoding(token_embeddings, positional_encoding):
"""
token_embeddings: shape (seq_len, d_model) — from an embedding layer
positional_encoding: shape (seq_len, d_model) — from Section 3
"""
return token_embeddings + positional_encoding # simple element-wise ADDITION
seq_len, d_model = 5, 16
token_embeddings = np.random.randn(seq_len, d_model) * 0.1
pos_encoding = sinusoidal_positional_encoding(seq_len, d_model)
final_input = add_positional_encoding(token_embeddings, pos_encoding)
print(f"Final input to the Transformer: {final_input.shape}")6. Learned Positional Embeddings — the Alternative
Rather than a fixed sinusoidal formula, some Transformer variants (including several covered in the NLP & LLM Notes) instead learn a positional embedding for each position, exactly like a token embedding — simply looking up a learned vector by position index.
Fixed (Sinusoidal) vs Learned Positional Encodings
─────────────────────────────────────────
Sinusoidal (Section 3): no additional parameters to
learn; naturally extends to
sequence lengths beyond
training (Section 4, point 3)
Learned: one MORE embedding table to
train, alongside the token
embedding table — often
performs comparably or
slightly better in practice,
but does NOT naturally
extend beyond the MAXIMUM
sequence length seen during
training
─────────────────────────────────────────
7. Positional Encoding in PyTorch
import torch
import torch.nn as nn
import math
class PositionalEncoding(nn.Module):
def __init__(self, d_model, max_len=5000):
super().__init__()
pe = torch.zeros(max_len, d_model)
position = torch.arange(0, max_len).unsqueeze(1).float()
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
self.register_buffer("pe", pe.unsqueeze(0)) # NOT a trainable parameter — fixed, computed once
def forward(self, x):
return x + self.pe[:, :x.size(1)] # Section 5's addition
embedding_layer = nn.Embedding(num_embeddings=10000, embedding_dim=512)
pos_encoder = PositionalEncoding(d_model=512)
token_ids = torch.randint(0, 10000, (1, 20)) # a 20-token sequence
embeddings = embedding_layer(token_ids)
final_input = pos_encoder(embeddings)
print(f"Final input shape: {final_input.shape}") # [1, 20, 512]8. Summary & Next Steps
Key Takeaways
- Self-attention has no inherent sense of position or order — it computes relevance based purely on content, meaning positional information must be added explicitly.
- Sinusoidal positional encoding uses sine and cosine waves at varying frequencies to produce a unique, position-specific vector that encodes relative distance naturally and extends to unseen sequence lengths.
- Positional encoding is added element-wise to token embeddings, before the first attention layer, giving every subsequent layer access to both content and position.
- Learned positional embeddings are a common alternative, trading the ability to extend beyond training-time sequence lengths for potentially better task-specific performance.
Concept Check
- Why would swapping the word order in a sentence produce an identical self-attention output if no positional information were added?
- What two properties make sinusoidal encoding effective at representing both unique and relative position?
- What's the main tradeoff between fixed sinusoidal encoding and learned positional embeddings?
Next Chapter
→ Chapter 4: The Full Transformer Architecture
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index