Sequence Models
Sequence-to-Sequence & Encoder-Decoder Models
Chapter 2, Section 6 identified "many-to-many (unaligned)" as a distinct pattern: mapping an input sequence to an output sequence of a different length, without
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 3: LSTMs & GRUs Time to complete: ~20 minutes
Table of Contents
- The Unaligned Many-to-Many Problem
- The Encoder-Decoder Architecture
- The Context Vector — a Bottleneck
- Training with Teacher Forcing
- Inference: Generating One Step at a Time
- A Minimal Seq2Seq Example in PyTorch
- The Bottleneck Problem
- Summary & Next Steps
1. The Unaligned Many-to-Many Problem
Chapter 2, Section 6 identified "many-to-many (unaligned)" as a distinct pattern: mapping an input sequence to an output sequence of a different length, without a direct one-to-one correspondence between input and output positions. Machine translation is the canonical example: "I am a student" (4 words) translates to a French sentence that may have a different number of words entirely, in a different order.
2. The Encoder-Decoder Architecture
The standard solution splits the task into two separate recurrent networks (each built from Chapter 2-3's RNN/LSTM/GRU cells):
The Encoder-Decoder Pattern
─────────────────────────────────────────
ENCODER: reads the ENTIRE input sequence, one element
at a time, producing a final hidden state
that (ideally) summarizes the WHOLE input
Context vector: the encoder's FINAL hidden state — a
fixed-size vector meant to represent the
complete input sequence's meaning
DECODER: a SEPARATE recurrent network, initialized
with the encoder's context vector,
which generates the output sequence
ONE element at a time
─────────────────────────────────────────
ENCODER DECODER
"I"──►[LSTM]──►[LSTM]──►[LSTM]──►[LSTM] (context)
"am"───┘ │ │ │ │
"a"───────────────┘ │ │ ▼
"student"────────────────────┘ │ [LSTM]──►"Je"
│ │
final hidden ▼
state [LSTM]──►"suis"
│
▼
[LSTM]──►"étudiant"
3. The Context Vector — a Bottleneck
The context vector's job is compressing an entire input sequence — however long — into one fixed-size vector. This works reasonably well for short sequences, but becomes an increasingly severe bottleneck as input length grows (Section 7 returns to this in detail — it is precisely the limitation that motivates Module 6's attention mechanism).
4. Training with Teacher Forcing
During training, rather than feeding the decoder its own (possibly wrong, especially early in training) previous predictions, teacher forcing feeds it the true previous output token instead — stabilizing and speeding up training.
import numpy as np
def train_step_with_teacher_forcing(encoder, decoder, input_seq, target_seq, teacher_forcing_ratio=0.5):
context = encoder(input_seq) # Section 2's encoder pass
decoder_input = "<START>" # a special token marking the beginning of output
decoder_hidden = context
outputs = []
for t in range(len(target_seq)):
output, decoder_hidden = decoder(decoder_input, decoder_hidden)
outputs.append(output)
use_teacher_forcing = np.random.random() < teacher_forcing_ratio
decoder_input = target_seq[t] if use_teacher_forcing else output # KEY difference from inference
return outputsWhy NOT Always Use Teacher Forcing
─────────────────────────────────────────
ALWAYS feeding the TRUE previous token means the decoder
NEVER practices recovering from its OWN mistakes during
training — but at INFERENCE time (Section 5), there IS no
true previous token; the model must rely entirely on its
OWN previous predictions. Mixing teacher forcing with the
model's own predictions during training (a "ratio," as
above) helps bridge this training/inference mismatch.
─────────────────────────────────────────
5. Inference: Generating One Step at a Time
At inference time, there's no ground truth to feed the decoder — each predicted token becomes the input for generating the next one, a process called autoregressive generation.
def generate(encoder, decoder, input_seq, max_length=20):
context = encoder(input_seq)
decoder_input = "<START>"
decoder_hidden = context
generated_tokens = []
for _ in range(max_length):
output, decoder_hidden = decoder(decoder_input, decoder_hidden)
predicted_token = output.argmax() # greedy decoding — pick the SINGLE most likely token
if predicted_token == "<END>":
break
generated_tokens.append(predicted_token)
decoder_input = predicted_token # feed the PREDICTION back in as the next input
return generated_tokensThis exact autoregressive generation pattern — predict one token, feed it back in, predict the next — is the same mechanism used by modern large language models (covered in the NLP & LLM Notes), just built on Module 6's Transformer decoder instead of an RNN decoder.
6. A Minimal Seq2Seq Example in PyTorch
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, vocab_size, embed_size, hidden_size):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_size)
self.lstm = nn.LSTM(embed_size, hidden_size, batch_first=True)
def forward(self, x):
embedded = self.embedding(x)
_, (hidden, cell) = self.lstm(embedded)
return hidden, cell # the CONTEXT (Section 2-3)
class Decoder(nn.Module):
def __init__(self, vocab_size, embed_size, hidden_size):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_size)
self.lstm = nn.LSTM(embed_size, hidden_size, batch_first=True)
self.output_layer = nn.Linear(hidden_size, vocab_size)
def forward(self, x, hidden, cell):
embedded = self.embedding(x)
output, (hidden, cell) = self.lstm(embedded, (hidden, cell))
prediction = self.output_layer(output)
return prediction, hidden, cell
encoder = Encoder(vocab_size=1000, embed_size=64, hidden_size=128)
decoder = Decoder(vocab_size=1000, embed_size=64, hidden_size=128)
input_seq = torch.randint(0, 1000, (1, 5)) # a dummy 5-token input sequence
context_hidden, context_cell = encoder(input_seq)
decoder_input = torch.tensor([[1]]) # <START> token, e.g. index 1
prediction, _, _ = decoder(decoder_input, context_hidden, context_cell)
print(f"Decoder prediction shape: {prediction.shape}") # [1, 1, 1000] — a distribution over the vocabulary7. The Bottleneck Problem
Why Fixed-Size Context Vectors Struggle With Long Sequences
─────────────────────────────────────────
A 5-word sentence and a 50-word paragraph both get
compressed into the SAME fixed-size context vector. As
input length grows, more and more information must be
"squeezed" into that same fixed capacity — inevitably,
DETAILS get lost, particularly information from EARLY in
a long input sequence (by the time the encoder finishes
processing a long sequence, its hidden state has been
overwritten many times over).
This is a DIRECT, measurable limitation: seq2seq model
translation quality was empirically observed to degrade
noticeably as sentence length increased — a symptom
directly traceable to this fixed-size bottleneck.
─────────────────────────────────────────
This exact problem is what motivated the attention mechanism — instead of forcing the ENTIRE input into one fixed vector, why not let the decoder look back at ALL of the encoder's hidden states, and decide dynamically which parts of the input are most relevant at each decoding step? This is precisely where Chapter 5 and Module 6 pick up.
8. Summary & Next Steps
Key Takeaways
- Encoder-decoder architectures split sequence-to-sequence tasks into two networks: an encoder that compresses the input into a context vector, and a decoder that generates output autoregressively from it.
- Teacher forcing feeds the true previous output during training to stabilize learning, while inference must rely entirely on the model's own previous predictions (autoregressive generation).
- The context vector is a fixed-size bottleneck — as input sequences grow longer, compressing all relevant information into one fixed vector becomes increasingly lossy.
- This bottleneck is the direct motivation for the attention mechanism, covered next.
Concept Check
- Why can't the encoder-decoder architecture from Chapter 2's basic RNN directly solve unaligned many-to-many tasks like translation?
- What problem does teacher forcing solve during training, and why isn't it used during inference?
- Why does translation quality tend to degrade on longer sentences in a basic seq2seq model?
Next Chapter
→ Chapter 5: Limitations That Motivate Attention
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index