Sequence Models
Recurrent Neural Networks
A Recurrent Neural Network (RNN) processes a sequence one element at a time, maintaining a hidden state — a running summary of everything seen so far — that get
Jr Codex Deep Learning Notes
Level: Intermediate–Advanced Prerequisites: Chapter 1: Why Sequences Need Special Architectures Time to complete: ~25 minutes
Table of Contents
- The Core Idea: a Hidden State That Persists
- The RNN Cell, Step by Step
- Unrolling an RNN Through Time
- Backpropagation Through Time
- The Vanishing Gradient Problem, Revisited
- RNN Variants by Input/Output Shape
- RNNs in PyTorch
- Summary & Next Steps
1. The Core Idea: a Hidden State That Persists
A Recurrent Neural Network (RNN) processes a sequence one element at a time, maintaining a hidden state — a running summary of everything seen so far — that gets updated at every step and carried forward to the next.
The RNN's Defining Property
─────────────────────────────────────────
Unlike Module 2's MLP or Module 4's CNN, an RNN's output at
step t depends on BOTH the current input AND the hidden
state carried over from step t-1 — creating a form of
MEMORY across the sequence, directly addressing Chapter 1,
Section 4's "maintain order/position" requirement.
─────────────────────────────────────────
2. The RNN Cell, Step by Step
h(t-1) ──┐
├──► [RNN Cell] ──► h(t) ──► (passed to NEXT step)
x(t) ────┘ │
▼
output(t)
import numpy as np
def rnn_cell(x_t, h_prev, W_xh, W_hh, b_h):
"""
x_t: input at the current time step
h_prev: hidden state from the PREVIOUS time step
W_xh: weights for the INPUT->hidden connection
W_hh: weights for the PREVIOUS hidden->CURRENT hidden connection
"""
h_t = np.tanh(W_xh @ x_t + W_hh @ h_prev + b_h) # combines current input AND memory
return h_t
# A tiny example: processing a 3-step sequence, 2 features per step
np.random.seed(42)
hidden_size, input_size = 4, 2
W_xh = np.random.randn(hidden_size, input_size) * 0.1
W_hh = np.random.randn(hidden_size, hidden_size) * 0.1
b_h = np.zeros(hidden_size)
h = np.zeros(hidden_size) # initial hidden state — typically all zeros
sequence = [np.array([0.5, 0.3]), np.array([0.1, 0.9]), np.array([0.7, 0.2])]
for t, x_t in enumerate(sequence):
h = rnn_cell(x_t, h, W_xh, W_hh, b_h)
print(f"Step {t}: hidden state = {h}")Critically, the SAME weights (W_xh, W_hh, b_h) are reused at every time step — this weight sharing across time is what lets an RNN handle sequences of any length with a fixed number of parameters, directly solving Chapter 1, Section 3's variable-length problem.
3. Unrolling an RNN Through Time
Visualizing the same RNN cell applied repeatedly, once per time step, makes the flow of information explicit.
An RNN "Unrolled" Across a 3-Word Sequence
─────────────────────────────────────────
h0 ──►[Cell]──► h1 ──►[Cell]──► h2 ──►[Cell]──► h3
│ │ │
x1 x2 x3
("The") ("cat") ("sat")
Each [Cell] uses the SAME shared weights — this is NOT
three different networks, it's the SAME network applied
three times in sequence, with the hidden state carrying
information FORWARD between applications.
─────────────────────────────────────────
4. Backpropagation Through Time
Training an RNN uses Backpropagation Through Time (BPTT) — conceptually identical to Module 2, Chapter 4's backpropagation, but applied across the unrolled sequence from Section 3, treating each time step like an additional "layer."
BPTT, Conceptually
─────────────────────────────────────────
1. Run the FORWARD pass across the entire sequence (Section
2), producing a prediction/loss at each step (or just
the final step, depending on the task)
2. Compute gradients moving BACKWARD through time — from
the LAST time step back to the FIRST — accumulating
gradients for the SHARED weights at every step
3. Since the SAME weights are used at every step, their
TOTAL gradient is the SUM of the gradient contributions
from every individual time step
─────────────────────────────────────────
5. The Vanishing Gradient Problem, Revisited
Module 3, Chapter 6 previewed that RNNs are especially prone to vanishing (and exploding) gradients — BPTT makes clear exactly why: a sequence of length 100 is equivalent, for gradient purposes, to a 100-layer network, with the same weight matrix multiplied repeatedly.
Why Long Sequences Are Especially Vulnerable
─────────────────────────────────────────
Gradients flowing backward through BPTT get multiplied by
the SAME W_hh matrix at every step. If W_hh's dominant
characteristics push values below 1 (vanishing) or above 1
(exploding), this compounds identically to Module 3,
Chapter 6's deep-network analysis — except here, "depth" is
the SEQUENCE LENGTH, which can easily reach into the
hundreds or thousands for long documents or audio.
In practice, plain ("vanilla") RNNs struggle to retain
information from more than roughly 10-20 steps earlier —
far short of what's needed for many real sequences.
─────────────────────────────────────────
This specific limitation is exactly what motivates Chapter 3's LSTMs and GRUs.
6. RNN Variants by Input/Output Shape
Four Structural Patterns
─────────────────────────────────────────
One-to-many: a single input produces a SEQUENCE of
outputs — e.g. image captioning (one
image in, a sequence of words out)
Many-to-one: a sequence of inputs produces a SINGLE
output — e.g. sentiment classification
(a sentence in, one label out)
Many-to-many a sequence produces an EQUAL-length
(aligned): sequence of outputs — e.g.
labeling each word's part of
speech
Many-to-many a sequence produces a
(unaligned): DIFFERENT-length sequence
— e.g. machine translation
(Chapter 4's seq2seq
models handle this case
specifically)
─────────────────────────────────────────
7. RNNs in PyTorch
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
# (batch_size, sequence_length, input_size)
sequence = torch.randn(1, 5, 10) # 1 example, 5 time steps, 10 features per step
output, final_hidden = rnn(sequence)
print(f"Output at EVERY time step: {output.shape}") # [1, 5, 20]
print(f"Final hidden state ONLY: {final_hidden.shape}") # [1, 1, 20]
# A many-to-one classifier (Section 6), using ONLY the final hidden state
classifier = nn.Sequential(
nn.Linear(20, 2), # e.g. binary sentiment
)
prediction = classifier(final_hidden.squeeze(0))8. Summary & Next Steps
Key Takeaways
- An RNN maintains a hidden state, updated at every time step, that carries information forward through a sequence, using the same shared weights at every step.
- Training an RNN uses Backpropagation Through Time, which treats each time step like a layer in an unrolled network.
- Because the same weight matrix is applied repeatedly across time, plain RNNs are especially vulnerable to vanishing/exploding gradients on long sequences, limiting practical memory to roughly 10-20 steps.
- RNNs support several input/output shape patterns (one-to-many, many-to-one, many-to-many), covering a wide range of sequence tasks.
Concept Check
- Why does an RNN use the SAME weights at every time step, rather than different weights for each position?
- Why is a long sequence, for gradient purposes, analogous to a very deep feedforward network?
- Which RNN input/output pattern would you use for a sentiment classifier that reads a whole sentence and outputs one label?
Next Chapter
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index