Neural Networks From Scratch
Backpropagation Explained
Chapter 3 showed how to compute a loss for a given set of weights. The remaining question — the one that stalled the field for over a decade (Module 1, Chapter
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 3: Forward Propagation & Loss Functions Time to complete: ~30 minutes
Table of Contents
- The Problem Backpropagation Solves
- The Chain Rule — the Core Idea
- Backpropagation Through a Single Layer
- Backpropagation Through the Full Network
- Implementing Backpropagation by Hand
- Automatic Differentiation in PyTorch
- Why This Chapter Matters Even Though PyTorch Automates It
- Summary & Next Steps
1. The Problem Backpropagation Solves
Chapter 3 showed how to compute a loss for a given set of weights. The remaining question — the one that stalled the field for over a decade (Module 1, Chapter 3) — is: how much should each individual weight change to reduce that loss?
For a network with millions of weights across many layers, this requires computing the gradient of the loss with respect to every single weight — that is, the partial derivative of the loss with respect to each one. Backpropagation is the algorithm that computes all of these efficiently, in a single backward pass through the network.
2. The Chain Rule — the Core Idea
Backpropagation is, at its core, a systematic application of the chain rule from calculus: if a change in x affects y, and a change in y affects z, then the chain rule tells you exactly how a change in x affects z, by multiplying the intermediate rates of change together.
The Chain Rule, Applied to a Simple 2-Step Computation
─────────────────────────────────────────
If: y = f(x) and z = g(y)
Then: dz/dx = (dz/dy) × (dy/dx)
In a neural network, EVERY layer is one more "step" in this
chain — the loss depends on the output layer, which depends
on the last hidden layer, which depends on the layer before
it, all the way back to the input. Backpropagation applies
the chain rule repeatedly, MULTIPLYING gradients together as
it moves backward, layer by layer.
─────────────────────────────────────────
3. Backpropagation Through a Single Layer
Consider the 2-layer network from Chapter 3: z1 = W1x + b1, a1 = relu(z1), z2 = W2a1 + b2, a2 = sigmoid(z2), then loss L = BCE(y, a2).
Working Backward, One Step at a Time
─────────────────────────────────────────
Step 1: dL/da2 — how loss changes with the output
(depends on the loss function's
own derivative)
Step 2: dL/dz2 = dL/da2 × da2/dz2 — apply the chain
rule through the
activation function
Step 3: dL/dW2 = dL/dz2 × dz2/dW2 = dL/dz2 × a1ᵀ
— the gradient for W2 — this is what
gradient descent (Chapter 5) will use
to UPDATE W2
Step 4: dL/da1 = dL/dz2 × dz2/da1 = W2ᵀ × dL/dz2
— CONTINUE the chain backward into the
hidden layer
Step 5: dL/dz1 = dL/da1 × da1/dz1 — through the hidden
layer's activation
Step 6: dL/dW1 = dL/dz1 × dz1/dW1 = dL/dz1 × xᵀ
— the gradient for W1
─────────────────────────────────────────
Notice the pattern: each layer's gradient computation reuses the gradient computed for the layer after it (dL/da1 depends on dL/dz2) — this reuse, moving backward through the network, is precisely what makes backpropagation efficient rather than recomputing everything from scratch for every single weight.
4. Backpropagation Through the Full Network
The General Algorithm
─────────────────────────────────────────
1. Run a FORWARD pass (Chapter 3) — compute and CACHE every
intermediate value (z's and a's at every layer)
2. Compute the loss's gradient with respect to the FINAL
output (Step 1 above)
3. Move BACKWARD through the network, layer by layer:
a. Compute this layer's WEIGHT gradient (needed for
Chapter 5's weight update)
b. Compute the gradient to PASS BACKWARD to the
previous layer (Step 4 above)
4. Once every layer's weight gradients are computed, hand
them to an optimizer (Chapter 5) to actually update
the weights
─────────────────────────────────────────
5. Implementing Backpropagation by Hand
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def sigmoid_derivative(z):
s = sigmoid(z)
return s * (1 - s)
def relu(z):
return np.maximum(0, z)
def relu_derivative(z):
return (z > 0).astype(float)
# Forward pass (from Chapter 3), keeping the cache
def forward(x, W1, b1, W2, b2):
z1 = W1 @ x + b1
a1 = relu(z1)
z2 = W2 @ a1 + b2
a2 = sigmoid(z2)
return a2, (x, z1, a1, z2, a2)
# Backward pass — implementing Section 3's six steps
def backward(y_true, cache, W2):
x, z1, a1, z2, a2 = cache
dL_da2 = -(y_true / a2 - (1 - y_true) / (1 - a2)) # BCE loss derivative
dL_dz2 = dL_da2 * sigmoid_derivative(z2) # Step 2
dL_dW2 = np.outer(dL_dz2, a1) # Step 3
dL_db2 = dL_dz2
dL_da1 = W2.T @ dL_dz2 # Step 4
dL_dz1 = dL_da1 * relu_derivative(z1) # Step 5
dL_dW1 = np.outer(dL_dz1, x) # Step 6
dL_db1 = dL_dz1
return dL_dW1, dL_db1, dL_dW2, dL_db2
# Run it end to end
np.random.seed(42)
W1, b1 = np.random.randn(3, 4) * 0.1, np.zeros(3)
W2, b2 = np.random.randn(1, 3) * 0.1, np.zeros(1)
x = np.array([0.5, 0.3, 0.9, 0.1])
y_true = np.array([1.0])
prediction, cache = forward(x, W1, b1, W2, b2)
gradients = backward(y_true, cache, W2)
print(f"Gradient for W1:\n{gradients[0]}")This hand-written implementation is exactly what PyTorch's .backward() call (Section 6) does automatically, for networks of any size and architecture — understanding it once, by hand, on a small example, is what makes every later chapter's use of .backward() meaningful rather than "magic."
6. Automatic Differentiation in PyTorch
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(4, 3),
nn.ReLU(),
nn.Linear(3, 1),
nn.Sigmoid(),
)
x = torch.tensor([0.5, 0.3, 0.9, 0.1])
y_true = torch.tensor([1.0])
prediction = model(x) # forward pass
loss = nn.BCELoss()(prediction, y_true) # compute loss
loss.backward() # THIS ONE LINE performs the ENTIRE backward pass from Section 5
# Every parameter's gradient is now populated automatically
for name, param in model.named_parameters():
print(f"{name}: gradient shape {param.grad.shape}")What Actually Happens Inside .backward()
─────────────────────────────────────────
PyTorch builds a COMPUTATIONAL GRAPH during the forward
pass — recording every operation performed on every tensor
with requires_grad=True (Module 1, Ch.4). Calling
.backward() walks this graph BACKWARD, applying the chain
rule automatically at every recorded operation — precisely
Section 3-4's algorithm, generalized to work for ANY
sequence of operations, not just the specific layers
hand-coded in Section 5.
─────────────────────────────────────────
7. Why This Chapter Matters Even Though PyTorch Automates It
The Practical Payoff of Understanding Backprop
─────────────────────────────────────────
Debugging: when a network "isn't learning," recognizing
symptoms of vanishing/exploding gradients
(Module 3, Ch.6) requires understanding
what's actually flowing backward
Architecture understanding WHY certain architectures
design intuition: (e.g. residual connections in ResNet,
Module 4) were specifically designed
to help gradients flow better
requires this foundation
Reading papers: virtually all deep learning research
papers reference gradient behavior
directly — this chapter is the
prerequisite for that vocabulary
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Backpropagation computes the gradient of the loss with respect to every weight in the network, using the chain rule applied repeatedly, moving backward layer by layer.
- Each layer's gradient computation reuses the gradient computed for the layer after it — this reuse is what makes the algorithm efficient.
- Implementing backpropagation by hand for a small network builds the intuition needed to understand PyTorch's automatic
.backward(), which performs the identical computation via a recorded computational graph. - Understanding backpropagation directly enables debugging training problems and understanding why certain architectural choices exist.
Concept Check
- What mathematical principle underlies backpropagation, and how is it applied moving from the output layer back to the input layer?
- Why does each layer's gradient computation depend on the gradient already computed for the layer after it?
- What does PyTorch's computational graph actually record during the forward pass?
Next Chapter
→ Chapter 5: Gradient Descent & Its Variants
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index