Training Deep Networks Effectively
Vanishing/Exploding Gradients & Diagnosing Training Problems
This closing chapter connects every technique in this module back to the specific problem each one solves — vanishing and exploding gradients are the underlying
Jr Codex Deep Learning Notes
Level: Intermediate–Advanced Prerequisites: Chapter 5: Learning Rate Schedules & Warmup Time to complete: ~25 minutes
Table of Contents
- Tying Together This Module's Threads
- The Vanishing Gradient Problem, Formally
- The Exploding Gradient Problem
- Gradient Clipping
- Residual Connections — a Structural Fix
- A Systematic Debugging Checklist
- Diagnosing in PyTorch
- Summary & Next Steps
1. Tying Together This Module's Threads
This closing chapter connects every technique in this module back to the specific problem each one solves — vanishing and exploding gradients are the underlying failure modes that weight initialization, optimizers, normalization, and regularization all partially address, from different angles.
How This Module's Chapters Connect to Gradient Flow
─────────────────────────────────────────
Ch.1 (Initialization): poor scale directly causes
vanishing/exploding activations,
which causes vanishing/exploding
gradients during backprop
Ch.2 (Optimizers): Adam's adaptive per-parameter
rates partially compensate for
uneven gradient scales
Ch.3 (Normalization): keeps activations (and
therefore gradients) in a
stable range THROUGHOUT
training, not just at
initialization
Ch.4 (Regularization): indirectly helps by
discouraging extreme
weight values that
would otherwise produce
extreme gradients
─────────────────────────────────────────
2. The Vanishing Gradient Problem, Formally
Module 2, Chapter 4 showed that backpropagation multiplies gradients together, layer by layer, moving backward. If each layer's gradient contribution is consistently less than 1, the product shrinks exponentially with depth.
Why This Is Exponential, Not Linear
─────────────────────────────────────────
If each of 20 layers contributes a gradient factor of just
0.9 (seemingly not that small!):
0.9^20 ≈ 0.12 — already a 8x reduction
If each layer contributes 0.5 (sigmoid's derivative, Module
2 Ch.2, is often smaller than this at typical activation
values):
0.5^20 ≈ 0.0000009 — the gradient reaching early
layers is essentially ZERO
Early layers effectively STOP LEARNING — their weights
barely change from their random initialization, no matter
how long training continues.
─────────────────────────────────────────
Symptoms of Vanishing Gradients
─────────────────────────────────────────
- Training loss plateaus early and doesn't improve further
- Early/lower layers' weights barely change from their
initial (random) values over the course of training
- The problem gets WORSE as network depth increases
- Sigmoid/tanh-heavy networks (Module 2, Ch.2) are
especially prone to this
─────────────────────────────────────────
3. The Exploding Gradient Problem
The opposite failure: if each layer's gradient contribution is consistently greater than 1, the product grows exponentially instead of shrinking.
Symptoms of Exploding Gradients
─────────────────────────────────────────
- Loss suddenly jumps to a very large value or NaN
(not-a-number) during training
- Weight values become extremely large, or NaN
- Training that was progressing normally suddenly
"diverges" without warning
- Especially common in RNNs (Module 5) processing LONG
sequences — the SAME weight matrix is applied
repeatedly at every time step, compounding the effect
─────────────────────────────────────────
4. Gradient Clipping
Gradient clipping directly addresses exploding gradients by capping the gradient's magnitude before it's used to update weights.
import numpy as np
def clip_gradient_by_norm(gradient, max_norm=1.0):
gradient_norm = np.linalg.norm(gradient)
if gradient_norm > max_norm:
gradient = gradient * (max_norm / gradient_norm) # SCALE DOWN, preserving direction
return gradient
large_gradient = np.array([10.0, 15.0, -20.0])
clipped = clip_gradient_by_norm(large_gradient, max_norm=1.0)
print(f"Original norm: {np.linalg.norm(large_gradient):.2f}")
print(f"Clipped norm: {np.linalg.norm(clipped):.2f}") # exactly 1.0import torch
import torch.nn as nn
model = nn.Linear(10, 1)
# ... loss.backward() computes gradients ...
# Clip gradients BEFORE the optimizer step — standard practice for RNNs (Module 5) especially
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# optimizer.step() # now safe to applyGradient clipping is a near-universal default when training RNNs (Module 5), and is commonly applied when training Transformers (Module 6) as well, as cheap insurance against occasional gradient spikes.
5. Residual Connections — a Structural Fix
Beyond clipping (which treats a symptom), residual connections (introduced with ResNet, covered in Module 4, Chapter 2) are a structural fix to vanishing gradients — they add a "shortcut" path that lets gradients flow backward without necessarily passing through every intermediate layer's potentially-shrinking transformation.
The Residual Connection Idea (Preview of Module 4)
─────────────────────────────────────────
Standard layer: output = F(input)
Residual layer: output = F(input) + input
────────────────┬─────
the "SHORTCUT" —
during backprop, the
gradient can flow
through this shortcut
UNCHANGED, bypassing
F's potentially small
gradient contribution
─────────────────────────────────────────
This single idea is what allowed networks to scale from dozens of layers to hundreds — directly addressing this chapter's core problem at the architectural level, rather than only through training techniques.
6. A Systematic Debugging Checklist
When Training Isn't Working — Check in This Order
─────────────────────────────────────────
1. Loss is NaN immediately: check for a learning
rate that's too large
(Chapter 5), or apply
gradient clipping
(Section 4)
2. Loss doesn't decrease AT ALL: check weight
initialization
(Chapter 1) — verify
it isn't all-zero
(the symmetry
problem)
3. Loss decreases then PLATEAUS check for
early: vanishing
gradients
(Section 2)
— consider
ReLU (Module
2, Ch.2),
better
initialization
(Ch.1), or
normalization
(Ch.3)
4. Training loss LOW, validation OVERFITTING
loss HIGH: — apply
Chapter
4's
regularization
5. Both training AND validation check
loss are HIGH: for a
data
or
labeling
bug,
or an
architecture
too
simple
for
the
problem
─────────────────────────────────────────
7. Diagnosing in PyTorch
import torch
def check_gradient_health(model):
"""Run AFTER loss.backward(), BEFORE optimizer.step()"""
for name, param in model.named_parameters():
if param.grad is not None:
grad_norm = param.grad.norm().item()
if grad_norm < 1e-7:
print(f"WARNING: {name} has a near-ZERO gradient ({grad_norm:.2e}) — possible vanishing gradient")
elif grad_norm > 100:
print(f"WARNING: {name} has a very LARGE gradient ({grad_norm:.2e}) — possible exploding gradient")Tools like TensorBoard or Weights & Biases (covered in Module 9, Chapter 3) can track gradient norms like this automatically, across every training run, rather than checking manually.
8. Summary & Next Steps
Key Takeaways
- Vanishing and exploding gradients are the underlying failure modes that this module's earlier chapters (initialization, optimizers, normalization, regularization) all partially address.
- Vanishing gradients occur when per-layer gradient contributions are consistently below 1, causing an exponential shrinkage with depth; exploding gradients occur when they're consistently above 1.
- Gradient clipping caps gradient magnitude before the weight update, and is a near-standard default for training RNNs.
- Residual connections provide a structural fix, letting gradients bypass potentially-shrinking transformations entirely — a key enabler of very deep networks like ResNet.
Module 3 Complete — What's Next
You now have the complete toolkit for training deep networks reliably: proper initialization, modern optimizers, normalization, regularization, learning rate scheduling, and gradient diagnosis. Module 4 applies all of this to the first major architecture family: Convolutional Neural Networks for computer vision.
Concept Check
- Why is the vanishing gradient problem exponential rather than linear in the number of layers?
- What's the difference between how gradient clipping and residual connections address gradient problems — one is a training technique, the other structural. Which is which?
- If training loss is very low but validation loss is high, which of this module's techniques would you reach for first?
Next Module
→ Module 4: Convolutional Neural Networks
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index