Training Deep Networks Effectively
Regularization: Dropout, Weight Decay, Early Stopping
The ML Notes' bias-variance tradeoff applies directly here: a model that fits training data extremely well but generalizes poorly to new data has high variance
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 3: Batch Normalization & Layer Normalization Time to complete: ~20 minutes
Table of Contents
- Recap: Overfitting from the ML Notes
- Why Deep Networks Overfit Especially Easily
- Dropout
- Why Dropout Works
- Weight Decay (L2 Regularization)
- Early Stopping
- Data Augmentation as Regularization
- Regularization in PyTorch
- Summary & Next Steps
1. Recap: Overfitting from the ML Notes
The ML Notes' bias-variance tradeoff applies directly here: a model that fits training data extremely well but generalizes poorly to new data has high variance — it has essentially memorized noise and idiosyncrasies specific to the training set, rather than learning genuinely general patterns.
2. Why Deep Networks Overfit Especially Easily
Deep Networks Are Especially Prone to Overfitting Because:
─────────────────────────────────────────
1. HUGE parameter counts (Module 2, Ch.1) — millions or
billions of weights provide enormous capacity to
memorize training examples individually
2. Trained for MANY epochs — repeated exposure to the same
training examples gives ample opportunity to memorize
specific quirks
3. Real-world data (images, text) often contains far more
PATTERNS than a small training set can representatively
capture — the network may latch onto spurious
correlations unique to the training set
─────────────────────────────────────────
This module's regularization techniques exist specifically to counteract this tendency, complementing the ML Notes' regularization chapter (Ridge/Lasso) with techniques specific to neural networks.
3. Dropout
Dropout is deep learning's most distinctive regularization technique: during training, it randomly "turns off" (zeroes out) a fraction of neurons in a layer, on every forward pass.
import numpy as np
def dropout(activations, dropout_rate=0.5, training=True):
if not training:
return activations # dropout is DISABLED during inference (Section 4 explains why)
mask = np.random.binomial(1, 1 - dropout_rate, size=activations.shape)
return activations * mask / (1 - dropout_rate) # SCALE UP surviving activations (see below)
activations = np.array([0.5, 1.2, 0.8, 2.1, 0.3])
dropped_out = dropout(activations, dropout_rate=0.5, training=True)
print(f"Original: {activations}")
print(f"After dropout: {dropped_out}") # roughly HALF the values are now zeroWhy "Scale Up Surviving Activations" (Inverted Dropout)
─────────────────────────────────────────
If HALF the neurons are randomly zeroed during training,
the SUM of activations flowing to the next layer is roughly
HALF what it would be without dropout. Dividing surviving
activations by (1 - dropout_rate) compensates for this,
keeping the EXPECTED total activation magnitude consistent
between training and inference (when dropout is disabled
entirely, per Section 4).
─────────────────────────────────────────
4. Why Dropout Works
The Intuition
─────────────────────────────────────────
With a RANDOM subset of neurons disabled on every forward
pass, no single neuron can "rely" on any specific OTHER
neuron always being present — this forces the network to
learn REDUNDANT, more ROBUST representations, rather than
brittle co-dependencies between specific neurons that
happen to work well together on the TRAINING set
specifically.
A common analogy: dropout is similar to training a large
ENSEMBLE of many smaller, overlapping sub-networks
simultaneously (echoing the ML Notes' ensemble learning
module) — at inference time (with dropout disabled), the
full network behaves somewhat like an AVERAGE over all
those sub-networks.
─────────────────────────────────────────
Dropout is only active during training — at inference time, the network uses all neurons (with the compensating scale-up from Section 3 already baked in), producing deterministic, reproducible predictions.
5. Weight Decay (L2 Regularization)
Weight decay directly extends the ML Notes' Ridge regression concept to neural networks: it adds a penalty term to the loss proportional to the sum of squared weights, discouraging any single weight from growing excessively large.
def loss_with_weight_decay(base_loss, weights, weight_decay_lambda=0.01):
l2_penalty = weight_decay_lambda * sum(np.sum(W ** 2) for W in weights)
return base_loss + l2_penaltyWhy Smaller Weights Generalize Better
─────────────────────────────────────────
Large weights let a network respond VERY sharply to small
input changes — a hallmark of having memorized specific
training examples rather than learned smooth, general
patterns. Penalizing large weights encourages SMOOTHER
functions that are more likely to generalize to new,
unseen inputs.
─────────────────────────────────────────
6. Early Stopping
Early stopping monitors validation loss (ML Notes' train/validation/test discipline) during training, and halts training once validation loss stops improving — even if training loss is still decreasing.
Training vs Validation Loss Over Time (Typical Overfitting Pattern)
─────────────────────────────────────────
Loss
│ Training loss: \____________________
│ (keeps decreasing)
│ Validation loss: \______/‾‾‾‾‾‾‾‾‾‾‾‾
│ ↑
│ STOP HERE — validation
│ loss has started INCREASING,
│ even though training loss
│ keeps dropping (memorization)
└──────────────────────────────────────► Epoch
─────────────────────────────────────────
best_val_loss = float("inf")
patience = 5 # how many epochs to wait for improvement before stopping
epochs_without_improvement = 0
for epoch in range(100):
# train_loss = ... (one epoch of training)
# val_loss = ... (evaluate on validation set)
val_loss = 0.5 # placeholder
if val_loss < best_val_loss:
best_val_loss = val_loss
epochs_without_improvement = 0
# save_checkpoint(model) # save the BEST model seen so far
else:
epochs_without_improvement += 1
if epochs_without_improvement >= patience:
print(f"Early stopping at epoch {epoch}")
break7. Data Augmentation as Regularization
Artificially expanding the effective training set — covered in depth for images in Module 4, Chapter 5 — is itself a powerful regularization technique: a network exposed to many slightly varied versions of each training example is far less likely to memorize any one specific version.
8. Regularization in PyTorch
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Dropout(p=0.5), # Section 3 — 50% dropout rate
nn.Linear(256, 128),
nn.ReLU(),
nn.Dropout(p=0.3),
nn.Linear(128, 10),
)
# Weight decay (Section 5) — passed directly to the OPTIMIZER, not the loss function
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)
# CRITICAL: switch modes correctly (echoing Chapter 3's BatchNorm train/eval distinction)
model.train() # dropout IS active
# ... training loop ...
model.eval() # dropout is DISABLED — all neurons active, no scaling needed9. Summary & Next Steps
Key Takeaways
- Deep networks' huge parameter counts and long training make them especially prone to overfitting, directly connecting to the ML Notes' bias-variance tradeoff.
- Dropout randomly disables neurons during training, forcing the network to learn redundant, robust representations rather than brittle co-dependencies; it's disabled entirely at inference time.
- Weight decay penalizes large weights (extending the ML Notes' Ridge regression concept), encouraging smoother, more generalizable functions.
- Early stopping halts training when validation loss stops improving, even if training loss continues to decrease — directly preventing the network from over-memorizing training data.
Concept Check
- Why must dropout scale up the surviving activations during training, and why is dropout disabled entirely during inference?
- How does weight decay relate to the Ridge regression regularization covered in the ML Notes?
- What signal in the training vs validation loss curves indicates it's time to apply early stopping?
Next Chapter
→ Chapter 5: Learning Rate Schedules & Warmup
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index