Dl Engineering And Capstone
Debugging & Experiment Tracking
A traditional software bug usually causes a crash or an obviously wrong output. A deep learning "bug" often produces something much harder to spot: code that ru
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 2: GPUs, Mixed Precision & Scaling Training Time to complete: ~25 minutes
Table of Contents
- Why Deep Learning Debugging Is Different
- Overfitting a Tiny Batch First
- Systematic Debugging, Revisited
- Experiment Tracking Tools
- Logging Metrics in Practice
- Comparing Experiments Rigorously
- A Debugging and Tracking Checklist
- Summary & Next Steps
1. Why Deep Learning Debugging Is Different
A traditional software bug usually causes a crash or an obviously wrong output. A deep learning "bug" often produces something much harder to spot: code that runs without errors, but trains a model that's silently far worse than it should be — a subtly wrong loss function, a data leak, or a preprocessing mismatch (Module 7, Chapter 3, Section 5) rarely throws an exception.
2. Overfitting a Tiny Batch First
The single most valuable debugging technique in deep learning: before training on a full dataset, verify the model can overfit a tiny subset (e.g. just 5-10 examples) to near-zero loss.
import torch
def sanity_check_overfit_tiny_batch(model, criterion, optimizer, images, labels, num_iterations=200):
model.train()
for i in range(num_iterations):
optimizer.zero_grad()
outputs = model(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
if i % 50 == 0:
print(f"Iteration {i}: loss = {loss.item():.6f}")
final_loss = loss.item()
if final_loss > 0.01:
print("WARNING: model failed to overfit a tiny batch — likely a BUG, not a data/capacity issue")
else:
print("PASSED: model can overfit a tiny batch — the training LOOP itself is functioning correctly")Why This Works as a Diagnostic
─────────────────────────────────────────
A model with ENOUGH capacity (Module 2, Ch.1) should ALWAYS
be able to memorize a HANDFUL of examples perfectly — if it
CAN'T, the problem is almost certainly a genuine BUG in the
training loop itself (a wrong loss function, gradients not
flowing, labels misaligned with inputs, Module 2 Ch.6's
common zero_grad() mistake) — NOT a data quantity or model
capacity issue, which only become relevant concerns AFTER
this basic sanity check passes.
─────────────────────────────────────────
3. Systematic Debugging, Revisited
Module 3, Chapter 6, Section 6 introduced a debugging checklist for gradient-related problems. This section extends it with engineering-specific failure modes.
Additional Common Bugs, Beyond Module 3's Gradient Issues
─────────────────────────────────────────
Data/label misalignment: shuffling images and labels
INDEPENDENTLY (rather than
together) silently scrambles
which label belongs to which
image — the tiny-batch test
(Section 2) often catches this
Train/eval mode forgotten: Module 3, Ch.3-4's
model.train()/model.eval()
mismatch — BatchNorm and
Dropout behave incorrectly
Data leakage: preprocessing statistics
(normalization, Module
4 Ch.3 Sec.2) computed
on the FULL dataset
instead of training data
only — inflates
validation performance
artificially
Learning rate too loss oscillates or
high/too low: plateaus (Module 3,
Ch.5's discussion)
─────────────────────────────────────────
4. Experiment Tracking Tools
Manually noting hyperparameters and results in a notebook doesn't scale past a handful of experiments. TensorBoard and Weights & Biases (W&B) are the two dominant tools for tracking training runs systematically.
# TensorBoard — included with PyTorch, logs locally
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter("runs/experiment_1")
for epoch in range(num_epochs):
train_loss = train_one_epoch(model, train_loader, optimizer, criterion, device) # Chapter 1
val_loss, val_acc = validate(model, val_loader, criterion, device)
writer.add_scalar("Loss/train", train_loss, epoch)
writer.add_scalar("Loss/validation", val_loss, epoch)
writer.add_scalar("Accuracy/validation", val_acc, epoch)
writer.close()
# Then run: tensorboard --logdir=runs# Weights & Biases — cloud-based, includes hyperparameter tracking and comparison across runs
import wandb
wandb.init(project="image-classifier", config={
"learning_rate": 0.001,
"batch_size": 32,
"architecture": "ResNet50", # Module 4, Ch.2
"optimizer": "AdamW", # Module 3, Ch.2
})
for epoch in range(num_epochs):
train_loss = train_one_epoch(model, train_loader, optimizer, criterion, device)
val_loss, val_acc = validate(model, val_loader, criterion, device)
wandb.log({"train_loss": train_loss, "val_loss": val_loss, "val_accuracy": val_acc, "epoch": epoch})5. Logging Metrics in Practice
What to Log, at Minimum
─────────────────────────────────────────
Training AND validation loss, per epoch — the MINIMUM
EVERY epoch: needed to detect
overfitting (Module 3,
Ch.4)
Learning rate, per epoch: essential when using
a schedule (Module 3,
Ch.5) — confirms it's
actually changing as
expected
Gradient norms catches vanishing/
(occasionally): exploding gradients
(Module 3, Ch.6)
EARLY, before
wasting hours of
training time
Sample predictions for vision
(periodically): (Module 4)/
generation
(Module 8)
tasks,
VISUALLY
inspecting
outputs
catches
problems
numbers
alone might
miss
─────────────────────────────────────────
6. Comparing Experiments Rigorously
Directly echoing the ML Notes' statistical model comparison discipline — comparing two training runs meaningfully requires controlling for randomness (Chapter 1, Section 6's fixed seeds) and changing only ONE variable at a time.
A Common Mistake
─────────────────────────────────────────
Comparing "Run A: lr=0.001, batch_size=32" against
"Run B: lr=0.0001, batch_size=64" tells you NOTHING about
WHICH change (learning rate OR batch size) caused any
observed difference — change ONE hyperparameter at a time,
or use the ML Notes' systematic hyperparameter search
methods (Module 7, Ch.2-3) to explore multiple combinations
properly, rather than ad-hoc, uncontrolled comparisons.
─────────────────────────────────────────
7. A Debugging and Tracking Checklist
Before Trusting a Training Run's Results
─────────────────────────────────────────
☐ Model successfully overfits a tiny batch (Section 2)
☐ Random seed is fixed for reproducible comparison (Ch.1,
Sec.6)
☐ Train/eval mode switches correctly at the right points
(Module 3, Ch.3-4)
☐ Preprocessing statistics come from TRAINING data only
(Module 4, Ch.3)
☐ Experiment configuration (hyperparameters, architecture)
is LOGGED alongside results (Section 4)
☐ Only ONE variable changed between compared runs (Section 6)
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Deep learning bugs often produce silently degraded results rather than crashes, making the overfit-a-tiny-batch sanity check the single most valuable debugging technique.
- Common engineering bugs beyond gradient issues include data/label misalignment, forgotten train/eval mode switches, and data leakage from improperly scoped preprocessing statistics.
- TensorBoard and Weights & Biases both track training metrics, hyperparameters, and configurations systematically, replacing ad-hoc manual note-taking.
- Rigorous experiment comparison requires fixed random seeds and changing only one variable at a time between runs.
Concept Check
- Why does successfully overfitting a tiny batch of data indicate the training loop itself is functioning correctly?
- Name three engineering bugs (beyond gradient-related issues) that can silently degrade model performance without throwing an error.
- Why is it problematic to change both learning rate and batch size simultaneously when comparing two training runs?
Next Chapter
→ Chapter 4: Capstone — Building an Image Classifier End-to-End
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index