Neural Networks From Scratch
Gradient Descent & Its Variants
Gradient descent was already introduced in the ML Notes, Module 3, Chapter 1, for fitting linear regression's handful of parameters. The core idea is unchanged
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 4: Backpropagation Explained Time to complete: ~20 minutes
Table of Contents
- Recap: Gradient Descent from the ML Notes
- Batch, Stochastic, and Mini-Batch Gradient Descent
- The Learning Rate
- Implementing Mini-Batch Gradient Descent
- Epochs, Batches, and Iterations
- A Preview of Better Optimizers
- Summary & Next Steps
1. Recap: Gradient Descent from the ML Notes
Gradient descent was already introduced in the ML Notes, Module 3, Chapter 1, for fitting linear regression's handful of parameters. The core idea is unchanged here: repeatedly nudge every weight slightly in the direction that reduces the loss, using the gradients that Chapter 4's backpropagation computed.
The Weight Update Rule — IDENTICAL in Principle to ML Notes
─────────────────────────────────────────
W_new = W_old - (learning_rate × gradient)
The ONLY thing that changed moving from linear regression
to a deep network is HOW the gradient is computed
(backpropagation, Chapter 4, instead of a direct formula)
and HOW MANY parameters are being updated (millions,
instead of a handful).
─────────────────────────────────────────
2. Batch, Stochastic, and Mini-Batch Gradient Descent
Three Ways to Compute the Gradient Before Each Update
─────────────────────────────────────────
Batch Gradient Descent: compute the gradient using
the ENTIRE training dataset,
then take ONE update step —
accurate, but extremely slow
and memory-intensive for large
datasets
Stochastic Gradient compute the gradient using
Descent (SGD): just ONE random example,
then update — very fast
per step, but noisy and
erratic
Mini-Batch Gradient compute the gradient
Descent: using a SMALL batch
(e.g. 32-256
examples), then
update — the
practical standard,
balancing speed and
stability
─────────────────────────────────────────
Why Mini-Batches Won in Practice
─────────────────────────────────────────
1. Compute efficiency: GPUs (Module 1, Ch.4) are optimized
for parallel matrix operations —
processing a batch of, say, 64
examples at once is barely slower
than processing ONE, since the
GPU parallelizes across the batch
2. Gradient stability: averaging over a batch smooths out
the NOISE of single-example
gradients (pure SGD), without
the full COST of computing over
the entire dataset (pure batch)
3. Regularization effect: the noise from using SMALL
batches (rather than the
full dataset) can actually
help the model avoid poor
local minima — a mild,
often-beneficial side effect
─────────────────────────────────────────
3. The Learning Rate
The learning rate — already introduced conceptually in the ML Notes — is, if anything, an even more critical hyperparameter in deep learning, given how many more weight updates a deep network undergoes.
The Classic Learning Rate Tradeoff
─────────────────────────────────────────
Too LARGE: updates overshoot the minimum, loss
oscillates wildly or DIVERGES (increases
instead of decreasing)
Too SMALL: training converges extremely slowly,
potentially requiring impractically many
epochs, and can get stuck in poor local
regions
Just right: loss decreases steadily and reasonably
quickly toward a good minimum
─────────────────────────────────────────
Loss vs Training Step, for Different Learning Rates
─────────────────────────────────────────
Loss
│ \
│ \___ <- learning rate too small: slow, steady
│ \___
│ \_______________
│ \ /\ /\ /\ <- learning rate too large: oscillates
│ \/ \/ \/ \/
│ \ <- good learning rate: fast, smooth
│ \___
│ \________________
└──────────────────────────────► Training Step
─────────────────────────────────────────
Module 3, Chapter 5 covers learning rate schedules — strategies for adjusting the learning rate during training, rather than fixing it at a single value for the entire process.
4. Implementing Mini-Batch Gradient Descent
import numpy as np
def mini_batch_gradient_descent(X, y, W, b, learning_rate=0.01, batch_size=32, epochs=10):
n_samples = X.shape[0]
for epoch in range(epochs):
# Shuffle data EVERY epoch — prevents the network from learning
# any spurious pattern tied to the ORDER of examples
indices = np.random.permutation(n_samples)
X_shuffled, y_shuffled = X[indices], y[indices]
epoch_loss = 0
for start in range(0, n_samples, batch_size):
end = start + batch_size
X_batch, y_batch = X_shuffled[start:end], y_shuffled[start:end]
# Forward pass (Chapter 3) + backward pass (Chapter 4) would go here,
# producing dW and db for this batch — simplified here for illustration:
predictions = X_batch @ W + b
errors = predictions - y_batch
dW = (X_batch.T @ errors) / len(X_batch)
db = np.mean(errors)
# The weight update rule from Section 1
W -= learning_rate * dW
b -= learning_rate * db
epoch_loss += np.mean(errors ** 2)
print(f"Epoch {epoch+1}/{epochs}, Loss: {epoch_loss:.4f}")
return W, b# In PyTorch, this is handled by a DataLoader + a training loop (fully covered in Module 9)
import torch
from torch.utils.data import DataLoader, TensorDataset
X = torch.randn(1000, 4)
y = torch.randn(1000, 1)
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=32, shuffle=True) # handles Section 2's shuffling + batching automatically
for epoch in range(10):
for X_batch, y_batch in loader:
pass # forward pass, loss, backward(), optimizer step — Module 9 covers the full loop5. Epochs, Batches, and Iterations
Terminology, Precisely
─────────────────────────────────────────
Batch (or mini-batch): a small subset of the training
data used for ONE weight update
Iteration: ONE weight update — i.e.
processing ONE batch
Epoch: ONE full pass through the
ENTIRE training dataset
Example: 10,000 training examples, batch size 100
→ 100 iterations per epoch (10,000 / 100 = 100)
→ training for 20 epochs = 2,000 total iterations
─────────────────────────────────────────
6. A Preview of Better Optimizers
Plain mini-batch gradient descent, as implemented in Section 4, works but converges slowly and is sensitive to the learning rate choice from Section 3. Module 3, Chapter 2 covers modern optimizers — Momentum, RMSProp, and especially Adam — that adapt the effective learning rate automatically and converge substantially faster in practice. Every architecture from Module 4 onward is trained almost exclusively with Adam or one of its variants, not plain gradient descent as shown here.
7. Summary & Next Steps
Key Takeaways
- Gradient descent's core update rule is unchanged from the ML Notes; what's new is computing gradients via backpropagation across many layers, for many more parameters.
- Mini-batch gradient descent balances the compute efficiency of GPUs, gradient stability, and a mild regularization effect — making it the practical standard over pure batch or pure stochastic gradient descent.
- The learning rate remains one of the most important hyperparameters — too large causes divergence, too small causes impractically slow convergence.
- An epoch is one full pass through the training data; a batch is a subset used for one weight update (an iteration); these terms are used constantly in the rest of this curriculum.
Concept Check
- Why does mini-batch gradient descent tend to outperform both pure batch and pure stochastic gradient descent in practice?
- If you have 5,000 training examples and use a batch size of 50, how many iterations occur in one epoch?
- What symptom in the loss curve suggests the learning rate is too large?
Next Chapter
→ Chapter 6: Building a Neural Net From Scratch in NumPy
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index