Foundations Of Machine Learning
Bias-Variance Tradeoff, Overfitting & Underfitting
Every supervised learning algorithm in Modules 3-5 faces the same fundamental tension: a model needs to be complex enough to capture real patterns, but not so c
JrCodex·8 min read
Jr Codex ML Notes
Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~25 minutes
Table of Contents
- The Central Tension in All of Machine Learning
- Underfitting
- Overfitting
- Visualizing the Difference
- Bias and Variance, Defined Precisely
- The Bias-Variance Decomposition
- Model Complexity and the Sweet Spot
- Practical Strategies
- Summary & Next Steps
1. The Central Tension in All of Machine Learning
Every supervised learning algorithm in Modules 3-5 faces the same fundamental tension: a model needs to be complex enough to capture real patterns, but not so complex that it starts memorizing noise instead of learning genuine structure. This tension is called the bias-variance tradeoff, and understanding it deeply is arguably the single most important conceptual skill in this entire curriculum.
The Tension, Stated Simply
─────────────────────────────────────────
Too SIMPLE a model: misses real patterns in the data
→ UNDERFITTING (Section 2)
Too COMPLEX a model: memorizes noise/randomness specific
to the training data, mistaking it
for real signal
→ OVERFITTING (Section 3)
─────────────────────────────────────────
2. Underfitting
Underfitting occurs when a model is too simple to capture the real underlying pattern in the data — it performs poorly even on the data it was trained on.
import numpy as np
# Data with a clear CURVED (quadratic) relationship
np.random.seed(0)
X = np.linspace(-3, 3, 50)
y_true = X ** 2 + np.random.normal(0, 1, 50) # a curved relationship, plus noise
# UNDERFITTING: forcing a STRAIGHT LINE onto clearly curved data
# (Module 3's linear regression, applied to data it fundamentally
# can't represent well)
linear_fit_coefficients = np.polyfit(X, y_true, deg=1) # degree 1 = straight line
predictions = np.polyval(linear_fit_coefficients, X)
training_error = np.mean((y_true - predictions) ** 2)
print(f"Training error with underfit model: {training_error:.2f}") # HIGH — even on TRAINING dataSigns of Underfitting
─────────────────────────────────────────
- HIGH error on the TRAINING data itself (not just new data)
- The model is too SIMPLE relative to the true pattern's complexity
- Adding MORE training data does NOT meaningfully help — the
model's fundamental structure can't represent the pattern,
regardless of how much data it sees
─────────────────────────────────────────
3. Overfitting
Overfitting occurs when a model is so complex it starts fitting the noise specific to the training data, rather than the genuine underlying pattern — it performs excellently on training data but poorly on new, unseen data.
# OVERFITTING: forcing an EXTREMELY high-degree polynomial through
# the same data — it will pass almost exactly through every training
# point, including the RANDOM NOISE, not just the true pattern
overfit_coefficients = np.polyfit(X, y_true, deg=15) # degree 15 — way too flexible
overfit_predictions = np.polyval(overfit_coefficients, X)
training_error_overfit = np.mean((y_true - overfit_predictions) ** 2)
print(f"Training error with overfit model: {training_error_overfit:.2f}") # LOW — looks GREAT on training data!
# But test it on NEW data drawn from the SAME true relationship:
X_new = np.linspace(-3, 3, 50) + 0.05 # slightly different sample points
y_new_true = X_new ** 2 + np.random.normal(0, 1, 50)
new_predictions = np.polyval(overfit_coefficients, X_new)
test_error_overfit = np.mean((y_new_true - new_predictions) ** 2)
print(f"Test error with overfit model: {test_error_overfit:.2f}") # MUCH worse — failed to generalize!Signs of Overfitting
─────────────────────────────────────────
- LOW error on training data, but HIGH error on new/test data
(a big GAP between the two is the classic signature)
- The model has MEMORIZED specifics of the training set
(including its random noise) rather than learning the
general pattern
- Adding MORE training data often HELPS reduce overfitting,
since noise patterns don't repeat consistently across a
larger dataset, while the true signal does
─────────────────────────────────────────
This is precisely why Module 1, Chapter 3's train/test split exists — training accuracy alone cannot distinguish a genuinely good model from an overfit one; only performance on genuinely unseen data reveals the difference.
4. Visualizing the Difference
Underfitting Just Right Overfitting
───────────────── ───────────────── ─────────────────
● ● ● ● ●● ●●
──────── ╱‾╲ ╱ ╲ ╱ ╲
● ● ● ╱ ╲ ● ╱● ╲╱● ╲
(straight line (smooth curve (wiggly curve
through curved matching the passing through
data — misses TRUE pattern) EVERY point,
the curve) including noise)
High error on BOTH Low error on BOTH LOW error on training,
training AND test training AND test HIGH error on test
─────────────────────────────────────────
5. Bias and Variance, Defined Precisely
Bias
─────────────────────────────────────────
The ERROR introduced by approximating a real, possibly
complex relationship with a TOO-SIMPLE model.
HIGH bias = the model's fundamental assumptions (e.g. "this
relationship is a straight line") are WRONG for this problem
→ leads to UNDERFITTING (Section 2)
─────────────────────────────────────────
Variance
─────────────────────────────────────────
How much the model's predictions would CHANGE if trained on
a DIFFERENT sample of training data from the same underlying
distribution.
HIGH variance = the model is EXTREMELY sensitive to the
SPECIFIC training data it happened to see, including its
noise → leads to OVERFITTING (Section 3)
─────────────────────────────────────────
# Illustrating variance directly: train the SAME flexible model on
# several DIFFERENT random samples from the same true relationship
np.random.seed(1)
fits = []
for i in range(5):
X_sample = np.linspace(-3, 3, 20)
y_sample = X_sample ** 2 + np.random.normal(0, 2, 20) # same true pattern, different noise
coeffs = np.polyfit(X_sample, y_sample, deg=10) # a flexible, high-variance model
fits.append(coeffs)
# A HIGH-VARIANCE model's predictions at any given x will differ
# SUBSTANTIALLY across these 5 fits — sensitive to the specific
# noise in each sample. A LOW-VARIANCE model (like a straight
# line) would give much more CONSISTENT predictions across samples,
# even though it might be consistently WRONG (high bias) instead.6. The Bias-Variance Decomposition
A model's total expected error can be mathematically decomposed into three parts:
Total Error = Bias² + Variance + Irreducible Error
─────────────────────────────────────────
Bias²: error from WRONG assumptions
(model too simple)
Variance: error from SENSITIVITY to the
specific training sample
(model too complex/flexible)
Irreducible Error: noise INHERENT to the problem
itself — no model, however
good, can eliminate this
(e.g. genuinely random
measurement noise)
─────────────────────────────────────────
The Tradeoff, Visualized
─────────────────────────────────────────
Error
│ ╲ ╱
│ ╲ ╱
│ ╲ Bias² Variance
│ ╲______ ______╱
│ ╲ ╱
│ ╲ ╱ ← Total error is MINIMIZED
│ ╲╱ at some INTERMEDIATE
│ (sweet spot) complexity — NOT at either
└───────────────────────── extreme
Simple ← Model Complexity → Complex
─────────────────────────────────────────
This is why "make the model as complex as possible" and "make the model as simple as possible" are both wrong strategies — the goal is finding the complexity level that minimizes total error (on new data), which requires balancing bias against variance, not minimizing either one alone.
7. Model Complexity and the Sweet Spot
Where Different Algorithms (Later Modules) Sit on This Spectrum
─────────────────────────────────────────
Naturally LOWER variance, Naturally HIGHER variance,
HIGHER bias (simpler): LOWER bias (more flexible):
───────────────────────── ─────────────────────────
Linear Regression (Module 3) Deep Decision Trees (Module 4)
Logistic Regression (Module 4) K-Nearest Neighbors with
small k (Module 4)
Neural Networks (Deep
Learning Notes)
─────────────────────────────────────────
This is exactly why later modules discuss each algorithm's tendency toward bias or variance — Module 4's decision trees are explicitly called out as prone to overfitting (high variance) when grown too deep, and Module 5's ensemble methods exist specifically to reduce variance (bagging/random forests) or reduce bias (boosting) as targeted fixes.
8. Practical Strategies
Fighting Underfitting (High Bias)
─────────────────────────────────────────
- Use a MORE COMPLEX model (Module 4's decision trees or
ensembles instead of Module 3's plain linear regression)
- Add MORE or better FEATURES (Module 2's feature engineering)
- Reduce REGULARIZATION strength (Module 3, Ch.2) if it's
currently constraining the model too much
─────────────────────────────────────────
Fighting Overfitting (High Variance)
─────────────────────────────────────────
- Get MORE training data (reduces variance directly — noise
patterns don't repeat consistently across a larger sample)
- Use a SIMPLER model, or REGULARIZATION (Module 3, Ch.2)
- Use CROSS-VALIDATION (Chapter 5) to detect overfitting
reliably, not just eyeball training accuracy
- Use ENSEMBLE methods specifically designed to reduce
variance, like bagging/random forests (Module 5, Ch.1)
- Reduce the NUMBER of features, or apply dimensionality
reduction (Module 6, Ch.3-4)
─────────────────────────────────────────
9. Summary & Next Steps
Key Takeaways
- Underfitting means a model is too simple to capture real patterns, resulting in high error on both training and test data; overfitting means a model has memorized training-data noise, resulting in low training error but high test error.
- Bias is error from overly simplistic assumptions; variance is error from sensitivity to the specific training sample — total error decomposes into bias², variance, and irreducible error.
- The bias-variance tradeoff means minimizing total error requires balancing model complexity, not maximizing or minimizing it — there's a sweet spot, not an extreme.
- Different algorithms naturally sit at different points on the bias-variance spectrum, and specific techniques (regularization, ensembles, more data, cross-validation) target one side of the tradeoff or the other.
Concept Check
- What's the key difference in symptoms between an underfit model and an overfit model?
- Why can adding more training data help fix overfitting but not underfitting?
- Why is "always use the most complex model available" not a sound strategy, given the bias-variance tradeoff?
Next Chapter
→ Chapter 5: Train/Test Splits & Cross-Validation Basics
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index