Supervised Learning Classification
Evaluating Classification Models
Module 2, Chapter 5 already demonstrated the core problem: a model that never predicts fraud can be 99.5% "accurate" on an imbalanced dataset while being comple
Jr Codex ML Notes
Level: Intermediate–Advanced Prerequisites: Chapter 5 Time to complete: ~30 minutes
Table of Contents
- Why Accuracy Alone Is Dangerous
- The Confusion Matrix
- Precision
- Recall
- The Precision-Recall Trade-off
- F1-Score
- The ROC Curve and AUC
- Precision-Recall Curves — for Severe Imbalance
- Multi-Class Evaluation
- Choosing the Right Metric for Your Problem
- Summary & Next Steps
1. Why Accuracy Alone Is Dangerous
Module 2, Chapter 5 already demonstrated the core problem: a model that never predicts fraud can be 99.5% "accurate" on an imbalanced dataset while being completely useless. This chapter builds the full toolkit of metrics needed to evaluate classification models honestly, regardless of class balance.
from sklearn.metrics import accuracy_score
import numpy as np
y_actual = np.array([0]*95 + [1]*5) # 95% negative, 5% positive
y_predicted = np.array([0]*100) # NEVER predicts positive
print(f"Accuracy: {accuracy_score(y_actual, y_predicted):.3f}") # 0.95 — looks great, is USELESS2. The Confusion Matrix
The confusion matrix is the foundation every other metric in this chapter is built from — it breaks predictions down into four specific categories.
from sklearn.metrics import confusion_matrix
import numpy as np
y_actual = np.array([1, 0, 1, 1, 0, 0, 1, 0, 1, 0])
y_predicted = np.array([1, 0, 0, 1, 0, 1, 1, 0, 1, 0])
cm = confusion_matrix(y_actual, y_predicted)
print(cm)
# [[4 1]
# [1 4]]The Four Quadrants
─────────────────────────────────────────
Predicted Negative Predicted Positive
Actual Negative True Negative (TN) False Positive (FP)
"correctly "FALSE ALARM —
rejected" wrongly flagged"
Actual Positive False Negative (FN) True Positive (TP)
"MISSED — "correctly
should've been caught"
caught"
─────────────────────────────────────────
tn, fp, fn, tp = confusion_matrix(y_actual, y_predicted).ravel()
print(f"True Negatives: {tn}, False Positives: {fp}, False Negatives: {fn}, True Positives: {tp}")Every metric in this chapter is computed from these four numbers — understanding this matrix deeply is the single most important skill for evaluating any classifier honestly.
3. Precision
Precision answers: of everything the model flagged as positive, how much was actually positive?
from sklearn.metrics import precision_score
precision = precision_score(y_actual, y_predicted)
print(f"Precision: {precision:.3f}")
# Manually, from the confusion matrix:
manual_precision = tp / (tp + fp)
print(f"Manual precision: {manual_precision:.3f}")Precision Formula
─────────────────────────────────────────
Precision = TP / (TP + FP)
"When the model says YES, how often is it RIGHT?"
─────────────────────────────────────────
When Precision Matters Most
─────────────────────────────────────────
Spam filtering: a FALSE POSITIVE (flagging a real,
important email as spam) is genuinely
costly — the user might miss it entirely.
HIGH PRECISION matters most here.
─────────────────────────────────────────
4. Recall
Recall (also called sensitivity or true positive rate) answers: of everything that's actually positive, how much did the model catch?
from sklearn.metrics import recall_score
recall = recall_score(y_actual, y_predicted)
print(f"Recall: {recall:.3f}")
manual_recall = tp / (tp + fn)
print(f"Manual recall: {manual_recall:.3f}")Recall Formula
─────────────────────────────────────────
Recall = TP / (TP + FN)
"Of all the ACTUAL positives out there, how many did we FIND?"
─────────────────────────────────────────
When Recall Matters Most
─────────────────────────────────────────
Disease screening: a FALSE NEGATIVE (missing a real case
of a serious disease) can be
catastrophic. HIGH RECALL matters
most here — even at the cost of
more false alarms requiring follow-up.
─────────────────────────────────────────
5. The Precision-Recall Trade-off
Directly reprising Chapter 1's threshold discussion — precision and recall almost always trade off against each other as the classification threshold changes.
from sklearn.linear_model import LogisticRegression
import numpy as np
np.random.seed(0)
X = np.random.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
model = LogisticRegression().fit(X, y)
probabilities = model.predict_proba(X)[:, 1]
for threshold in [0.2, 0.5, 0.8]:
predictions = (probabilities >= threshold).astype(int)
p = precision_score(y, predictions)
r = recall_score(y, predictions)
print(f"Threshold={threshold}: Precision={p:.3f}, Recall={r:.3f}")Why They Trade Off
─────────────────────────────────────────
LOWER threshold: model predicts "positive" MORE often
→ catches MORE true positives (higher
recall), but also flags MORE false
positives (lower precision)
HIGHER threshold: model predicts "positive" LESS often
→ fewer false positives (higher
precision), but MISSES more true
positives (lower recall)
─────────────────────────────────────────
There is no threshold that maximizes both simultaneously — the right choice is a genuine business decision (spam filtering needs high precision; disease screening needs high recall), not a purely technical one, echoing the AI Notes' fairness trade-off discussions.
6. F1-Score
When you need a single number balancing precision and recall, the F1-score is the harmonic mean of the two.
from sklearn.metrics import f1_score
f1 = f1_score(y_actual, y_predicted)
print(f"F1-score: {f1:.3f}")
# Manually:
manual_f1 = 2 * (precision * recall) / (precision + recall)
print(f"Manual F1: {manual_f1:.3f}")Why the HARMONIC Mean, Not a Simple Average?
─────────────────────────────────────────
The harmonic mean punishes EXTREME imbalance between
precision and recall much more than a simple average would.
Example: precision=1.0, recall=0.01
Simple average: (1.0 + 0.01) / 2 = 0.505 ← looks
deceptively OK!
Harmonic mean (F1): 2×(1.0×0.01)/(1.0+0.01) = 0.0198
← correctly reveals this is a BAD
trade-off overall
─────────────────────────────────────────
from sklearn.metrics import fbeta_score
# F-beta generalizes F1, letting you WEIGHT recall more or less heavily
f2 = fbeta_score(y_actual, y_predicted, beta=2) # beta > 1 weights RECALL more heavily
f0_5 = fbeta_score(y_actual, y_predicted, beta=0.5) # beta < 1 weights PRECISION more heavily7. The ROC Curve and AUC
The ROC curve (Receiver Operating Characteristic) plots the trade-off between true positive rate (recall) and false positive rate across every possible threshold — a complete picture, rather than a single threshold's snapshot.
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt
fpr, tpr, thresholds = roc_curve(y, probabilities)
auc = roc_auc_score(y, probabilities)
plt.plot(fpr, tpr, label=f"ROC curve (AUC = {auc:.3f})")
plt.plot([0, 1], [0, 1], linestyle="--", color="gray", label="Random guessing")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate (Recall)")
plt.legend()Reading AUC (Area Under the Curve)
─────────────────────────────────────────
AUC = 1.0: PERFECT classifier — separates classes
completely at every threshold
AUC = 0.5: NO BETTER than random guessing (the
diagonal line)
AUC = 0.8-0.9: generally considered GOOD discrimination
ability
AUC < 0.5: WORSE than random — the model's
predictions are inverted somehow
(a red flag worth investigating)
─────────────────────────────────────────
AUC's key advantage: it's threshold-independent — it summarizes the model's ability to rank positive examples above negative ones across all possible thresholds, rather than committing to any one specific cutoff.
8. Precision-Recall Curves — for Severe Imbalance
For severely imbalanced problems (Module 2, Chapter 5), ROC-AUC can look misleadingly optimistic — the precision-recall curve is often more informative.
from sklearn.metrics import precision_recall_curve, average_precision_score
precision_vals, recall_vals, _ = precision_recall_curve(y, probabilities)
avg_precision = average_precision_score(y, probabilities)
plt.plot(recall_vals, precision_vals, label=f"PR curve (AP = {avg_precision:.3f})")
plt.xlabel("Recall")
plt.ylabel("Precision")Why ROC-AUC Can Mislead Under Severe Imbalance
─────────────────────────────────────────
ROC's FALSE POSITIVE RATE is computed relative to the
(very large) number of TRUE NEGATIVES — with severe
imbalance, even a LARGE number of false positives can look
like a SMALL rate relative to the huge negative class,
making the model look better than it deserves.
Precision-recall curves don't have this issue, since
PRECISION is computed relative to PREDICTED positives only —
directly connecting to Module 2, Chapter 5's guidance to
prefer precision/recall-based metrics for imbalanced problems.
─────────────────────────────────────────
9. Multi-Class Evaluation
Every metric in this chapter extends to more than two classes, typically by computing per-class scores and averaging.
from sklearn.metrics import classification_report
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)
model = LogisticRegression(max_iter=200).fit(X_train, y_train)
predictions = model.predict(X_test)
print(classification_report(y_test, predictions, target_names=iris.target_names))Averaging Strategies for Multi-Class Metrics
─────────────────────────────────────────
Macro average: compute the metric PER CLASS, then average
them EQUALLY — treats every class as
equally important, regardless of size
Weighted average: compute PER CLASS, then average WEIGHTED
by how many examples each class has —
dominant classes influence the overall
score more
Micro average: aggregate TP/FP/FN across ALL classes
FIRST, then compute the metric once
— effectively the same as accuracy
for multi-class single-label problems
─────────────────────────────────────────
Use macro averaging when every class matters equally regardless of frequency (e.g., a rare disease subtype matters as much as a common one); use weighted averaging when overall performance, dominated by the most frequent classes, is the priority.
10. Choosing the Right Metric for Your Problem
Decision Guide
─────────────────────────────────────────
Classes roughly BALANCED, no strong asymmetric cost
→ Accuracy is fine as a quick check, F1 for more nuance
FALSE POSITIVES are more costly (spam, fraud FLAGGING)
→ Prioritize PRECISION
FALSE NEGATIVES are more costly (disease screening,
safety-critical detection)
→ Prioritize RECALL
Need a SINGLE balanced number
→ F1-score (or F-beta, weighted toward whichever matters more)
Need to evaluate OVERALL ranking ability, independent of
any specific threshold
→ ROC-AUC (balanced data) or PR-AUC (imbalanced data,
Section 8)
SEVERELY imbalanced classes (Module 2, Ch.5)
→ Precision, recall, F1, and PR-AUC — NEVER rely on
accuracy or ROC-AUC alone
─────────────────────────────────────────
11. Summary & Next Steps
Key Takeaways
- The confusion matrix (TP, TN, FP, FN) is the foundation every classification metric builds on — understand it deeply before anything else in this chapter.
- Precision (of predicted positives, how many were correct) and recall (of actual positives, how many were caught) trade off against each other as the classification threshold changes — the right balance is a business decision, not purely technical.
- F1-score is the harmonic mean of precision and recall, appropriately punishing extreme imbalance between the two more than a simple average would.
- ROC-AUC summarizes threshold-independent ranking ability, but can look misleadingly good under severe class imbalance — precision-recall curves are more informative there.
- Multi-class metrics extend via macro (equal per-class weight) or weighted (frequency-weighted) averaging, chosen based on whether rare classes matter as much as common ones.
Module 4 Complete — Next Module
You now have five classification algorithms and the complete toolkit to evaluate any of them rigorously, avoiding the accuracy trap on imbalanced data. Module 5 builds on Chapter 3's decision trees specifically, showing how combining many trees together (ensembles) produces dramatically more powerful, stable models.
Concept Check
- Why does a spam filter typically prioritize precision, while a disease screening test typically prioritizes recall?
- Why is F1-score's harmonic mean more informative than a simple average of precision and recall?
- Why can ROC-AUC look misleadingly optimistic on a severely imbalanced dataset, and what metric is preferred instead?
Next Chapter
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index