Evaluating Regression Models

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~25 minutes

Table of Contents

1. Why a Single "Accuracy" Doesn't Exist for Regression
2. Mean Absolute Error (MAE)
3. Mean Squared Error (MSE) & Root Mean Squared Error (RMSE)
4. MAE vs RMSE — Which to Use
5. R² (Coefficient of Determination)
6. Adjusted R²
7. Putting It Together: a Full Evaluation
8. Summary & Next Steps

1. Why a Single "Accuracy" Doesn't Exist for Regression

Classification (Module 4) has an intuitive notion of "percent correct." Regression predicts continuous numbers, where being "off by 2" versus "off by 200" are both technically "wrong" but to wildly different degrees — this chapter's metrics quantify how wrong, not simply right or wrong.

2. Mean Absolute Error (MAE)

The average absolute difference between actual and predicted values — the most directly interpretable regression metric.

from sklearn.metrics import mean_absolute_error
import numpy as np
 
y_actual = np.array([250000, 340000, 410000, 275000])
y_predicted = np.array([245000, 355000, 395000, 290000])
 
mae = mean_absolute_error(y_actual, y_predicted)
print(f"MAE: ${mae:,.2f}")      # average, the model is off by this much

MAE Formula
─────────────────────────────────────────
  MAE = (1/n) × Σ |actual_i - predicted_i|

  "On AVERAGE, how far off are our predictions, in the SAME
  units as the original target (dollars, for house prices)?"
─────────────────────────────────────────

MAE treats every error equally — an error of $20,000 contributes exactly twice as much to MAE as an error of $10,000, no more, no less.

3. Mean Squared Error (MSE) & Root Mean Squared Error (RMSE)

Directly reusing Chapter 1's least-squares loss function as an evaluation metric — squaring errors before averaging.

from sklearn.metrics import mean_squared_error
import numpy as np
 
mse = mean_squared_error(y_actual, y_predicted)
rmse = np.sqrt(mse)      # take the square root to get back to ORIGINAL units
 
print(f"MSE: {mse:,.2f}")      # in SQUARED dollars — hard to interpret directly
print(f"RMSE: ${rmse:,.2f}")     # back in dollars — directly comparable to MAE

MSE/RMSE Formula
─────────────────────────────────────────
  MSE = (1/n) × Σ (actual_i - predicted_i)²
  RMSE = √MSE

  Squaring means LARGE errors are penalized DISPROPORTIONATELY
  more than small ones (exactly Chapter 1's least-squares
  rationale, now applied as an evaluation choice rather than a
  training objective)
─────────────────────────────────────────

# Demonstrating WHY RMSE penalizes large errors more than MAE does
errors_consistent = np.array([10, 10, 10, 10])      # four errors of exactly 10 each
errors_one_big = np.array([1, 1, 1, 37])              # one huge error, rest tiny — SAME total sum
 
print(f"MAE (consistent errors): {np.mean(np.abs(errors_consistent)):.2f}")      # 10.0
print(f"MAE (one big error): {np.mean(np.abs(errors_one_big)):.2f}")                # 10.0 — IDENTICAL!
 
print(f"RMSE (consistent errors): {np.sqrt(np.mean(errors_consistent**2)):.2f}")      # 10.0
print(f"RMSE (one big error): {np.sqrt(np.mean(errors_one_big**2)):.2f}")               # 18.5 — MUCH higher!

This demonstrates precisely why MAE and RMSE can tell different stories about the same model — RMSE reveals the presence of occasional large errors that MAE, treating all errors equally, would completely hide.

4. MAE vs RMSE — Which to Use

Decision Guide
─────────────────────────────────────────
  Want a metric that's EASY to explain to non-technical
  stakeholders, directly in the target's units, treating all
  errors proportionally
      → MAE

  Care MORE about avoiding occasional LARGE errors than about
  many small ones (e.g. predicting hospital wait times, where
  one wildly-wrong prediction is much worse than several
  slightly-off ones)
      → RMSE (or MSE during training, since it's what OLS and
        gradient descent, Chapter 1, actually optimize)

  Data has SIGNIFICANT outliers, and you don't want a few
  extreme cases to dominate the evaluation
      → MAE (more robust to outliers — echoing Module 2, Ch.3's
        robust scaling discussion)
─────────────────────────────────────────

5. R² (Coefficient of Determination)

Answers a fundamentally different question than MAE/RMSE: what proportion of the variance in the target does the model explain?

from sklearn.metrics import r2_score
 
r2 = r2_score(y_actual, y_predicted)
print(f"R²: {r2:.3f}")

R² Formula and Interpretation
─────────────────────────────────────────
  R² = 1 - (Sum of Squared Errors / Total Variance in y)

  R² = 1.0    → PERFECT predictions, model explains 100% of
                  the variance
  R² = 0.0       → the model is NO BETTER than always predicting
                     the MEAN of y (a trivial baseline)
  R² < 0.0          → the model is WORSE than just predicting
                        the mean every time (a genuinely bad sign)
─────────────────────────────────────────

Direct Connection to the Data Science Notes
─────────────────────────────────────────
  R² is closely related to Data Science Notes, Module 1,
  Chapter 5's correlation coefficient — for SIMPLE linear
  regression with one feature, R² is EXACTLY the square of the
  Pearson correlation between x and y.
─────────────────────────────────────────

R² is unitless and scale-independent, unlike MAE/RMSE — this makes it useful for comparing model quality across different problems (e.g., "this house price model has R²=0.85, this stock prediction model has R²=0.40") in a way raw dollar-denominated errors cannot.

6. Adjusted R²

Plain R² has a subtle problem: it can only stay the same or increase as you add more features, even if those features are genuinely useless — directly echoing Module 2, Chapter 4's overfitting-via-too-many-features concern.

def adjusted_r2(r2, n_samples, n_features):
    """Penalizes R² for having MORE features, unless they genuinely help."""
    return 1 - (1 - r2) * (n_samples - 1) / (n_samples - n_features - 1)
 
# Comparing two models: one with 3 features, one with 10 (mostly useless) features
r2_simple_model = 0.75
r2_complex_model = 0.76      # barely improved despite 7 MORE features!
 
adj_r2_simple = adjusted_r2(r2_simple_model, n_samples=100, n_features=3)
adj_r2_complex = adjusted_r2(r2_complex_model, n_samples=100, n_features=10)
 
print(f"Simple model — R²: {r2_simple_model}, Adjusted R²: {adj_r2_simple:.3f}")
print(f"Complex model — R²: {r2_complex_model}, Adjusted R²: {adj_r2_complex:.3f}")
# Adjusted R² correctly reveals the complex model ISN'T actually better —
# its tiny R² improvement doesn't justify the extra features

Adjusted R² is the correct tool for comparing models with different numbers of features — plain R² alone would misleadingly favor the more complex model just for having more (even useless) inputs.

7. Putting It Together: a Full Evaluation

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
 
np.random.seed(42)
X = np.random.rand(200, 3)
y = 3*X[:,0] + 2*X[:,1] - X[:,2] + np.random.normal(0, 0.1, 200)
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
 
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
 
print(f"MAE:  {mean_absolute_error(y_test, predictions):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, predictions)):.4f}")
print(f"R²:   {r2_score(y_test, predictions):.4f}")

A Complete Evaluation Report, Interpreted
─────────────────────────────────────────
  MAE:  0.0821    → predictions are off by ~0.08, on average
  RMSE: 0.1024     → slightly higher than MAE — a FEW larger
                       errors exist, but not extreme
  R²:   0.9847        → the model explains ~98.5% of the
                          variance in y — a very strong fit
                          for this (synthetic, clean) example
─────────────────────────────────────────

Report multiple metrics together, not just one — MAE/RMSE describe error magnitude in real units; R² describes explained variance in a scale-free way; together, they give a much fuller picture than any single number.

8. Summary & Next Steps

Key Takeaways

• Regression has no single "accuracy" — MAE, RMSE, and R² each answer a different question about model quality.
• MAE treats all errors proportionally and is robust to outliers; RMSE penalizes large errors disproportionately more, matching what most regression algorithms actually optimize during training.
• R² measures the proportion of variance explained, ranging conceptually from a trivial mean-prediction baseline (0) to perfect (1) — unitless and comparable across different problems.
• Adjusted R² corrects plain R²'s tendency to reward additional features even when they don't genuinely improve the model, making it the right tool for comparing models with different feature counts.

Module 3 Complete — Next Module

You now have the complete regression toolkit: fitting linear models, regularizing them against overfitting, handling non-linear relationships, and evaluating results properly. Module 4 covers supervised learning's other branch — classification, predicting categories rather than numbers.

Concept Check

1. Why can MAE and RMSE give a model very different-looking scores despite describing the "same" errors?
2. What does an R² of 0 actually mean, in terms of the trivial baseline it's being compared against?
3. Why is adjusted R², not plain R², the right metric for comparing two models with different numbers of features?

Next Chapter

→ Module 4: Supervised Learning — Classification

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index