Model Selection Best Practices

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 3: Bayesian Optimization & Advanced Tuning Time to complete: ~25 minutes

Table of Contents

1. Bringing This Module Together
2. Start Simple — Always
3. The Multiple Comparisons Problem in Model Selection
4. Statistical Significance of Model Differences
5. Don't Over-Tune on a Small Validation Set
6. Practical vs Statistical Significance, Revisited
7. A Complete, Correct Model Selection Workflow
8. Summary & Next Steps

1. Bringing This Module Together

Chapters 1-3 covered increasingly sophisticated tuning techniques. This chapter covers the discipline surrounding how to use them correctly — common pitfalls that undermine even technically correct tuning and cross-validation.

2. Start Simple — Always

Before reaching for Module 5's ensembles or Chapter 3's Bayesian optimization, establish a simple baseline first.

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
 
# ALWAYS start with a trivial baseline
dummy = DummyClassifier(strategy="most_frequent")      # always predicts the MAJORITY class
dummy_scores = cross_val_score(dummy, X_train, y_train, cv=5, scoring="f1")
print(f"Dummy baseline F1: {dummy_scores.mean():.3f}")
 
# Then a SIMPLE, interpretable model
simple_model = LogisticRegression()
simple_scores = cross_val_score(simple_model, X_train, y_train, cv=5, scoring="f1")
print(f"Logistic regression F1: {simple_scores.mean():.3f}")
 
# ONLY THEN consider more complex models, and ask: is the improvement WORTH the added complexity?

Why Starting Simple Is a Genuine Best Practice
─────────────────────────────────────────
  - A dummy baseline reveals whether the problem is even
    LEARNABLE with the available features at all
  - A simple model's performance is the BAR any complex model
    must clear to justify its added complexity, training time,
    and reduced interpretability (echoes the AI Notes'
    Explainable AI trade-off)
  - Directly connects to Module 1, Chapter 4: complex models
    aren't automatically better — they trade bias for variance,
    and that trade isn't always worth it
─────────────────────────────────────────

3. The Multiple Comparisons Problem in Model Selection

Directly reprising the Data Science Notes' multiple comparisons chapter — trying many models/hyperparameter combinations and picking whichever scores highest on validation risks selecting a combination that got lucky, not genuinely best.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.dummy import DummyClassifier
 
# Simulating: trying 50 RANDOM hyperparameter combinations on a problem
# where NONE of them genuinely differ in quality
np.random.seed(0)
scores = [cross_val_score(DummyClassifier(strategy="stratified"), X_train, y_train, cv=5).mean()
          for _ in range(50)]
 
print(f"Best of 50 'random' trials: {max(scores):.3f}")
print(f"Average of all 50 trials: {np.mean(scores):.3f}")
# The BEST of 50 trials will look notably better than the AVERAGE,
# purely from LUCK — even though every trial used the SAME
# underlying (non-learning) strategy

Why This Matters for Hyperparameter Search (Chapters 2-3)
─────────────────────────────────────────
  The MORE combinations you try (Chapter 2's grid/random
  search, Chapter 3's Bayesian optimization), the MORE
  likely you are to find one that scores well on your
  validation folds PURELY BY CHANCE, not because it's
  genuinely better.

  This is EXACTLY why Chapter 1's nested cross-validation
  and this chapter's Section 5 matter — an honest FINAL test
  set evaluation is what catches this "got lucky during
  tuning" problem.
─────────────────────────────────────────

4. Statistical Significance of Model Differences

Directly applying the Data Science Notes' hypothesis testing chapter to model comparison — is Model A genuinely better than Model B, or is the observed difference just noise?

from scipy import stats
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
import numpy as np
 
model_a = LogisticRegression()
model_b = RandomForestClassifier(random_state=42)
 
scores_a = cross_val_score(model_a, X_train, y_train, cv=10)
scores_b = cross_val_score(model_b, X_train, y_train, cv=10)
 
print(f"Model A mean: {scores_a.mean():.3f}, Model B mean: {scores_b.mean():.3f}")
 
# A PAIRED t-test — comparing the SAME folds for both models directly
t_stat, p_value = stats.ttest_rel(scores_a, scores_b)
print(f"p-value: {p_value:.4f}")
 
if p_value < 0.05:
    print("The difference IS likely statistically significant")
else:
    print("The difference could plausibly be due to CHANCE alone")

Why a PAIRED Test, Specifically
─────────────────────────────────────────
  Both models were evaluated on the SAME 10 folds — a paired
  test accounts for this shared structure (some folds are
  just "harder" than others for EVERY model), directly
  mirroring the Data Science Notes' hypothesis testing chapter's
  emphasis on choosing the test that matches the data's actual structure.
─────────────────────────────────────────

5. Don't Over-Tune on a Small Validation Set

A subtle risk connecting Sections 3-4 directly: with a small validation set, even genuinely meaningless hyperparameter differences can appear to matter, simply due to noise.

The Sample Size Connection (Directly from the Data Science Notes)
─────────────────────────────────────────
  Module 1, Chapter 4 of the Data Science Notes established:
  smaller samples have LARGER standard error — noisier estimates.

  A validation set of 50 examples gives a MUCH noisier
  performance estimate than one of 5,000 — tuning aggressively
  against a SMALL validation set risks chasing NOISE, not
  genuine hyperparameter improvements.
─────────────────────────────────────────

Practical mitigation: the smaller your validation data, the more you should prefer simpler models and fewer tuning iterations (Chapters 2-3) — an elaborate Bayesian optimization search over a tiny validation set is likely to overfit to that set's specific noise, not find genuinely better hyperparameters.

6. Practical vs Statistical Significance, Revisited

Directly echoing the Data Science Notes' statistical-vs-practical-significance chapter — a statistically significant difference between two models isn't automatically worth acting on.

# A genuinely tiny, but "statistically significant" (given enough folds/data) difference
model_a_score = 0.8500
model_b_score = 0.8503      # a 0.03 PERCENTAGE POINT improvement
 
# Is this worth: the added complexity of Model B (Module 5's ensembles vs
# Module 4's simpler algorithms), longer training time, harder deployment
# (Module 9), reduced interpretability (AI Notes' Explainable AI chapter)?
#
# Often, NO — the practical cost of complexity outweighs a marginal gain

The Decision Isn't Purely About the Number
─────────────────────────────────────────
  A genuinely useful model selection process weighs:
    - The SIZE of the performance improvement (practical
      significance, not just p < 0.05)
    - The COST of added complexity (training time, maintenance,
      interpretability — Module 9's production concerns)
    - The BUSINESS context (does this specific improvement
      actually matter for the problem being solved?)
─────────────────────────────────────────

7. A Complete, Correct Model Selection Workflow

Bringing this entire module together into one coherent, defensible process:

from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
from scipy import stats
 
# 1. Split: TRAIN+VALIDATION vs a LOCKED-AWAY test set (Module 1, Ch.5)
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
 
# 2. Establish baselines (Section 2)
dummy_score = cross_val_score(DummyClassifier(strategy="most_frequent"), X_trainval, y_trainval, cv=5).mean()
simple_score = cross_val_score(LogisticRegression(), X_trainval, y_trainval, cv=5).mean()
print(f"Dummy: {dummy_score:.3f}, Logistic Regression: {simple_score:.3f}")
 
# 3. Tune a MORE complex candidate, using appropriate CV strategy (Chapter 1)
# and search method (Chapter 2-3)
search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    {"n_estimators": randint(50, 300), "max_depth": randint(3, 20)},
    n_iter=30, cv=5, random_state=42, n_jobs=-1,
)
search.fit(X_trainval, y_trainval)
 
# 4. Compare candidates STATISTICALLY (Section 4), not just by raw score
rf_scores = cross_val_score(search.best_estimator_, X_trainval, y_trainval, cv=10)
lr_scores = cross_val_score(LogisticRegression(), X_trainval, y_trainval, cv=10)
_, p_value = stats.ttest_rel(rf_scores, lr_scores)
print(f"Is the improvement significant? p={p_value:.4f}")
 
# 5. Weigh PRACTICAL significance (Section 6) — is the gain worth the complexity?
# 6. ONLY THEN, evaluate the FINAL chosen model on the untouched test set — ONCE
final_model = search.best_estimator_
final_test_score = final_model.score(X_test, y_test)
print(f"Final, honest test score: {final_test_score:.3f}")

8. Summary & Next Steps

Key Takeaways

• Always establish a dummy baseline and a simple model's performance before investing in complex tuning — they define the bar any added complexity must clear.
• Trying many hyperparameter combinations risks selecting one that performed well by chance, not genuine merit — a direct instance of the Data Science Notes' multiple comparisons problem.
• Compare candidate models with a paired statistical test on cross-validation folds, not just by eyeballing which mean score is higher.
• Smaller validation sets give noisier estimates — aggressive tuning against small validation data risks fitting noise rather than genuine improvement.
• A statistically significant improvement isn't automatically worth acting on — weigh it against the practical cost of added complexity, training time, and reduced interpretability.

Module 7 Complete — Next Module

You now have the complete model selection toolkit: proper cross-validation strategies for different data structures, grid/random search, Bayesian optimization, and the discipline to avoid common tuning pitfalls. Module 8 shifts to reinforcement learning — supervised and unsupervised learning's counterpart paradigm, building directly on the AI Notes' foundational RL module.

Concept Check

1. Why should a dummy baseline always be established before investing time in complex model tuning?
2. Why is trying many hyperparameter combinations itself a form of the multiple comparisons problem?
3. Why might a statistically significant improvement between two models still not be worth adopting in practice?

Next Chapter

→ Module 8: Reinforcement Learning for ML Practitioners

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index