Machine Learning

Model Selection And Hyperparameter Tuning

Grid Search & Random Search

A distinction worth making precise before tuning anything.

JrCodex·7 min read

Jr Codex ML Notes

Level: Intermediate–Advanced Prerequisites: Chapter 1: Cross-Validation Strategies Time to complete: ~25 minutes

Table of Contents

  1. Hyperparameters vs Parameters
  2. Manual Tuning's Limits
  3. Grid Search
  4. The Combinatorial Explosion Problem
  5. Random Search
  6. Why Random Search Often Outperforms Grid Search
  7. A Complete Tuning Workflow
  8. Summary & Next Steps

1. Hyperparameters vs Parameters

A distinction worth making precise before tuning anything.

Parameters                              Hyperparameters
─────────────────────────────         ─────────────────────────────
  LEARNED from data during                SET before training begins
  training (Module 3, Ch.1's                — control HOW the learning
  coefficients, Module 4, Ch.3's              happens
  tree splits)

  e.g. linear regression's β                e.g. Ridge's alpha (Module
  coefficients                                3, Ch.2), a tree's
                                              max_depth (Module 4, Ch.3),
                                              K in KNN (Module 4, Ch.2)
─────────────────────────────         ─────────────────────────────

Every algorithm from Modules 3-6 has hyperparameters — this module is specifically about finding good values for them systematically, rather than the ad-hoc guessing used in earlier chapters' examples.

2. Manual Tuning's Limits

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
 
# Manually trying a FEW combinations — tedious, unsystematic, easy to miss the best one
for n_estimators in [50, 100, 200]:
    for max_depth in [5, 10, None]:
        model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth, random_state=42)
        scores = cross_val_score(model, X_train, y_train, cv=5)
        print(f"n_estimators={n_estimators}, max_depth={max_depth}: {scores.mean():.3f}")
Why This Doesn't Scale
─────────────────────────────────────────
  With just 2 hyperparameters and 3 values each, that's already
  9 combinations to manually write and track. Real models often
  have 5+ tunable hyperparameters (Module 5, Ch.3's XGBoost) —
  manual nested loops become unwieldy and error-prone FAST.
─────────────────────────────────────────

Automates exactly the nested-loop pattern above — define a grid of values for each hyperparameter, and exhaustively try every combination.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
 
param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [5, 10, None],
    "min_samples_leaf": [1, 5, 10],
}
 
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,               # Chapter 1's cross-validation, used to evaluate EACH combination
    scoring="f1",         # Module 4, Ch.6's metrics — choose based on the actual problem
    n_jobs=-1,              # use ALL available CPU cores in parallel
)
grid_search.fit(X_train, y_train)
 
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-validation score: {grid_search.best_score_:.3f}")
 
best_model = grid_search.best_estimator_      # the REFIT model, using the best combination found
What GridSearchCV Actually Does
─────────────────────────────────────────
  For EVERY combination in the grid (3×3×3 = 27 combinations here):
    Run FULL cross-validation (Chapter 1) using that combination
    Record the average CV score

  Pick the combination with the BEST average score, then REFIT
  a final model on ALL training data using those best
  hyperparameters.
─────────────────────────────────────────

4. The Combinatorial Explosion Problem

Grid search's exhaustive nature is exactly its weakness — the number of combinations grows multiplicatively, not additively, with each new hyperparameter.

# 3 hyperparameters, 3 values each: 3×3×3 = 27 combinations — manageable
# 5 hyperparameters, 5 values each: 5^5 = 3,125 combinations — expensive
# 6 hyperparameters, 10 values each: 10^6 = 1,000,000 combinations — infeasible!
 
n_hyperparameters = 6
n_values_each = 10
total_combinations = n_values_each ** n_hyperparameters
print(f"Total combinations: {total_combinations:,}")      # 1,000,000
 
# Each combination requires a FULL cross-validation run (Chapter 1) —
# with 5-fold CV, that's 5,000,000 total model trainings!
Directly Connects to the AI Notes' Search Chapter
─────────────────────────────────────────
  This is EXACTLY the same combinatorial explosion the AI
  Notes' search algorithms confront — an exhaustive search over
  a large space becomes intractable as the space grows.
  Grid search is essentially BREADTH-FIRST search (AI Notes,
  Module 2, Ch.2) over the hyperparameter space — thorough,
  but exponentially expensive.
─────────────────────────────────────────

Instead of trying every combination, random search samples a fixed number of random combinations from the hyperparameter space — trading exhaustiveness for tractability.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
 
param_distributions = {
    "n_estimators": randint(50, 500),           # a RANGE, not a fixed list
    "max_depth": randint(3, 20),
    "min_samples_leaf": randint(1, 20),
}
 
random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions,
    n_iter=50,          # try only 50 RANDOM combinations, regardless of the grid's true size
    cv=5,
    scoring="f1",
    random_state=42,
    n_jobs=-1,
)
random_search.fit(X_train, y_train)
 
print(f"Best parameters: {random_search.best_params_}")
print(f"Best score: {random_search.best_score_:.3f}")
Grid Search vs Random Search
─────────────────────────────────────────
  Grid Search:      tries EVERY combination in a FIXED, discrete
                       grid — guaranteed to find the best combo
                       WITHIN that grid, but the grid itself may
                       be too coarse, and cost grows exponentially

  Random Search:       samples a FIXED NUMBER of combinations
                          from a (possibly CONTINUOUS) range —
                          cost is controlled directly by n_iter,
                          regardless of how many hyperparameters
                          or values are involved
─────────────────────────────────────────

A genuinely counter-intuitive, well-established result: for a fixed compute budget, random search frequently finds better hyperparameters than grid search.

The Key Insight — Not Every Hyperparameter Matters Equally
─────────────────────────────────────────
  In practice, often only ONE OR TWO hyperparameters have a
  LARGE effect on performance — the rest matter much less.

  Grid search WASTES effort testing every combination of the
  UNIMPORTANT hyperparameters, at EVERY value of the important
  ones — effectively duplicating the SAME few important-
  hyperparameter values repeatedly.

  Random search, by SAMPLING continuously, tries MANY MORE
  DISTINCT values for the truly important hyperparameters,
  within the SAME total number of trials.
─────────────────────────────────────────

Visualizing Why (2 Hyperparameters: One Important, One Not)
─────────────────────────────────────────
  Grid Search (9 trials, 3×3 grid):        Random Search (9 trials):
    ●  ●  ●   ← only 3 DISTINCT              ●     ●   ●
    ●  ●  ●      values tried for               ●    ●
    ●  ●  ●      the IMPORTANT axis           ●  ●      ●
   (important)                              (important axis: 9
                                              DISTINCT values tried!)
─────────────────────────────────────────

Practical guidance: for a large hyperparameter space or limited compute budget, prefer random search over grid search — reserve grid search for a FINAL, narrow refinement once random search (or domain knowledge) has already identified a promising region.

7. A Complete Tuning Workflow

from sklearn.model_selection import train_test_split, RandomizedSearchCV, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
 
# 1. Split off a FINAL test set — never touched during tuning (Module 1, Ch.5)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
 
# 2. BROAD random search FIRST — explore the space efficiently
broad_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    {"n_estimators": randint(50, 500), "max_depth": randint(3, 30), "min_samples_leaf": randint(1, 20)},
    n_iter=100, cv=5, random_state=42, n_jobs=-1,
)
broad_search.fit(X_train, y_train)
print(f"Promising region found: {broad_search.best_params_}")
 
# 3. NARROW grid search around the promising region found — final refinement
narrow_grid = {
    "n_estimators": [broad_search.best_params_["n_estimators"] - 20,
                     broad_search.best_params_["n_estimators"],
                     broad_search.best_params_["n_estimators"] + 20],
    "max_depth": [broad_search.best_params_["max_depth"] - 2,
                  broad_search.best_params_["max_depth"],
                  broad_search.best_params_["max_depth"] + 2],
}
final_search = GridSearchCV(RandomForestClassifier(random_state=42), narrow_grid, cv=5, n_jobs=-1)
final_search.fit(X_train, y_train)
 
# 4. Evaluate the FINAL model on the untouched test set — ONCE
final_model = final_search.best_estimator_
final_test_score = final_model.score(X_test, y_test)
print(f"Final, honest test score: {final_test_score:.3f}")

8. Summary & Next Steps

Key Takeaways

  • Hyperparameters are set before training and control how learning happens (unlike parameters, which are learned from data) — this entire module is about finding good hyperparameter values systematically.
  • Grid search exhaustively tries every combination in a defined grid, guaranteed to find the best within that grid, but its cost grows exponentially with each added hyperparameter.
  • Random search samples a fixed number of random combinations, controlling cost directly and often outperforming grid search on a fixed budget, since it explores more distinct values of the truly important hyperparameters.
  • A practical workflow combines both: broad random search to find a promising region, then a narrow grid search to refine within it, evaluating the final model on an untouched test set exactly once.

Concept Check

  1. Why does grid search's cost grow exponentially rather than linearly as more hyperparameters are added?
  2. Why can random search outperform grid search on a fixed compute budget, even though it doesn't try every combination?
  3. In the combined workflow, why does the narrow grid search happen after the broad random search, not before?

Next Chapter

→ Chapter 3: Bayesian Optimization & Advanced Tuning

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index