Machine Learning

Model Selection And Hyperparameter Tuning

Bayesian Optimization & Advanced Tuning

Chapter 2's grid and random search share a common blind spot: neither uses information from past trials to guide future ones. Every combination is chosen indepe

JrCodex·7 min read

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 2: Grid Search & Random Search Time to complete: ~25 minutes

Table of Contents

  1. What Grid and Random Search Both Ignore
  2. The Bayesian Optimization Idea
  3. The Surrogate Model
  4. The Acquisition Function — Exploration vs Exploitation
  5. Bayesian Optimization in Practice
  6. Early Stopping — a Complementary Technique
  7. When Bayesian Optimization Is (and Isn't) Worth It
  8. Summary & Next Steps

1. What Grid and Random Search Both Ignore

Chapter 2's grid and random search share a common blind spot: neither uses information from past trials to guide future ones. Every combination is chosen independently, in advance, without learning "combination A performed well, so combinations similar to A are probably worth trying next."

The Missed Opportunity
─────────────────────────────────────────
  After trying max_depth=5 (score: 0.75) and max_depth=20
  (score: 0.60), a HUMAN would naturally guess that something
  BETWEEN these — or maybe slightly below 5 — is worth trying
  next, rather than continuing to sample uniformly at random.

  Grid and random search have NO mechanism to make this
  inference — Bayesian optimization exists specifically to
  add it.
─────────────────────────────────────────

2. The Bayesian Optimization Idea

Directly connects to the AI Notes' probabilistic reasoning module — Bayesian optimization builds a probabilistic model of "which hyperparameter values are likely to perform well," updating this belief after every trial, exactly the prior-to-posterior update from the AI Notes' Bayes' theorem chapter.

The Bayesian Optimization Loop
─────────────────────────────────────────
  1. Build a SURROGATE MODEL — a probabilistic guess at how
     performance varies across the hyperparameter space,
     based on trials SO FAR (Section 3)
  2. Use an ACQUISITION FUNCTION to decide which point to try
     NEXT — balancing trying PROMISING regions against
     UNEXPLORED ones (Section 4)
  3. Evaluate the model at that chosen point (an ACTUAL
     cross-validation run, Chapter 1)
  4. UPDATE the surrogate model with this new result
  5. REPEAT until a trial budget is exhausted
─────────────────────────────────────────

3. The Surrogate Model

Most commonly, a Gaussian Process — a probabilistic model that predicts both an expected performance and an uncertainty estimate for any untested hyperparameter combination.

What a Gaussian Process Provides
─────────────────────────────────────────
  Performance
     │           ┌─╮
     │      ┌─╮ ╱   ╲            ┌╮
     │ ╱‾╲ ╱   ╲     ╲          ╱  ╲
     │╱    ╲     ‾╲   ‾╲___╱‾╲╱     ╲
     │              (shaded = UNCERTAINTY band —
     │                WIDER where FEWER trials
     │                have been tried nearby)
     └──────────────────────────────────
              Hyperparameter value

  Near ALREADY-TESTED points: LOW uncertainty (we KNOW roughly
    what happens there)
  Far from tested points: HIGH uncertainty (we're GUESSING)
─────────────────────────────────────────

This uncertainty estimate is precisely what makes Bayesian optimization smarter than random search — it explicitly tracks not just where performance might be good, but how confident it is about each region, directly informing Section 4's decision about where to look next.

4. The Acquisition Function — Exploration vs Exploitation

A direct, concrete instance of the AI Notes' Module 5 exploration-exploitation trade-off from reinforcement learning, applied here to hyperparameter search instead of a multi-armed bandit.

The Same Trade-off, Different Context
─────────────────────────────────────────
  EXPLOIT:    try hyperparameters NEAR the best-performing
                point found SO FAR — likely to be good, but
                might miss something even better elsewhere

  EXPLORE:       try hyperparameters in a REGION with HIGH
                   uncertainty (Section 3) — might discover
                   something better, but might also waste a
                   trial on a bad combination
─────────────────────────────────────────

# Conceptual sketch — a common acquisition function, "Expected Improvement"
def expected_improvement(predicted_mean, predicted_std, best_so_far):
    """
    Balances EXPLOITATION (predicted_mean close to or above best_so_far)
    against EXPLORATION (high predicted_std, meaning UNCERTAIN, possibly
    surprising regions).
    """
    improvement = predicted_mean - best_so_far
    # Regions with HIGH uncertainty (predicted_std) get a BOOST even if
    # their predicted_mean isn't the current best — this is the EXPLORATION term
    z = improvement / (predicted_std + 1e-9)
    return improvement * some_cdf(z) + predicted_std * some_pdf(z)      # conceptual, not exact formula

This is the exact same tension the AI Notes' epsilon-greedy strategy addressed for the multi-armed bandit problem — Bayesian optimization's acquisition function is a more sophisticated, information-aware way of navigating the same fundamental trade-off.

5. Bayesian Optimization in Practice

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
 
def objective(trial):
    """Optuna calls this repeatedly, using its surrogate model to CHOOSE parameters intelligently."""
    n_estimators = trial.suggest_int("n_estimators", 50, 500)
    max_depth = trial.suggest_int("max_depth", 3, 30)
    min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 20)
 
    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_leaf=min_samples_leaf,
        random_state=42,
    )
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
    return scores.mean()
 
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)      # Optuna INTELLIGENTLY chooses each of the 50 trials
 
print(f"Best parameters: {study.best_params}")
print(f"Best score: {study.best_value}")
Optuna's Trials, Conceptually, vs Random Search's
─────────────────────────────────────────
  Random Search (Ch.2):   50 trials, chosen COMPLETELY
                             independently, no learning between them

  Optuna (Bayesian):         50 trials, but EACH ONE informed by
                                everything learned from ALL
                                PREVIOUS trials — later trials
                                increasingly focus on promising
                                regions
─────────────────────────────────────────

6. Early Stopping — a Complementary Technique

A different, often-combined optimization: stop evaluating a clearly poor trial early, rather than wasting the full compute budget on a hyperparameter combination that's obviously not competitive.

import optuna
from sklearn.ensemble import GradientBoostingClassifier
 
def objective_with_pruning(trial):
    n_estimators = trial.suggest_int("n_estimators", 50, 500)
    learning_rate = trial.suggest_float("learning_rate", 0.01, 0.3, log=True)
 
    model = GradientBoostingClassifier(
        n_estimators=n_estimators, learning_rate=learning_rate, random_state=42
    )
 
    # Real implementations report INTERMEDIATE scores during training,
    # letting Optuna's PRUNER decide to ABANDON a clearly bad trial
    # partway through, rather than completing it in full
    scores = cross_val_score(model, X_train, y_train, cv=3)
    return scores.mean()
 
study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.optimize(objective_with_pruning, n_trials=100)
Why Pruning Saves Real Compute
─────────────────────────────────────────
  If a trial's early performance is CLEARLY worse than the
  median of previous trials at the SAME point in training,
  it's abandoned EARLY — freeing up compute budget for MORE,
  potentially better trials instead.
─────────────────────────────────────────

7. When Bayesian Optimization Is (and Isn't) Worth It

Worth the Added Complexity When...
─────────────────────────────────────────
  - Each individual trial (a full cross-validation run) is
    EXPENSIVE (deep learning models, large datasets, Module
    5, Ch.3's XGBoost with many estimators)
  - You have a LARGE hyperparameter space and a LIMITED trial
    budget, making every trial's information value matter
  - Squeezing out the last increment of performance genuinely matters
─────────────────────────────────────────

Often NOT Worth It When...
─────────────────────────────────────────
  - Individual trials are CHEAP and fast — random search
    (Chapter 2) with MANY trials is simpler and often nearly
    as effective
  - The hyperparameter space is SMALL (a handful of values
    each) — plain grid search (Chapter 2) is simpler and
    exhaustive anyway
  - Added tooling complexity (Optuna, scikit-optimize) isn't
    justified by the marginal improvement over simpler methods
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

  • Grid and random search treat every trial independently; Bayesian optimization builds a probabilistic surrogate model of the hyperparameter space, using past trials to intelligently choose future ones.
  • The surrogate model (commonly a Gaussian Process) provides both an expected performance and an uncertainty estimate for any untested combination.
  • The acquisition function balances exploiting promising regions against exploring uncertain ones — the exact same trade-off the AI Notes' reinforcement learning module covers for the multi-armed bandit problem.
  • Early stopping/pruning complements Bayesian optimization by abandoning clearly poor trials partway through, freeing compute for more promising ones.
  • Bayesian optimization is worth its added complexity when individual trials are expensive and the hyperparameter space is large; for cheap trials or small spaces, random or grid search remain simpler and often sufficient.

Concept Check

  1. What key piece of information does Bayesian optimization use that grid and random search both ignore?
  2. How does the acquisition function's exploration-exploitation trade-off relate to the AI Notes' reinforcement learning module?
  3. When would the added complexity of Bayesian optimization not be worth it compared to random search?

Next Chapter

→ Chapter 4: Model Selection Best Practices

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index