Machine Learning

Ensemble Learning

Boosting: AdaBoost & Gradient Boosting

Chapter 1's bagging trains many models independently and in parallel, then averages them. Boosting trains models sequentially, where each new model specifically

JrCodex·8 min read

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 1: Bagging & Random Forests Time to complete: ~30 minutes

Table of Contents

  1. A Fundamentally Different Ensemble Strategy
  2. The Boosting Idea
  3. AdaBoost — Adaptive Boosting
  4. AdaBoost Step by Step
  5. Gradient Boosting — a More General Framework
  6. Gradient Boosting Step by Step
  7. Bagging vs Boosting — Bias and Variance, Revisited
  8. The Overfitting Risk Unique to Boosting
  9. Summary & Next Steps

1. A Fundamentally Different Ensemble Strategy

Chapter 1's bagging trains many models independently and in parallel, then averages them. Boosting trains models sequentially, where each new model specifically focuses on correcting the mistakes of the ones before it.

Bagging (Chapter 1)                    Boosting (this chapter)
─────────────────────────────         ─────────────────────────────
  Trees trained INDEPENDENTLY,           Trees trained SEQUENTIALLY,
  in PARALLEL — each on a                  each one specifically
  random resample                           targeting the PREVIOUS
                                              ones' ERRORS

  Reduces VARIANCE (Ch.1, Sec.3)             Reduces BIAS primarily
                                                (Section 7 explains
                                                 this precisely)
─────────────────────────────         ─────────────────────────────

2. The Boosting Idea

The General Boosting Recipe
─────────────────────────────────────────
  1. Train a SIMPLE, WEAK model (often a shallow decision tree,
     sometimes just a single split — called a "stump")
  2. Identify which examples it got WRONG
  3. Train the NEXT weak model, specifically focused on
     getting those WRONG examples right
  4. Repeat, building up a SEQUENCE of models, each correcting
     the previous ensemble's mistakes
  5. Combine ALL models' predictions (typically a WEIGHTED sum,
     not a simple average)
─────────────────────────────────────────

Why "Weak" Learners Specifically
─────────────────────────────────────────
  Boosting deliberately uses WEAK models (barely better than
  random guessing individually) — the POWER comes from
  combining MANY of them, each correcting a specific,
  previously-uncorrected mistake, not from any single strong
  model.
─────────────────────────────────────────

3. AdaBoost — Adaptive Boosting

The original, historically foundational boosting algorithm — adapts by increasing the weight of misclassified examples, forcing subsequent weak learners to pay more attention to them.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
 
model = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),      # a "stump" — the classic weak learner
    n_estimators=50,
    learning_rate=1.0,
    random_state=42,
)
model.fit(X_train, y_train)

4. AdaBoost Step by Step

import numpy as np
 
def adaboost_conceptual(X, y, n_estimators=5):
    """A simplified, from-scratch sketch of AdaBoost's core idea."""
    n = len(y)
    weights = np.ones(n) / n      # start: every example weighted EQUALLY
 
    models = []
    model_weights = []
 
    for t in range(n_estimators):
        # Train a weak learner, giving MORE ATTENTION to HIGH-WEIGHT examples
        # (real implementations pass `sample_weight=weights` directly to the fit() call)
        weak_model = train_weak_learner(X, y, sample_weight=weights)
        predictions = weak_model.predict(X)
 
        # Compute this learner's WEIGHTED error rate
        incorrect = (predictions != y)
        error = np.sum(weights * incorrect) / np.sum(weights)
 
        # Compute this learner's VOTING WEIGHT — better learners get MORE say
        model_weight = 0.5 * np.log((1 - error) / (error + 1e-10))
 
        # INCREASE weight on MISCLASSIFIED examples — the crucial "adaptive" step
        weights *= np.exp(model_weight * incorrect * 2 - model_weight)
        weights /= np.sum(weights)      # renormalize back to sum=1
 
        models.append(weak_model)
        model_weights.append(model_weight)
 
    return models, model_weights
AdaBoost's Weighting Mechanism, Visualized
─────────────────────────────────────────
  Round 1:  ●  ●  ●  ✗  ●  ●     ← ONE example misclassified (✗)
  Round 2:  ●  ●  ●  ✗✗✗ ●  ●    ← that example's WEIGHT increased —
                                    the NEXT weak learner is FORCED
                                    to pay it more attention

  Each subsequent round FOCUSES increasingly on whatever the
  ensemble-so-far still gets WRONG.
─────────────────────────────────────────

A better-performing weak learner gets a higher voting weight in the final combination — this is precisely what model_weight computes: learners that achieve low error get to "vote" more strongly in the ensemble's final prediction.

5. Gradient Boosting — a More General Framework

Gradient Boosting generalizes AdaBoost's idea using gradient descent (Module 3, Chapter 1) — instead of reweighting examples, each new model is trained to predict the residual errors (how far off the current ensemble is) of the previous ensemble.

from sklearn.ensemble import GradientBoostingClassifier
 
model = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42,
)
model.fit(X_train, y_train)
Gradient Boosting for Regression — the Clearest Way to See It
─────────────────────────────────────────
  Step 1:  Fit a SIMPLE model to the DATA. Predictions are
             imperfect — there's a RESIDUAL (actual - predicted)
             for every point.

  Step 2:     Fit a NEW model — NOT to the original target y,
                but to the RESIDUALS from Step 1.

  Step 3:       ADD this new model's predictions (scaled by the
                  learning_rate) to the ensemble's running total.

  Step 4:         Repeat: compute NEW residuals (from the UPDATED
                    ensemble), fit ANOTHER model to THOSE.
─────────────────────────────────────────

6. Gradient Boosting Step by Step

import numpy as np
from sklearn.tree import DecisionTreeRegressor
 
def gradient_boosting_conceptual(X, y, n_estimators=10, learning_rate=0.1):
    """A simplified, from-scratch sketch for REGRESSION."""
    predictions = np.zeros(len(y))      # start: predict 0 for everyone
    models = []
 
    for t in range(n_estimators):
        residuals = y - predictions      # how WRONG is the CURRENT ensemble?
 
        # Fit a NEW weak model to predict THESE RESIDUALS specifically
        weak_model = DecisionTreeRegressor(max_depth=2)
        weak_model.fit(X, residuals)
 
        # Add this model's prediction, SCALED by the learning rate,
        # to the running ensemble total
        predictions += learning_rate * weak_model.predict(X)
        models.append(weak_model)
 
    return models
 
# Notice: this is EXACTLY gradient descent (Module 3, Ch.1), but taking
# a "step" in FUNCTION SPACE (adding a new tree) rather than adjusting
# a fixed set of coefficients directly
Why This Is Called "GRADIENT" Boosting
─────────────────────────────────────────
  For the common case of minimizing squared error, the RESIDUAL
  (actual - predicted) is EXACTLY the NEGATIVE GRADIENT of the
  squared error loss function with respect to the current
  prediction — fitting a new model to residuals IS, mathematically,
  taking a gradient descent step (Module 3, Ch.1) in the space of
  possible ENSEMBLE predictions, not just coefficient values.
─────────────────────────────────────────

learning_rate controls how much each new tree's correction contributes — smaller values need more n_estimators to converge but generally generalize better; this is directly analogous to gradient descent's own step-size parameter.

7. Bagging vs Boosting — Bias and Variance, Revisited

Why Bagging Reduces VARIANCE (Chapter 1)
─────────────────────────────────────────
  Averaging many INDEPENDENT, individually reasonably-strong
  models cancels out their RANDOM errors — the ensemble is
  MORE STABLE than any single tree.
─────────────────────────────────────────

Why Boosting Reduces BIAS Primarily
─────────────────────────────────────────
  Each new WEAK model is specifically forced to correct the
  CURRENT ensemble's systematic mistakes — the ensemble
  progressively gets LESS BIASED (able to represent more
  complex patterns) as more weak learners are added, since
  even simple building blocks, combined ENOUGH times in a
  targeted way, can represent very complex functions.
─────────────────────────────────────────
Bagging (Chapter 1)Boosting (this chapter)
TrainingParallel, independentSequential, each depends on the last
Primary effectReduces varianceReduces bias
Base learner strengthReasonably strong (full trees)Deliberately weak (stumps/shallow trees)
Overfitting riskLow, even with many estimatorsHigher — more rounds can eventually overfit (Section 8)

8. The Overfitting Risk Unique to Boosting

Unlike bagging (where n_estimators essentially never hurts), boosting's n_estimators genuinely IS a bias-variance dial that can overfit if pushed too far.

from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score
import numpy as np
 
np.random.seed(0)
X = np.random.randn(300, 5)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
 
for n_est in [10, 50, 200, 1000]:
    model = GradientBoostingClassifier(n_estimators=n_est, learning_rate=0.1, random_state=42)
    model.fit(X_train, y_train)
    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_acc = accuracy_score(y_test, model.predict(X_test))
    print(f"n_estimators={n_est:5d}: train acc={train_acc:.3f}, test acc={test_acc:.3f}")
    # Eventually, TRAIN accuracy keeps climbing toward 1.0, while
    # TEST accuracy may start to PLATEAU or even DECLINE — classic
    # overfitting (Module 1, Ch.4), directly visible here

This is precisely why boosting's hyperparameters (n_estimators, learning_rate, max_depth) need careful tuning via cross-validation (Module 7) — unlike Chapter 1's random forests, simply adding more estimators isn't a safe default here.

9. Summary & Next Steps

Key Takeaways

  • Boosting trains weak models sequentially, each specifically correcting the previous ensemble's mistakes — a fundamentally different strategy from bagging's independent, parallel training.
  • AdaBoost adapts by increasing the weight of misclassified examples, forcing subsequent weak learners to focus on them, and weights each learner's final vote by its accuracy.
  • Gradient boosting generalizes this by fitting each new model to the current ensemble's residual errors — mathematically equivalent to gradient descent in the space of possible ensemble predictions.
  • Bagging primarily reduces variance (averaging independent errors); boosting primarily reduces bias (progressively correcting systematic mistakes) — a genuinely different mechanism.
  • Unlike bagging, boosting can overfit with too many estimators — its hyperparameters require careful tuning rather than "more is always better."

Concept Check

  1. What's the fundamental difference in training strategy between bagging and boosting?
  2. Why is a residual in gradient boosting for regression described as a "negative gradient"?
  3. Why can adding more estimators eventually hurt a boosting model's test performance, when it essentially never hurts a random forest's?

Next Chapter

→ Chapter 3: XGBoost, LightGBM & Modern Boosting

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index