Bagging & Random Forests

Jr Codex ML Notes

Level: Intermediate–Advanced Prerequisites: Module 4, Chapter 3: Decision Trees Time to complete: ~30 minutes

Table of Contents

1. Why Combine Models At All?
2. Bootstrap Aggregating (Bagging)
3. Why Averaging Reduces Variance
4. Random Forests — Adding a Second Layer of Randomness
5. Out-of-Bag Evaluation — a Free Validation Set
6. Feature Importance from Random Forests
7. Key Hyperparameters
8. Strengths, Weaknesses & When to Use
9. Summary & Next Steps

1. Why Combine Models At All?

Module 4, Chapter 3 identified decision trees' central weakness: high variance — small changes in training data can produce a dramatically different tree (Module 1, Chapter 4's bias-variance tradeoff, in its high-variance extreme). Ensemble methods combine many models together, on the theory that their individual errors will often cancel out.

The Core Ensemble Insight
─────────────────────────────────────────
  A single decision tree might overfit to the SPECIFIC noise
  in its training sample.

  MANY trees, each trained slightly DIFFERENTLY, will likely
  make DIFFERENT mistakes — averaging their predictions
  cancels out much of this individual noise, while the GENUINE
  signal (which all trees tend to agree on) survives.
─────────────────────────────────────────

2. Bootstrap Aggregating (Bagging)

Bagging (BOOTSTRAP AGGregating) trains many copies of the same algorithm, each on a different random resample of the training data, then averages their predictions.

import numpy as np
 
def bootstrap_sample(X, y):
    """Sample WITH REPLACEMENT — the same size as the original, but some
    points appear multiple times, others not at all."""
    n = len(X)
    indices = np.random.choice(n, size=n, replace=True)
    return X[indices], y[indices]
 
np.random.seed(0)
X = np.arange(10).reshape(-1, 1)
y = np.arange(10)
 
X_sample, y_sample = bootstrap_sample(X, y)
print(y_sample)      # e.g. [3 3 7 1 0 9 3 5 2 8] — duplicates AND omissions, same TOTAL size

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
 
model = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=100,      # train 100 DIFFERENT trees
    random_state=42,
)
model.fit(X_train, y_train)      # each of the 100 trees sees a DIFFERENT bootstrap sample

Bagging, Visualized
─────────────────────────────────────────
  Original Training Data
          │
    ┌─────┼─────┬─────┬─────┐
    ▼     ▼     ▼     ▼     ▼
  Sample1 Sample2 Sample3 ... Sample100    (each a DIFFERENT
    │       │       │              │        bootstrap resample)
    ▼       ▼       ▼              ▼
  Tree1   Tree2   Tree3     ...  Tree100    (each trained
    │       │       │              │         INDEPENDENTLY)
    └───────┴───────┴──────────────┘
                    │
              AVERAGE (regression) or
              MAJORITY VOTE (classification)
                    │
              Final Prediction
─────────────────────────────────────────

3. Why Averaging Reduces Variance

A direct mathematical consequence, worth understanding precisely rather than taking on faith.

import numpy as np
 
np.random.seed(1)
 
# Simulating: 100 "noisy" individual predictions, each with genuine random error
true_value = 50
individual_predictions = true_value + np.random.normal(0, 10, 100)      # each off by ~10, on average
 
single_prediction_error = np.abs(individual_predictions[0] - true_value)
averaged_prediction_error = np.abs(np.mean(individual_predictions) - true_value)
 
print(f"Single tree's error: {single_prediction_error:.2f}")
print(f"Average of 100 trees' error: {averaged_prediction_error:.2f}")      # typically MUCH smaller

The Statistical Reason (Directly from the Data Science Notes)
─────────────────────────────────────────
  This is EXACTLY the Data Science Notes' Central Limit Theorem
  and standard error concept (Module 1, Ch.4), applied to model
  predictions instead of sample means.

  If individual trees' errors are reasonably INDEPENDENT
  (achieved by bagging's random resampling), averaging many of
  them reduces the VARIANCE of the combined prediction — the
  SAME mathematical principle behind "larger samples give more
  reliable estimates."
─────────────────────────────────────────

This is precisely why bagging fixes decision trees' variance problem without needing to touch their bias at all — averaging cancels out random, independent errors, but a systematic bias shared by every tree (e.g. from a fundamentally wrong assumption) would NOT be fixed by bagging alone.

4. Random Forests — Adding a Second Layer of Randomness

A Random Forest is bagging applied specifically to decision trees, with one crucial addition: at each split, only a random subset of features is considered — not all of them.

from sklearn.ensemble import RandomForestClassifier
 
model = RandomForestClassifier(
    n_estimators=100,
    max_features="sqrt",      # each split considers only √(total features) candidates
    random_state=42,
)
model.fit(X_train, y_train)

Why This SECOND Layer of Randomness Matters
─────────────────────────────────────────
  Without it, if ONE feature is extremely strong, EVERY
  bagged tree would likely choose to split on THAT SAME
  feature first — making the trees highly CORRELATED with
  each other (all making SIMILAR mistakes), which undermines
  Section 3's variance-reduction argument (averaging only
  helps when errors are reasonably INDEPENDENT).

  By restricting each split to a RANDOM subset of features,
  different trees are FORCED to consider different features,
  producing more DIVERSE, less correlated trees — and
  therefore MORE effective variance reduction when averaged.
─────────────────────────────────────────

5. Out-of-Bag Evaluation — a Free Validation Set

Bootstrap sampling (Section 2) leaves out, on average, about 37% of the original data for each tree — these "out-of-bag" (OOB) samples provide a free, built-in validation set, without needing a separate train/test split.

from sklearn.ensemble import RandomForestClassifier
 
model = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
model.fit(X_train, y_train)
 
print(f"Out-of-bag score: {model.oob_score_:.3f}")

Why ~37% Are Left Out
─────────────────────────────────────────
  Sampling n items WITH REPLACEMENT from n items: each specific
  item has a (1 - 1/n)ⁿ probability of NEVER being selected —
  as n grows large, this converges to 1/e ≈ 0.368 (~37%).

  Each tree therefore has its OWN unused ~37% "held out" sample
  it never trained on — evaluating each tree on ITS OWN
  out-of-bag samples gives an honest, essentially FREE
  validation estimate (Module 1, Ch.5's concerns, elegantly
  sidestepped as a side effect of bagging itself).
─────────────────────────────────────────

6. Feature Importance from Random Forests

A valuable byproduct: random forests naturally compute which features contributed most to reducing impurity (Module 4, Chapter 3) across all trees — a fast, built-in approximation of Module 2, Chapter 4's feature importance concept.

import pandas as pd
 
feature_names = ["income", "age", "credit_score"]
importances = model.feature_importances_
 
importance_df = pd.DataFrame({
    "feature": feature_names,
    "importance": importances,
}).sort_values("importance", ascending=False)
 
print(importance_df)

How This Is Computed (Conceptually)
─────────────────────────────────────────
  For EACH feature, sum up how much TOTAL impurity reduction
  (Module 4, Ch.3's information gain/Gini decrease) it was
  responsible for, across EVERY split, in EVERY tree —
  features used for many HIGH-IMPACT splits score higher.
─────────────────────────────────────────

This connects directly to the AI Notes' Explainable AI chapter's feature importance discussion — random forest importance is one of the most commonly used, readily-available "explanation" tools in practical ML work, though it should be interpreted alongside model-agnostic techniques (permutation importance, SHAP) for a fuller picture.

7. Key Hyperparameters

from sklearn.ensemble import RandomForestClassifier
 
model = RandomForestClassifier(
    n_estimators=200,        # MORE trees = generally better, with DIMINISHING returns, more compute cost
    max_depth=10,             # controls INDIVIDUAL tree complexity (Module 4, Ch.3's pruning)
    max_features="sqrt",        # size of the random feature subset per split (Section 4)
    min_samples_leaf=5,           # minimum examples per leaf — a pruning control
    random_state=42,
)

Hyperparameter Guidance
─────────────────────────────────────────
  n_estimators:     more trees almost ALWAYS help (or at worst,
                      don't hurt) — the main cost is COMPUTE TIME,
                      not overfitting risk, unlike most other
                      hyperparameters in this curriculum
  max_features:        smaller = MORE diverse trees (Section 4),
                          but each individual tree may be WEAKER
  max_depth /               controls INDIVIDUAL tree overfitting —
  min_samples_leaf:            random forests are more forgiving
                                  of deep trees than a SINGLE
                                  decision tree would be, since
                                  averaging (Section 3) compensates
─────────────────────────────────────────

Module 7 covers systematically searching this hyperparameter space rather than guessing — worth returning to random forests once that module's grid/random search techniques are available.

8. Strengths, Weaknesses & When to Use

Strengths
─────────────────────────────────────────
  - Dramatically more ACCURATE and STABLE than a single
    decision tree, directly fixing Module 4, Ch.3's variance problem
  - Handles both classification AND regression
  - Requires NO feature scaling (inherits this from decision trees)
  - Built-in feature importance and out-of-bag evaluation
  - Resistant to overfitting even with MANY trees (unlike
    boosting, Chapter 2, which can overfit with too many rounds)
─────────────────────────────────────────

Weaknesses
─────────────────────────────────────────
  - LESS interpretable than a single decision tree — you can't
    easily "read" a forest of 100+ trees the way you can one tree
  - Larger memory footprint (storing MANY trees)
  - Generally OUTPERFORMED by well-tuned boosting methods
    (Chapters 2-3) on many structured/tabular problems, though
    often easier to tune well with less risk of overfitting
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

• Bagging trains many models on different bootstrap resamples of the training data, then averages their predictions — directly fixing decision trees' high-variance instability by canceling out independent errors.
• Averaging reduces variance for the same statistical reason the Data Science Notes' Central Limit Theorem gives larger samples more reliable estimates.
• Random forests add a second layer of randomness — considering only a random subset of features at each split — specifically to reduce correlation between trees, making the variance-reduction from averaging more effective.
• Out-of-bag samples (the ~37% left out of each bootstrap resample) provide a free, built-in validation estimate without needing a separate split.
• More trees (n_estimators) in a random forest essentially never hurts accuracy, unlike most other hyperparameters — the main cost is compute time, not overfitting risk.

Concept Check

1. Why does averaging many independently-trained trees reduce variance, but not necessarily bias?
2. Why does a random forest restrict each split to a random subset of features, rather than letting every tree consider all features?
3. What is an out-of-bag sample, and why does it provide a "free" validation estimate?

Next Chapter

→ Chapter 2: Boosting — AdaBoost & Gradient Boosting

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index