Ensemble Learning
Stacking & Blending
Chapters 1-3 combined many versions of the same algorithm (trees). Stacking and blending go a step further, combining predictions from fundamentally different a
Jr Codex ML Notes
Level: Advanced Prerequisites: Chapter 3: XGBoost, LightGBM & Modern Boosting Time to complete: ~25 minutes
Table of Contents
- Combining Fundamentally Different Models
- Simple Voting/Averaging Ensembles
- Stacking — Learning How to Combine Models
- Why Stacking Needs Careful Cross-Validation
- Blending — a Simpler Alternative
- Why Diversity Among Base Models Matters
- When Stacking Is (and Isn't) Worth the Complexity
- Summary & Next Steps
1. Combining Fundamentally Different Models
Chapters 1-3 combined many versions of the same algorithm (trees). Stacking and blending go a step further, combining predictions from fundamentally different algorithms — a logistic regression, a random forest, an SVM, and a gradient boosting model, all working together.
Why This Can Help Beyond Chapters 1-3
─────────────────────────────────────────
Different algorithms make DIFFERENT kinds of mistakes:
- Logistic regression (Module 4, Ch.1) struggles with
NON-LINEAR patterns
- KNN (Module 4, Ch.2) struggles in HIGH dimensions
- Decision trees (Module 4, Ch.3) can be UNSTABLE
- SVM (Module 4, Ch.4) can struggle with VERY large datasets
If their ERRORS aren't perfectly correlated (they tend NOT
to be, given how differently each algorithm "thinks"),
combining them can capture strengths from EACH while
diluting each one's individual weaknesses.
─────────────────────────────────────────
2. Simple Voting/Averaging Ensembles
The most basic way to combine different models — directly analogous to Chapter 1's bagging, but across different algorithms instead of different bootstrap samples of the same one.
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
voting_model = VotingClassifier(
estimators=[
("lr", LogisticRegression()),
("rf", RandomForestClassifier(random_state=42)),
("svm", SVC(probability=True)),
],
voting="soft", # "soft" averages PREDICTED PROBABILITIES; "hard" uses MAJORITY VOTE
)
voting_model.fit(X_train, y_train)Hard Voting vs Soft Voting
─────────────────────────────────────────
Hard voting: each model casts ONE VOTE for its predicted
class — majority wins (like Chapter 1's
bagging classification)
Soft voting: AVERAGES each model's predicted PROBABILITY
for each class, then picks the highest
average — generally performs BETTER,
since it uses more information than a
simple vote (how CONFIDENT each model was,
not just its final answer)
─────────────────────────────────────────
3. Stacking — Learning How to Combine Models
Rather than a simple average or vote, stacking trains a meta-model that learns the best way to combine the base models' predictions.
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
stacking_model = StackingClassifier(
estimators=[
("rf", RandomForestClassifier(random_state=42)),
("svm", SVC(probability=True)),
],
final_estimator=LogisticRegression(), # the META-MODEL, learns HOW to combine the base models
cv=5, # Section 4 explains why this matters
)
stacking_model.fit(X_train, y_train)Stacking Architecture
─────────────────────────────────────────
Original Features (X)
│
┌────────────┼────────────┐
▼ ▼ ▼
Model A Model B Model C ← "base" or "level-0" models
│ │ │
▼ ▼ ▼
prediction_A prediction_B prediction_C
│ │ │
└────────────┼────────────┘
▼
[Meta-Model] ← "level-1" model, trained on
│ the BASE MODELS' predictions
▼ as ITS OWN input features
Final Prediction
─────────────────────────────────────────
The meta-model literally learns things like "when the random forest and SVM disagree, trust the random forest more in this situation" — a genuinely learned combination strategy, more sophisticated than Section 2's fixed average or vote.
4. Why Stacking Needs Careful Cross-Validation
A subtle but critical detail: if base models are trained on the same data used to generate the meta-model's training inputs, the meta-model would see predictions the base models have already "memorized" — a form of data leakage (Module 1, Chapter 5).
The Leakage Risk, Concretely
─────────────────────────────────────────
WRONG approach:
1. Train base models on ALL training data
2. Use those SAME base models to predict on the SAME
training data
3. Train the meta-model on THOSE predictions
Problem: base models' predictions on data they were TRAINED
ON are unrealistically ACCURATE (they've essentially
memorized it) — the meta-model would learn to trust them
MORE than it should, based on inflated, non-generalizable
confidence.
─────────────────────────────────────────
The Correct Approach — Out-of-Fold Predictions
─────────────────────────────────────────
Using K-FOLD CROSS-VALIDATION (Module 1, Ch.5):
1. Split training data into K folds
2. For EACH fold: train base models on the OTHER K-1 folds,
predict on THIS held-out fold
3. Collect these OUT-OF-FOLD predictions across all K folds
— each prediction comes from a model that NEVER saw that
specific example during training
4. Train the meta-model on THESE honest, out-of-fold predictions
─────────────────────────────────────────
This is exactly what scikit-learn's StackingClassifier's cv parameter automates — it handles the out-of-fold prediction generation correctly and safely, so this leakage risk doesn't require manual handling.
5. Blending — a Simpler Alternative
Blending achieves a similar goal to stacking with a simpler (if slightly less data-efficient) mechanism: a single, dedicated holdout set instead of full cross-validation.
from sklearn.model_selection import train_test_split
# Split training data further into a base-training set and a blend-holdout set
X_base, X_blend, y_base, y_blend = train_test_split(X_train, y_train, test_size=0.3, random_state=42)
# Train base models ONLY on X_base
rf = RandomForestClassifier(random_state=42).fit(X_base, y_base)
svm = SVC(probability=True).fit(X_base, y_base)
# Generate predictions on the HELD-OUT blend set — these are HONEST,
# since the base models never saw X_blend during training
import numpy as np
blend_features = np.column_stack([
rf.predict_proba(X_blend)[:, 1],
svm.predict_proba(X_blend)[:, 1],
])
# Train the meta-model on these blend-set predictions
meta_model = LogisticRegression().fit(blend_features, y_blend)Stacking vs Blending
─────────────────────────────────────────
Stacking: uses K-FOLD cross-validation — more DATA-
EFFICIENT (every example eventually contributes
to both base and meta training), but more
computationally expensive (K times the base
model training)
Blending: uses a SINGLE holdout split — simpler and
faster to implement, but "wastes" the
holdout portion for base model training,
and its result can be more sensitive to
which SPECIFIC split was chosen
─────────────────────────────────────────
6. Why Diversity Among Base Models Matters
Directly echoing Chapter 1, Section 4's random forest insight — combining models works best when they make different kinds of mistakes, not the same ones.
Diminishing Returns From Similar Models
─────────────────────────────────────────
Stacking TWO very similar models (e.g. two slightly different
random forests) provides LITTLE benefit — if they tend to make
the SAME mistakes, combining them doesn't cancel out much
error at all.
Stacking a LINEAR model (Module 4, Ch.1), a DISTANCE-based
model (Module 4, Ch.2), and a TREE-based model (Module 4,
Ch.3) — each with a fundamentally DIFFERENT way of "thinking"
about the data — provides much MORE benefit, since their
errors are more likely to be independent.
─────────────────────────────────────────
Practical guidance: when choosing base models for stacking, deliberately favor algorithmic diversity over simply throwing in every model you can train — a linear model, a tree-based model, and a distance-based model together typically outperform three variations of the same underlying algorithm.
7. When Stacking Is (and Isn't) Worth the Complexity
Stacking Is Worth It When...
─────────────────────────────────────────
- Squeezing out the LAST few percentage points of performance
genuinely matters (common in competitions, less common in
many production settings)
- You have SEVERAL genuinely diverse, individually decent
models already available
- The added COMPLEXITY (more models to train, maintain, and
explain — Module 9's production concerns) is an acceptable
cost for the accuracy gain
─────────────────────────────────────────
Stacking Is Often NOT Worth It When...
─────────────────────────────────────────
- A SINGLE well-tuned model (especially XGBoost/LightGBM,
Chapter 3) already performs adequately — the marginal gain
from stacking is often SMALL relative to its added complexity
- INTERPRETABILITY matters (a stack of models is much harder
to explain than one model — echoes the AI Notes' Explainable
AI discussion)
- PRODUCTION simplicity matters more than squeezing out
marginal accuracy (Module 9 covers exactly this trade-off)
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Stacking and blending combine predictions from fundamentally different algorithms, exploiting the fact that different model types tend to make different kinds of mistakes.
- Simple voting/averaging ensembles combine model outputs with a fixed rule; stacking trains a meta-model that learns the best way to combine base models' predictions.
- Stacking requires out-of-fold predictions (via cross-validation) to avoid leaking information from a base model's own training data into the meta-model.
- Blending achieves a similar goal more simply, using a single holdout split instead of full cross-validation, at some cost in data efficiency.
- Diversity among base models matters more than sheer quantity — combining a linear model, a tree-based model, and a distance-based model typically outperforms combining several variations of the same algorithm.
Module 5 Complete — Next Module
You now have the full ensemble toolkit: bagging/random forests for variance reduction, boosting for bias reduction, production-grade XGBoost/LightGBM implementations, and stacking/blending for combining fundamentally different models. Module 6 shifts to unsupervised learning — Chapter 2's second paradigm, where there are no labels to predict at all.
Concept Check
- Why does combining fundamentally different algorithms (not just different samples of the same one) often help?
- Why must stacking use out-of-fold predictions rather than base models' predictions on their own training data?
- When would a single well-tuned XGBoost model be a better practical choice than a stacked ensemble?
Next Chapter
→ Module 6: Unsupervised Learning
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index