ML In Production And Capstone
Capstone Project: End-to-End ML System
A telecom company wants to predict customer churn — which customers are likely to cancel their subscription in the next 30 days — so retention offers can be tar
JrCodex·6 min read
Jr Codex ML Notes
Level: Advanced Prerequisites: All of Modules 1–9 (Chapters 1-3) Time to complete: ~50 minutes
Table of Contents
- The Scenario
- Phase 1 — Problem Definition
- Phase 2 — Data Preparation
- Phase 3 — Baseline & Model Comparison
- Phase 4 — Ensemble & Tuning
- Phase 5 — Final Evaluation
- Phase 6 — Building the Deployable Pipeline
- Phase 7 — Serving & Monitoring Plan
- Where to Go From Here
1. The Scenario
A telecom company wants to predict customer churn — which customers are likely to cancel their subscription in the next 30 days — so retention offers can be targeted proactively. This single project pulls in a technique from nearly every module in this curriculum.
customer_data.csv — columns
─────────────────────────────────────────
customer_id, tenure_months, monthly_charges, total_charges,
contract_type, payment_method, internet_service, tech_support,
num_support_calls, churned (0/1 — the TARGET)
─────────────────────────────────────────
2. Phase 1 — Problem Definition
Applying Module 1, Chapter 3's workflow discipline before touching data:
Precise Problem Framing
─────────────────────────────────────────
Paradigm (Module 1, Ch.2): SUPERVISED — CLASSIFICATION
(churned: yes/no)
Performance measure: RECALL matters more than
precision — missing a
true churner (losing
them silently) is more
costly than a wasted
retention offer to a
happy customer (Module
4, Ch.6's precision/
recall trade-off)
Success threshold: Catch at least 70% of
actual churners
(recall >= 0.70),
while keeping
precision reasonably
usable (>= 0.40, so
retention offers
aren't wasted on too
many false alarms)
─────────────────────────────────────────
3. Phase 2 — Data Preparation
Applying Module 2's complete toolkit:
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("customer_data.csv")
# Split FIRST (Module 1, Ch.5) — before any preprocessing decisions
X = df.drop(["customer_id", "churned"], axis=1)
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42 # stratify — Module 2, Ch.5's imbalance concern
)
print(f"Churn rate in training data: {y_train.mean():.2%}") # e.g. 15% — meaningfully imbalancedfrom sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
numeric_features = ["tenure_months", "monthly_charges", "total_charges", "num_support_calls"]
categorical_features = ["contract_type", "payment_method", "internet_service", "tech_support"]
preprocessor = ColumnTransformer([
("num", Pipeline([
("imputer", SimpleImputer(strategy="median")), # Module 2, Ch.1
("scaler", StandardScaler()), # Module 2, Ch.3
]), numeric_features),
("cat", Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore")), # Module 2, Ch.2
]), categorical_features),
])4. Phase 3 — Baseline & Model Comparison
Applying Module 7, Chapter 4's discipline — start simple, compare rigorously:
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) # Module 7, Ch.1
candidates = {
"Dummy (baseline)": DummyClassifier(strategy="stratified"),
"Logistic Regression": LogisticRegression(class_weight="balanced"), # Module 2, Ch.5's class weights
"Decision Tree": DecisionTreeClassifier(class_weight="balanced", max_depth=8),
"Random Forest": RandomForestClassifier(class_weight="balanced", n_estimators=100, random_state=42),
}
results = {}
for name, model in candidates.items():
pipeline = Pipeline([("preprocessor", preprocessor), ("classifier", model)])
scores = cross_val_score(pipeline, X_train, y_train, cv=cv, scoring="recall") # our chosen metric
results[name] = scores.mean()
print(f"{name}: recall={scores.mean():.3f} (+/- {scores.std():.3f})")5. Phase 4 — Ensemble & Tuning
Applying Module 5's ensembles and Module 7's systematic tuning to the most promising candidate:
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
import xgboost as xgb
xgb_pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", xgb.XGBClassifier(random_state=42, scale_pos_weight=5)), # Module 2, Ch.5's imbalance fix
])
param_distributions = {
"classifier__n_estimators": randint(50, 300),
"classifier__max_depth": randint(3, 10),
"classifier__learning_rate": uniform(0.01, 0.3),
"classifier__subsample": uniform(0.6, 0.4),
}
search = RandomizedSearchCV(
xgb_pipeline, param_distributions, n_iter=50, cv=cv,
scoring="recall", random_state=42, n_jobs=-1,
)
search.fit(X_train, y_train)
print(f"Best parameters: {search.best_params_}")
print(f"Best cross-validation recall: {search.best_score_:.3f}")6. Phase 5 — Final Evaluation
Applying Module 4, Chapter 6's full evaluation toolkit, and Module 7, Chapter 4's statistical comparison, on the untouched test set:
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from scipy import stats
best_pipeline = search.best_estimator_
test_predictions = best_pipeline.predict(X_test)
test_probabilities = best_pipeline.predict_proba(X_test)[:, 1]
print(classification_report(y_test, test_predictions))
print(f"Confusion matrix:\n{confusion_matrix(y_test, test_predictions)}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
# Threshold tuning (Module 4, Ch.1) — recall matters more, adjust the cutoff
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_test, test_probabilities)
# Find the LOWEST threshold that still achieves our 70% recall requirement (Phase 1)
for p, r, t in zip(precisions, recalls, thresholds):
if r >= 0.70:
print(f"Threshold={t:.3f}: precision={p:.3f}, recall={r:.3f}")
break7. Phase 6 — Building the Deployable Pipeline
Applying Chapter 1's pipeline and serialization practices:
import joblib
from datetime import datetime
# The FULL pipeline — preprocessing AND the tuned model, as ONE object
final_pipeline = best_pipeline
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
joblib.dump(final_pipeline, f"churn_model_v{timestamp}.joblib")
metadata = {
"version": timestamp,
"test_recall": recalls[recalls >= 0.70][0] if any(recalls >= 0.70) else None,
"chosen_threshold": t,
"training_date": timestamp,
"notes": "XGBoost, tuned via RandomizedSearchCV, threshold adjusted for 70% recall target",
}
print(metadata)8. Phase 7 — Serving & Monitoring Plan
Applying Chapters 2-3's deployment and monitoring practices, as a concrete plan (not fully implemented here, but scoped):
Deployment Plan
─────────────────────────────────────────
Serving style (Ch.2, Sec.2): BATCH — churn risk scores
computed NIGHTLY, fed into
the retention team's
dashboard each morning
(no need for real-time
serving for this use case)
Monitoring (Ch.3): - Track INPUT feature drift
(PSI, Ch.3, Sec.5) on
tenure_months, monthly_charges
- Track PREDICTION distribution
(Ch.3, Sec.6) — % of
customers flagged as
at-risk, week over week
- Track ACTUAL churn rate
among flagged customers,
once outcomes are known
(concept drift check, Ch.3,
Sec.4)
Retraining trigger (Ch.3, Sec.7): Retrain MONTHLY on a
schedule, PLUS
immediately if PSI
exceeds 0.25 on any
key feature
─────────────────────────────────────────
9. Where to Go From Here
Module → Technique Used in This Capstone
─────────────────────────────────────────
Module 1 (Foundations) → problem framing, bias-variance
awareness, train/test discipline
Module 2 (Data Preparation) → imputation, encoding, scaling,
class weights for imbalance
Module 3-4 (Regression/ logistic regression, decision
Classification) trees as comparison baselines
Module 5 (Ensemble Learning) XGBoost as the final,
tuned model
Module 6 (Unsupervised) (not directly used here
— but would apply to
a follow-up customer
SEGMENTATION project)
Module 7 (Model Selection) cross-validation,
randomized search,
statistical
comparison
Module 8 (Reinforcement (not directly used
Learning) here — would
apply to
optimizing WHICH
retention OFFER
to show each
customer, a
genuine RL/
bandit problem)
Module 9 (This module) pipelines,
deployment,
monitoring
─────────────────────────────────────────
You've now completed a full, realistic ML project end-to-end, using nearly every technique across this entire curriculum. Chapter 5 closes out this module — and the whole Machine Learning curriculum — by mapping out exactly where to go next.
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index