Cross-Validation Strategies

Jr Codex ML Notes

Level: Intermediate–Advanced Prerequisites: Module 1, Chapter 5: Train/Test Splits & Cross-Validation Basics Time to complete: ~25 minutes

Beyond Plain K-Fold
Stratified K-Fold — Recapped and Extended
Time Series Cross-Validation
Group K-Fold — Preventing Group Leakage
Leave-One-Out Cross-Validation
Nested Cross-Validation
Choosing the Right Strategy
Summary & Next Steps

1. Beyond Plain K-Fold

Module 1, Chapter 5 introduced K-fold cross-validation as the standard way to get a robust performance estimate. Plain K-fold assumes data points are independent and identically distributed — an assumption that breaks down in several common, important scenarios this chapter addresses.

2. Stratified K-Fold — Recapped and Extended

Module 1, Chapter 5 briefly introduced stratified splitting for a single train/test split — StratifiedKFold extends this same idea across every fold of cross-validation.

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
import numpy as np
 
np.random.seed(0)
X = np.random.rand(200, 3)
y = np.array([0]*180 + [1]*20)      # imbalanced, echoing Module 2, Ch.5
 
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = LogisticRegression()
 
scores = cross_val_score(model, X, y, cv=skf, scoring="f1")      # Module 4, Ch.6's F1
print(f"F1 scores per fold: {scores}")
print(f"Mean F1: {scores.mean():.3f}")

Why This Matters for EVERY Fold, Not Just the Test Set
─────────────────────────────────────────
  Without stratification, plain K-Fold might randomly produce
  a fold with almost NO positive examples at all — making that
  fold's evaluation nearly meaningless for an imbalanced
  problem (Module 2, Ch.5).

  StratifiedKFold guarantees EVERY fold preserves the
  ORIGINAL class balance — a near-default choice for any
  classification cross-validation.
─────────────────────────────────────────

3. Time Series Cross-Validation

Directly extending the Data Science Notes' time series chapter's warning — randomly shuffling time-ordered data before splitting lets a model "see the future," producing a misleadingly optimistic evaluation.

from sklearn.model_selection import TimeSeriesSplit
import numpy as np
 
X = np.arange(20).reshape(-1, 1)      # 20 sequential time points
y = np.arange(20)
 
tscv = TimeSeriesSplit(n_splits=4)
 
for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
    print(f"Fold {fold}: train={train_idx}, test={test_idx}")

Time Series Split, Visualized
─────────────────────────────────────────
  Fold 1:  [train train][test]
  Fold 2:  [train train train][test]
  Fold 3:  [train train train train][test]
  Fold 4:  [train train train train train][test]

  Each fold's TEST set comes STRICTLY AFTER its training
  set — the model NEVER trains on future data to predict
  the past, exactly the chronological constraint the Data
  Science Notes' time series chapter established.
─────────────────────────────────────────

Using plain KFold on time series data is a subtle, common, and serious mistake — it can make a model look far more accurate during validation than it will ever be in real deployment, since real deployment can never actually use future data to predict the past.

4. Group K-Fold — Preventing Group Leakage

When data contains natural groups (e.g., multiple records from the same patient, multiple transactions from the same customer), randomly splitting individual rows can leak information — the model might see one of a patient's records in training and another in testing, making evaluation unrealistically easy.

from sklearn.model_selection import GroupKFold
import numpy as np
 
X = np.arange(12).reshape(-1, 1)
y = np.arange(12)
groups = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4])      # e.g. patient IDs — 3 records EACH
 
gkf = GroupKFold(n_splits=2)
 
for fold, (train_idx, test_idx) in enumerate(gkf.split(X, y, groups=groups)):
    print(f"Fold {fold}: train groups={set(groups[train_idx])}, test groups={set(groups[test_idx])}")
    # GUARANTEED: no group appears in BOTH train and test simultaneously

Why This Matters — a Concrete Failure Mode
─────────────────────────────────────────
  Imagine predicting whether a medical scan shows a tumor, with
  MULTIPLE scans per patient. If patient #7's scans are split
  ACROSS train and test, the model can partly "recognize" that
  specific patient's tissue characteristics from training,
  rather than genuinely learning to detect tumors in scans it's
  NEVER seen anything related to — a serious form of leakage
  (Module 1, Ch.5) that inflates apparent performance.

  GroupKFold guarantees ALL of any given group's records stay
  ENTIRELY within either train OR test, never split across both.
─────────────────────────────────────────

5. Leave-One-Out Cross-Validation

An extreme version of K-fold where K equals the number of samples — each fold tests on exactly one example, training on all the rest.

from sklearn.model_selection import LeaveOneOut, cross_val_score
from sklearn.linear_model import LogisticRegression
import numpy as np
 
X_small = np.random.rand(20, 3)      # a SMALL dataset — LOO is expensive on large ones
y_small = np.random.randint(0, 2, 20)
 
loo = LeaveOneOut()
model = LogisticRegression()
scores = cross_val_score(model, X_small, y_small, cv=loo)
print(f"Number of folds: {len(scores)}")      # 20 — one PER example
print(f"Mean accuracy: {scores.mean():.3f}")

Trade-offs of Leave-One-Out
─────────────────────────────────────────
  Pro:   uses NEARLY all data for training in every fold —
            maximizes training data usage, useful for VERY
            small datasets

  Con:      requires training N SEPARATE models (N = dataset
              size) — computationally PROHIBITIVE for anything
              beyond small datasets
  Con:         each test fold has only ONE example — individual
                 fold scores are noisy (though the AVERAGE
                 across all N folds is still meaningful)
─────────────────────────────────────────

6. Nested Cross-Validation

A subtle but important issue: if you use cross-validation to both tune hyperparameters (Chapter 2) and estimate final performance using the same folds, the performance estimate becomes optimistically biased — directly echoing Module 1, Chapter 5's validation-set contamination concern.

from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn.svm import SVC
import numpy as np
 
X = np.random.rand(100, 3)
y = np.random.randint(0, 2, 100)
 
# OUTER loop: for an HONEST final performance estimate
outer_cv = KFold(n_splits=5, shuffle=True, random_state=42)
 
# INNER loop: for hyperparameter tuning WITHIN each outer fold
inner_cv = KFold(n_splits=3, shuffle=True, random_state=1)
 
param_grid = {"C": [0.1, 1, 10], "kernel": ["linear", "rbf"]}
model = GridSearchCV(SVC(), param_grid, cv=inner_cv)
 
nested_scores = cross_val_score(model, X, y, cv=outer_cv)
print(f"Nested CV scores: {nested_scores}")
print(f"Honest performance estimate: {nested_scores.mean():.3f}")

Why Nesting Is Necessary
─────────────────────────────────────────
  Single-level CV used for BOTH tuning AND evaluation:
    The hyperparameters get "tuned" to perform well on the
    SAME folds later reported as the performance estimate —
    an optimistic bias, similar to Module 1, Ch.5's warning
    against letting the test set influence any decision.

  Nested CV: the OUTER loop provides genuinely UNSEEN data for
    the final estimate; the INNER loop handles tuning
    ENTIRELY separately, within each outer training fold —
    the outer test fold NEVER influences which hyperparameters
    were chosen.
─────────────────────────────────────────

7. Choosing the Right Strategy

Decision Guide
─────────────────────────────────────────
  Classification with imbalanced classes (Module 2, Ch.5)
      → StratifiedKFold (Section 2) — near-default

  Time-ordered data (echoes the Data Science Notes'
  time series chapter)
      → TimeSeriesSplit (Section 3) — mandatory, not optional

  Data has natural GROUPS (patients, customers, sessions)
  with multiple records each
      → GroupKFold (Section 4) — mandatory to prevent leakage

  Very SMALL dataset, need to maximize training data usage
      → Leave-One-Out (Section 5), if computationally feasible

  Tuning hyperparameters AND need an HONEST final performance
  estimate
      → Nested cross-validation (Section 6)
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

Plain K-fold assumes independent, identically distributed data — an assumption that breaks for time series, grouped, and imbalanced data, each requiring a specialized strategy.
StratifiedKFold preserves class balance across every fold, essential for imbalanced classification problems.
TimeSeriesSplit enforces chronological train/test ordering within every fold, preventing a model from ever training on future data to predict the past.
GroupKFold ensures all records from the same group (patient, customer) stay entirely within either train or test, preventing group-level leakage.
Nested cross-validation separates hyperparameter tuning from final performance estimation, avoiding the optimistic bias of tuning and evaluating on the same folds.

Concept Check

Why does using plain KFold on time series data risk an unrealistically optimistic evaluation?
What specific leakage problem does GroupKFold prevent that plain StratifiedKFold would not catch?
Why does using the same cross-validation folds for both hyperparameter tuning and final performance reporting introduce bias?

Next Chapter

→ Chapter 2: Grid Search & Random Search

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index