Train/Test Splits & Cross-Validation Basics

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 4 Time to complete: ~25 minutes

Table of Contents

1. Why You Can't Trust Training Accuracy
2. The Train/Test Split
3. The Validation Set — a Third Split
4. K-Fold Cross-Validation
5. Data Leakage — the Silent Killer of Valid Evaluation
6. Stratified Splits for Classification
7. Putting It Together
8. Summary & Next Steps

1. Why You Can't Trust Training Accuracy

Chapter 4 established that an overfit model can achieve near-perfect accuracy on its own training data while failing badly on new data. This chapter covers the practical machinery for detecting that gap honestly — the foundation every algorithm from Module 3 onward will rely on for evaluation.

The Core Principle
─────────────────────────────────────────
  NEVER evaluate a model on the SAME data it was trained on.
  Doing so tells you how well the model MEMORIZED that data,
  not how well it will PERFORM on new, real-world data.
─────────────────────────────────────────

2. The Train/Test Split

The simplest, most fundamental technique: split available data into two parts before training — one to learn from, one to honestly evaluate on.

from sklearn.model_selection import train_test_split
import numpy as np
 
np.random.seed(42)
X = np.random.rand(100, 3)      # 100 examples, 3 features each
y = np.random.randint(0, 2, 100)      # binary labels
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
 
print(f"Training set: {X_train.shape[0]} examples")      # 80
print(f"Test set: {X_test.shape[0]} examples")              # 20

The Workflow
─────────────────────────────────────────
  1. SPLIT the data BEFORE doing anything else
  2. TRAIN the model using ONLY X_train, y_train
  3. EVALUATE using ONLY X_test, y_test — data the model
     has NEVER seen during training
  4. The TEST performance is your HONEST estimate of how the
     model will perform on genuinely new, real-world data
─────────────────────────────────────────

Common Split Ratios
─────────────────────────────────────────
  80/20 or 70/30 (train/test) — the most common defaults
  90/10 — when data is abundant and you want more training data
  50/50 — rare, mainly for very small datasets needing a
            larger relative test set for statistical confidence
─────────────────────────────────────────

random_state=42 ensures reproducibility — the exact same split every time the code runs, which matters enormously for debugging and fair comparison between different models/approaches (a direct parallel to the Python Notes' random.seed()).

3. The Validation Set — a Third Split

A two-way split has a subtle problem: if you try many different models or settings and pick whichever does best on the test set, you're effectively "fitting" your choices to the test set — and its performance estimate becomes optimistically biased. The fix: a third split.

# First split off the test set — LOCKED AWAY until the very end
X_train_full, X_test, y_train_full, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
 
# Then split the REMAINING data into actual training + validation
X_train, X_val, y_train, y_val = train_test_split(
    X_train_full, y_train_full, test_size=0.25, random_state=42      # 0.25 of 80% = 20% of total
)
 
print(f"Train: {len(X_train)}, Validation: {len(X_val)}, Test: {len(X_test)}")      # 60, 20, 20

Three-Way Split — Each Set's Job
─────────────────────────────────────────
  Training set:      the model LEARNS from this
  Validation set:       used to COMPARE different models/settings,
                           and pick the best one (Module 7's
                           hyperparameter tuning happens here)
  Test set:                touched EXACTLY ONCE, at the very end,
                              for a final, honest, unbiased
                              performance estimate
─────────────────────────────────────────

The test set's entire value comes from never being used to make any decision — not which model to pick, not which features to use, not which hyperparameters to set. The moment it influences a decision, it stops being a valid, unbiased estimate of real-world performance.

4. K-Fold Cross-Validation

A single train/validation split can be unlucky — a particular split might happen to be easy or hard by chance. K-fold cross-validation gets a more robust estimate by repeating the process multiple times with different splits.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
 
model = LogisticRegression()
scores = cross_val_score(model, X_train_full, y_train_full, cv=5)      # 5-fold CV
 
print(f"Scores per fold: {scores}")
print(f"Mean accuracy: {scores.mean():.3f} (+/- {scores.std():.3f})")

How K-Fold Cross-Validation Works
─────────────────────────────────────────
  Split the training data into K equal parts ("folds").
  Repeat K times: train on K-1 folds, validate on the
  REMAINING fold — a DIFFERENT fold each time.

  Fold 1: [VALIDATE][train][train][train][train]
  Fold 2: [train][VALIDATE][train][train][train]
  Fold 3: [train][train][VALIDATE][train][train]
  Fold 4: [train][train][train][VALIDATE][train]
  Fold 5: [train][train][train][train][VALIDATE]

  Average the 5 validation scores for a MORE ROBUST estimate
  than any single train/validation split alone.
─────────────────────────────────────────

Why This Is More Reliable Than One Split
─────────────────────────────────────────
  A single validation split gives ONE number — you don't know
  if you got a lucky or unlucky split.

  K-fold gives K numbers — their SPREAD (standard deviation)
  tells you how STABLE the model's performance actually is,
  not just its average.
─────────────────────────────────────────

Common choice: K=5 or K=10 — larger K gives a more thorough estimate but costs more compute (training K separate models); K=5 or K=10 is the standard practical default balancing these concerns.

5. Data Leakage — the Silent Killer of Valid Evaluation

Data leakage occurs when information from outside the training set (often, inadvertently, from the test/validation set) influences training — producing evaluation results that look great but don't reflect real-world performance at all.

# WRONG — scaling BEFORE splitting leaks test-set information into training!
from sklearn.preprocessing import StandardScaler
 
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)      # fit() sees the ENTIRE dataset, including future "test" data!
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
# The scaler's mean/std were computed using TEST data too — a subtle leak!
 
 
# CORRECT — split FIRST, then fit the scaler ONLY on training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
 
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)      # fit() sees ONLY training data
X_test_scaled = scaler.transform(X_test)               # transform() reuses training statistics — no leak

The General Rule Against Leakage
─────────────────────────────────────────
  ANY step that "learns" something from data (scaling
  parameters, feature selection, imputation values — Module 2's
  entire preprocessing toolkit) must be FIT ONLY on the
  training set, then APPLIED (never re-fit) to the
  validation/test sets.

  Violating this makes evaluation results look BETTER than
  they'll actually be in the real world — a dangerous,
  easy-to-miss mistake.
─────────────────────────────────────────

Other Common Leakage Sources
─────────────────────────────────────────
  - Duplicate records split ACROSS train and test sets (the
    model effectively "sees" test examples during training)
  - Using FUTURE information a real deployed model wouldn't
    have access to yet (e.g. using next month's sales figures
    as a feature to predict THIS month's — the model will
    perform beautifully in testing but is USELESS in production,
    since that future data doesn't exist yet at prediction time)
  - Time-based data split RANDOMLY instead of chronologically
    (directly echoing the Data Science Notes' time series chapter)
─────────────────────────────────────────

6. Stratified Splits for Classification

For classification problems (Module 4), a plain random split can accidentally produce a training or test set with a very different class balance than the overall data — stratified splitting prevents this.

from sklearn.model_selection import train_test_split
import numpy as np
 
# Imagine a rare-disease dataset: 95% healthy, 5% disease
y_imbalanced = np.array([0] * 950 + [1] * 50)
X_imbalanced = np.random.rand(1000, 3)
 
# WITHOUT stratification — the test set's class balance is left to CHANCE
X_train, X_test, y_train, y_test = train_test_split(X_imbalanced, y_imbalanced, test_size=0.2)
print(f"Test set positive rate (no stratify): {y_test.mean():.3f}")      # could easily drift from 5%
 
# WITH stratification — GUARANTEES both sets preserve the original 95/5 balance
X_train, X_test, y_train, y_test = train_test_split(
    X_imbalanced, y_imbalanced, test_size=0.2, stratify=y_imbalanced, random_state=42
)
print(f"Test set positive rate (stratified): {y_test.mean():.3f}")      # very close to 5%

This connects directly to Module 2, Chapter 5's imbalanced data chapter — for any classification problem with meaningfully unbalanced classes, stratify=y should be treated as a near-default choice, not an optional extra.

7. Putting It Together

A complete, correct evaluation workflow, combining every technique from this chapter:

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
 
# 1. Split FIRST — test set locked away until the very end
X_train_full, X_test, y_train_full, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
 
# 2. Fit preprocessing ONLY on training data (no leakage)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_full)
X_test_scaled = scaler.transform(X_test)
 
# 3. Use cross-validation on the TRAINING data to compare models/settings
model = LogisticRegression()
cv_scores = cross_val_score(model, X_train_scaled, y_train_full, cv=5)
print(f"Cross-validation accuracy: {cv_scores.mean():.3f}")
 
# 4. ONLY AT THE END — train on all training data, evaluate on test ONCE
model.fit(X_train_scaled, y_train_full)
final_test_accuracy = model.score(X_test_scaled, y_test)
print(f"Final, honest test accuracy: {final_test_accuracy:.3f}")

8. Summary & Next Steps

Key Takeaways

• Never evaluate a model on data it was trained on — a train/test split (and ideally a separate validation set) is essential for an honest performance estimate.
• The test set's value depends on it being touched exactly once, at the very end, uninfluenced by any model or feature selection decisions.
• K-fold cross-validation gives a more robust performance estimate than a single split, and its spread across folds reveals how stable that estimate actually is.
• Data leakage — fitting preprocessing steps on data that includes the test set, or including future information a real deployment wouldn't have — silently inflates evaluation results, always fit preprocessing only on training data.
• Stratified splitting preserves class balance across train/test sets, essential for any meaningfully imbalanced classification problem.

Module 1 Complete — Next Module

You now have the conceptual foundation for all of machine learning: what ML is, the three paradigms, the full project workflow, the bias-variance tradeoff, and how to evaluate a model honestly. Module 2 covers the practical work of getting real data ready for any of the algorithms in Modules 3 onward.

Concept Check

1. Why does using the test set to choose between several models bias its performance estimate?
2. Why must preprocessing steps like scaling be fit only on the training set, not the full dataset?
3. Why is a stratified split particularly important for an imbalanced classification dataset?

Next Chapter

→ Module 2: Data Preparation for ML

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index