Feature Scaling & Normalization

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 2 Time to complete: ~20 minutes

Why Scale Matters for ML Specifically
Standardization (Z-Score Scaling)
Min-Max Scaling
Robust Scaling — for Data with Outliers
Which Algorithms Actually Need Scaling
The Golden Rule — Fit on Training Data Only
Summary & Next Steps

1. Why Scale Matters for ML Specifically

The Data Science Notes' feature engineering chapter introduced scaling briefly. This chapter explains precisely why it matters for ML algorithms — not just as a good practice, but as a genuine correctness issue for certain algorithm families.

# Imagine two features on wildly different scales:
example = {"age": 34, "annual_income": 65000}
 
# A distance-based algorithm (Module 4's KNN) computing "similarity"
# between two people would be COMPLETELY dominated by income,
# simply because its raw NUMBERS are thousands of times larger —
# even if age is EQUALLY or MORE important to the actual problem

The Core Problem
─────────────────────────────────────────
  distance = sqrt((age1-age2)² + (income1-income2)²)

  A 30-year age difference contributes 900 to this sum.
  A $10,000 income difference contributes 100,000,000.

  Income TRIVIALLY dominates, regardless of which feature
  actually matters more for the real-world problem — a scale
  ARTIFACT, not a genuine signal.
─────────────────────────────────────────

2. Standardization (Z-Score Scaling)

Rescales each feature to have mean 0 and standard deviation 1 — directly reusing the Data Science Notes' Z-score formula (Module 1, Chapter 1), now applied as a preprocessing step.

from sklearn.preprocessing import StandardScaler
import numpy as np
 
X = np.array([[25, 50000], [35, 65000], [45, 80000], [30, 55000]])
 
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled)
# Both columns now have mean ≈ 0, std ≈ 1 — directly comparable scales
 
print(f"Learned means: {scaler.mean_}")          # [33.75, 62500]
print(f"Learned std devs: {scaler.scale_}")         # [7.39, 11456]

Standardization Formula (from the Data Science Notes)
─────────────────────────────────────────
  z = (x - mean) / std_dev

  After transformation: mean = 0, std_dev = 1, for EVERY feature
─────────────────────────────────────────

Standardization is the standard default choice for most ML algorithms, since it doesn't assume any particular bounded range and handles moderately-skewed data reasonably well.

3. Min-Max Scaling

Rescales each feature to a fixed range, typically [0, 1] — useful when an algorithm expects or benefits from bounded input.

from sklearn.preprocessing import MinMaxScaler
 
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled)
# Every column now ranges from EXACTLY 0 to 1
 
print(f"Learned min: {scaler.data_min_}")      # [25, 50000]
print(f"Learned max: {scaler.data_max_}")        # [45, 80000]

Standardization vs Min-Max — When to Use Which
─────────────────────────────────────────
  Standardization:   result range is UNBOUNDED (can exceed
                        [0,1] for extreme values) — the safer
                        general-purpose default

  Min-Max scaling:      result is STRICTLY bounded to [0,1] —
                           required by some algorithms (e.g.
                           certain neural network activation
                           functions, covered in the Deep
                           Learning Notes) and useful when you
                           need a guaranteed range

  Both are SENSITIVE to outliers — an extreme value distorts
  either scaling for every OTHER data point (Section 4 addresses
  this directly)
─────────────────────────────────────────

4. Robust Scaling — for Data with Outliers

Both prior methods use the mean/std or min/max — statistics themselves highly sensitive to outliers (Data Science Notes, Module 1, Chapter 1). Robust scaling uses the median and IQR instead, exactly mirroring that chapter's robust-statistics argument.

from sklearn.preprocessing import RobustScaler
import numpy as np
 
X_with_outlier = np.array([[25], [30], [35], [40], [500]])      # 500 is a clear outlier
 
standard_scaled = StandardScaler().fit_transform(X_with_outlier)
robust_scaled = RobustScaler().fit_transform(X_with_outlier)
 
print("Standard scaling:", standard_scaled.ravel())
print("Robust scaling:", robust_scaled.ravel())

Why Robust Scaling Handles This Better
─────────────────────────────────────────
  StandardScaler's MEAN and STD are both dramatically pulled by
  the single outlier (500) — distorting the scaled values for
  EVERY other, genuinely normal data point.

  RobustScaler's MEDIAN and IQR (Data Science Notes, Module 1,
  Ch.1) are barely affected by one extreme value — the other
  four points scale sensibly, exactly the same robustness
  argument from that earlier chapter, now applied as a
  preprocessing technique.
─────────────────────────────────────────

Rule of thumb: if EDA (Data Science Notes, Module 3) revealed significant outliers in a numeric feature, prefer RobustScaler; otherwise, StandardScaler is a perfectly good default.

5. Which Algorithms Actually Need Scaling

Not every algorithm is sensitive to feature scale — knowing which ones are is genuinely useful, practical knowledge that saves unnecessary preprocessing work.

Algorithms That NEED Scaling
─────────────────────────────────────────
  - K-Nearest Neighbors (Module 4) — directly computes distances
  - Support Vector Machines (Module 4) — distance/margin-based
  - Linear/Logistic Regression WITH regularization (Module 3,
    Ch.2) — the penalty term treats all coefficients comparably,
    which only makes sense if features are on comparable scales
  - Principal Component Analysis (Module 6) — variance-based;
    a large-scale feature would dominate the "important
    directions" purely due to scale, not genuine variability
  - Neural Networks (Deep Learning Notes) — training stability
    and convergence speed depend heavily on comparable input scales
─────────────────────────────────────────

Algorithms That DON'T Need Scaling
─────────────────────────────────────────
  - Decision Trees (Module 4) — splits are based on threshold
    comparisons WITHIN one feature at a time, unaffected by
    other features' scales
  - Random Forests, Gradient Boosting (Module 5) — built from
    trees, inheriting this same scale-invariance
  - Naive Bayes (Module 4) — based on probability distributions
    per feature independently, not distances or magnitudes
    compared ACROSS features
─────────────────────────────────────────

Practical implication: scaling tree-based models (decision trees, random forests, gradient boosting) is harmless but unnecessary — while skipping it for KNN, SVM, or regularized regression can silently and significantly hurt performance.

6. The Golden Rule — Fit on Training Data Only

Directly reapplying Module 1, Chapter 5's anti-leakage principle, since scaling is one of the most common places this mistake actually happens in practice:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
 
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)      # fit_transform: LEARN scale from training data
X_test_scaled = scaler.transform(X_test)               # transform ONLY: REUSE training-derived scale
 
# NEVER do this — it leaks test-set statistics into the "learned" scale:
# X_test_scaled_WRONG = scaler.fit_transform(X_test)

fit_transform() vs transform() — the Critical Distinction
─────────────────────────────────────────
  .fit_transform(X_train)   → LEARNS mean/std (or min/max) FROM
                                this data, then applies it

  .transform(X_test)           → APPLIES the ALREADY-LEARNED
                                    statistics, without
                                    recalculating them
─────────────────────────────────────────

This exact pattern — fit_transform on training, transform only on test — recurs identically for every preprocessing step in this module, and is precisely what Chapter 1's Pipeline object automates and enforces correctly by construction.

7. Summary & Next Steps

Key Takeaways

Feature scaling matters because raw numeric scales are arbitrary — a feature measured in the thousands can trivially dominate distance or coefficient-based calculations, regardless of its true importance.
Standardization (mean 0, std 1) is the general-purpose default; min-max scaling bounds features to a fixed range like [0,1]; robust scaling (median/IQR) handles outliers better than either.
Distance-based and regularized-linear algorithms (KNN, SVM, regularized regression, PCA, neural networks) need scaling; tree-based algorithms (decision trees, random forests, gradient boosting) don't.
Always fit scaling parameters only on training data (fit_transform), then apply them unchanged to test data (transform) — never refit on test data, which would leak information.

Concept Check

Why does a feature like "annual income" risk dominating a KNN distance calculation compared to "age," even if age matters equally?
Why is RobustScaler a better choice than StandardScaler for a feature with significant outliers?
Why don't decision trees and random forests require feature scaling?

Next Chapter

→ Chapter 4: Feature Engineering & Selection for ML

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index