Unsupervised Learning
Principal Component Analysis (PCA)
Module 1, Chapter 2 identified two main unsupervised tasks: clustering (Chapters 1-2) and dimensionality reduction — compressing many features into fewer, while
Jr Codex ML Notes
Level: Intermediate–Advanced Prerequisites: Chapter 2: Hierarchical Clustering & DBSCAN Time to complete: ~30 minutes
Table of Contents
- The Second Unsupervised Task: Dimensionality Reduction
- Why Reduce Dimensions At All
- The Core Idea: Finding the Directions of Maximum Variance
- Principal Components
- PCA Step by Step
- Explained Variance — How Many Components to Keep
- PCA for Visualization
- What PCA Loses
- Summary & Next Steps
1. The Second Unsupervised Task: Dimensionality Reduction
Module 1, Chapter 2 identified two main unsupervised tasks: clustering (Chapters 1-2) and dimensionality reduction — compressing many features into fewer, while preserving the important structure.
2. Why Reduce Dimensions At All
Directly connects to Module 2, Chapter 4's curse of dimensionality preview and Module 4, Chapter 2's KNN vulnerability — high-dimensional data creates real, practical problems.
Problems With Too Many Features
─────────────────────────────────────────
- The curse of dimensionality (Module 2, Ch.4; Module 4, Ch.2) —
distance-based algorithms become unreliable as dimensions grow
- Increased overfitting risk (Module 1, Ch.4) — more features
relative to data size means more opportunity to fit noise
- Slower training and more memory usage
- Impossible to VISUALIZE directly beyond 3 dimensions —
dimensionality reduction to 2D/3D is often the ONLY way to
actually SEE high-dimensional data (Section 7)
─────────────────────────────────────────
3. The Core Idea: Finding the Directions of Maximum Variance
PCA's central insight: find new axes (directions) along which the data varies the most — these directions capture the most "information" about how the data spreads out.
Why VARIANCE Matters Here
─────────────────────────────────────────
A feature (or direction) with LOW variance carries LITTLE
information — if every example has almost the SAME value
along some direction, that direction doesn't help
DISTINGUISH between examples at all.
A feature (or direction) with HIGH variance carries MORE
information — examples are meaningfully SPREAD OUT along
it, which is exactly what's useful for distinguishing and
understanding different data points.
─────────────────────────────────────────
PCA, Visualized in 2D
─────────────────────────────────────────
y
│ ●
│ ● ●
│ ● ● ● PC1 (direction of MAXIMUM
│● ● ● ↗ variance — the data's
│ ● ● ↗ "long axis")
│ ● ↗
└──────────────── x
PC1 captures the direction the data varies MOST along —
PROJECTING every point onto just this ONE line captures MOST
of the data's original spread, using only 1 dimension
instead of 2.
─────────────────────────────────────────
4. Principal Components
Each principal component is a new axis — a specific direction, computed to capture progressively less of the remaining variance, and guaranteed to be uncorrelated (perpendicular) to all previous components.
Properties of Principal Components
─────────────────────────────────────────
PC1: captures the MOST variance possible, in ANY direction
PC2: captures the MOST of the REMAINING variance,
subject to being PERPENDICULAR to PC1
PC3: captures the MOST of what's left, perpendicular
to BOTH PC1 and PC2
... and so on, for as many original dimensions as exist
─────────────────────────────────────────
Because each component is perpendicular (mathematically: orthogonal) to all others, they capture genuinely non-overlapping information — directly solving Module 2, Chapter 2's multicollinearity concern as a side effect, since principal components are guaranteed uncorrelated with each other by construction.
5. PCA Step by Step
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
np.random.seed(0)
# Simulating correlated features (common in real data)
X = np.random.randn(200, 5)
X[:, 1] = X[:, 0] * 0.8 + np.random.randn(200) * 0.2 # feature 1 correlated with feature 0
X[:, 2] = X[:, 0] * 0.6 + np.random.randn(200) * 0.3 # feature 2 also correlated with feature 0
# Feature scaling FIRST — mandatory, Section 8 explains why
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
pca = PCA(n_components=2) # reduce from 5 dimensions down to 2
X_reduced = pca.fit_transform(X_scaled)
print(f"Original shape: {X.shape}") # (200, 5)
print(f"Reduced shape: {X_reduced.shape}") # (200, 2)# Inspecting WHAT each component represents — a linear combination of ORIGINAL features
print("Component loadings (how much each original feature contributes):")
print(pca.components_)
# e.g. PC1 = 0.55*feature0 + 0.51*feature1 + 0.48*feature2 + 0.02*feature3 + 0.01*feature4
# (heavily weighted toward the CORRELATED features 0,1,2 — makes sense,
# since they share most of their variance)6. Explained Variance — How Many Components to Keep
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy as np
pca_full = PCA() # keep ALL components, to examine variance distribution
pca_full.fit(X_scaled)
explained_variance_ratio = pca_full.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance_ratio)
print(f"Variance explained per component: {explained_variance_ratio.round(3)}")
print(f"Cumulative variance explained: {cumulative_variance.round(3)}")
plt.plot(range(1, len(cumulative_variance)+1), cumulative_variance, marker="o")
plt.axhline(y=0.95, color="red", linestyle="--", label="95% threshold")
plt.xlabel("Number of Components")
plt.ylabel("Cumulative Explained Variance")
plt.legend()A Common Practical Rule
─────────────────────────────────────────
Keep enough components to explain a CHOSEN threshold of
total variance — commonly 90-95%.
This directly echoes Chapter 1's elbow method — plotting
cumulative explained variance against component count, and
choosing where additional components stop adding much.
─────────────────────────────────────────
# Alternatively, let scikit-learn choose the component count automatically,
# given a target variance threshold
pca_auto = PCA(n_components=0.95) # keep enough components for 95% variance
X_reduced_auto = pca_auto.fit_transform(X_scaled)
print(f"Components needed for 95% variance: {pca_auto.n_components_}")7. PCA for Visualization
One of PCA's most common practical uses: compressing high-dimensional data down to 2 or 3 dimensions specifically to visualize it — directly extending the Data Science Notes' EDA visualization instincts to data that can't be plotted directly.
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
iris = load_iris() # 4-dimensional data — impossible to plot directly in full
X, y = iris.data, iris.target
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y)
plt.xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.1%} of variance)")
plt.ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.1%} of variance)")
plt.title("Iris Dataset, Compressed to 2D via PCA")
# The three species often form visually distinct clusters, even after
# compressing from 4 dimensions down to just 2!This is often the very FIRST thing worth trying on a new high-dimensional dataset — a quick PCA-to-2D visualization can reveal natural groupings or outliers that would otherwise be completely invisible, directly complementing the Data Science Notes' EDA workflow.
8. What PCA Loses
The Trade-offs of Dimensionality Reduction
─────────────────────────────────────────
Information loss: unless you keep ALL components, SOME
variance (information) is genuinely
discarded — the tradeoff is
explicitly quantified by Section 6's
explained variance ratio
Interpretability loss: principal components are LINEAR
COMBINATIONS of original features
("0.55×income + 0.3×age - 0.1×tenure")
— much HARDER to interpret directly
than the original, named features
(echoes the AI Notes' Explainable
AI interpretability trade-off)
Feature scaling is MANDATORY: PCA finds directions of maximum
VARIANCE — an unscaled feature
with naturally large numeric
range would DOMINATE the
principal components purely
due to scale, not genuine
importance (exactly Module 2,
Ch.3's scaling argument, now
applied to PCA specifically)
─────────────────────────────────────────
9. Summary & Next Steps
Key Takeaways
- PCA finds new axes (principal components) capturing the most variance in the data, each guaranteed uncorrelated with the others by construction.
- The first component captures the most variance possible; each subsequent component captures the most remaining variance while staying perpendicular to prior ones.
- The number of components to keep is typically chosen via a target cumulative explained variance threshold (commonly 90-95%), directly echoing the elbow method's logic.
- PCA is one of the most common tools for visualizing high-dimensional data by compressing it to 2-3 dimensions, often revealing structure invisible in the original feature space.
- PCA trades away some information and interpretability (principal components are linear combinations of original features, harder to name/explain) and requires feature scaling beforehand, since it's inherently variance/scale-sensitive.
Concept Check
- Why must principal components be perpendicular (orthogonal) to each other?
- Why is feature scaling mandatory before applying PCA?
- What's the trade-off PCA makes when compressing data to fewer dimensions?
Next Chapter
→ Chapter 4: t-SNE, UMAP & Association Rules
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index