Unsupervised Learning
K-Means Clustering
Module 1, Chapter 2 introduced unsupervised learning's defining trait: no labels, just discovering structure. Clustering is the most common unsupervised task —
JrCodex·7 min read
Jr Codex ML Notes
Level: Intermediate Prerequisites: Module 5: Ensemble Learning Time to complete: ~25 minutes
Table of Contents
- Finding Structure Without Labels
- The K-Means Algorithm
- K-Means Step by Step
- Choosing K — the Elbow Method
- The Silhouette Score
- Limitations of K-Means
- Feature Scaling Is Mandatory
- Summary & Next Steps
1. Finding Structure Without Labels
Module 1, Chapter 2 introduced unsupervised learning's defining trait: no labels, just discovering structure. Clustering is the most common unsupervised task — grouping similar data points together, without anyone specifying in advance what the groups should be.
import numpy as np
# Customer data with NO pre-existing "segment" label at all
customers = np.array([
[25, 1200], [22, 1350], [30, 1100], # cluster A? (young, low spend)
[52, 8500], [48, 7900], [55, 9200], # cluster B? (older, high spend)
])
# No one told the algorithm these form two groups — it must DISCOVER this2. The K-Means Algorithm
K-Means partitions data into K clusters, each represented by a centroid (the average position of all points currently assigned to it).
The K-Means Algorithm
─────────────────────────────────────────
1. Choose K (the number of clusters) — Section 4 covers HOW
2. Randomly place K initial centroids
3. ASSIGN each data point to its NEAREST centroid
(using Euclidean distance — Module 4, Ch.2's KNN concept, reused)
4. UPDATE each centroid to the AVERAGE position of its
currently-assigned points
5. REPEAT steps 3-4 until assignments stop changing
(the algorithm has CONVERGED)
─────────────────────────────────────────
from sklearn.cluster import KMeans
import numpy as np
X = np.array([
[25, 1200], [22, 1350], [30, 1100],
[52, 8500], [48, 7900], [55, 9200],
])
model = KMeans(n_clusters=2, random_state=42, n_init=10)
model.fit(X)
print(f"Cluster assignments: {model.labels_}") # e.g. [0 0 0 1 1 1]
print(f"Centroid locations:\n{model.cluster_centers_}")3. K-Means Step by Step
import numpy as np
def kmeans_from_scratch(X, k, max_iterations=100):
"""A from-scratch implementation, illustrating the algorithm directly."""
np.random.seed(42)
n_samples = X.shape[0]
# Step 2: randomly initialize centroids by picking K actual data points
centroid_indices = np.random.choice(n_samples, k, replace=False)
centroids = X[centroid_indices].copy()
for iteration in range(max_iterations):
# Step 3: assign each point to its NEAREST centroid
distances = np.array([[np.linalg.norm(x - c) for c in centroids] for x in X])
assignments = np.argmin(distances, axis=1)
# Step 4: update centroids to the MEAN of their assigned points
new_centroids = np.array([
X[assignments == cluster_id].mean(axis=0) for cluster_id in range(k)
])
# Step 5: check for CONVERGENCE — have assignments stabilized?
if np.allclose(centroids, new_centroids):
print(f"Converged after {iteration + 1} iterations")
break
centroids = new_centroids
return assignments, centroids
X = np.array([[25,1200],[22,1350],[30,1100],[52,8500],[48,7900],[55,9200]], dtype=float)
assignments, centroids = kmeans_from_scratch(X, k=2)
print(assignments)K-Means, Visualized
─────────────────────────────────────────
Iteration 1: Iteration 2: Converged:
×(centroid) × ×
● ● ● ● ● ● ● ● ●
(assignments
× ●●● × ●●● × ●●●
(random start) (centroids MOVED (STABLE —
toward their no more changes)
assigned points)
─────────────────────────────────────────
This is conceptually similar to the AI Notes' local search algorithms (hill climbing) — K-Means repeatedly improves its current solution (centroid positions) until no further improvement is found, but like hill climbing, it's not guaranteed to find the globally best clustering (Section 6 covers this limitation).
4. Choosing K — the Elbow Method
Unlike supervised learning (where the "correct" number of classes is usually known), clustering requires choosing K — the elbow method helps guide this choice.
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
inertias = []
k_values = range(1, 10)
for k in k_values:
model = KMeans(n_clusters=k, random_state=42, n_init=10)
model.fit(X)
inertias.append(model.inertia_) # sum of squared distances to nearest centroid
plt.plot(k_values, inertias, marker="o")
plt.xlabel("Number of Clusters (K)")
plt.ylabel("Inertia (within-cluster sum of squares)")
plt.title("The Elbow Method")Reading the Elbow Plot
─────────────────────────────────────────
Inertia
│ ●
│ ●
│ ●
│ ●___
│ ‾‾‾●___●___●___● ← the "ELBOW" — where
│ adding MORE clusters
└───────────────────────────── stops helping much
1 2 3 4 5 6 7 8 9
K
Inertia ALWAYS decreases as K increases (more clusters can
always fit the data at least as well) — the ELBOW POINT is
where this improvement sharply SLOWS DOWN, suggesting
additional clusters aren't capturing genuinely NEW structure.
─────────────────────────────────────────
5. The Silhouette Score
A more quantitative alternative (or complement) to the elbow method — measures how well-separated and internally cohesive clusters actually are.
from sklearn.metrics import silhouette_score
for k in range(2, 8):
model = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = model.fit_predict(X)
score = silhouette_score(X, labels)
print(f"K={k}: silhouette score={score:.3f}")Silhouette Score, Interpreted
─────────────────────────────────────────
Ranges from -1 to +1:
Close to +1: points are WELL-MATCHED to their own cluster,
and CLEARLY SEPARATED from other clusters
Close to 0: clusters OVERLAP significantly —
ambiguous assignment
Close to -1: points may be assigned to the WRONG
cluster entirely
─────────────────────────────────────────
Unlike the elbow method's somewhat subjective "eyeball the bend" approach, silhouette score gives a genuinely comparable number across different K values — a good practical default: pick the K that maximizes silhouette score, then sanity-check against the elbow plot.
6. Limitations of K-Means
Key Limitations
─────────────────────────────────────────
Assumes ROUGHLY SPHERICAL, EQUALLY-SIZED clusters:
K-Means struggles with elongated, irregularly-shaped, or
very differently-sized clusters — Chapter 2's DBSCAN
handles these cases much better
Sensitive to INITIAL centroid placement:
Different random starting points can converge to DIFFERENT
final clusterings (a local optimum problem, directly
echoing the AI Notes' local search limitations) — this is
exactly why scikit-learn's n_init=10 runs the algorithm
MULTIPLE times with different random starts and keeps the
BEST result
Requires CHOOSING K in advance:
Unlike some other clustering methods (Chapter 2's DBSCAN
doesn't require this), K must be specified upfront —
Sections 4-5 exist specifically to guide this choice
─────────────────────────────────────────
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
# K-Means fails on non-spherical clusters — a concrete demonstration
np.random.seed(0)
theta = np.linspace(0, 2*np.pi, 100)
inner_ring = np.column_stack([np.cos(theta), np.sin(theta)]) + np.random.normal(0, 0.1, (100, 2))
outer_ring = np.column_stack([3*np.cos(theta), 3*np.sin(theta)]) + np.random.normal(0, 0.1, (100, 2))
X_rings = np.vstack([inner_ring, outer_ring])
model = KMeans(n_clusters=2, random_state=42, n_init=10)
labels = model.fit_predict(X_rings)
# K-Means will INCORRECTLY split each ring in half, rather than
# correctly identifying "inner ring" and "outer ring" as the two clusters —
# it fundamentally assumes SPHERICAL cluster shapes7. Feature Scaling Is Mandatory
Directly reprising Module 2, Chapter 3 and Module 4, Chapter 2's KNN discussion — K-Means is entirely distance-based, making scaling absolutely essential.
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
# WITHOUT scaling, "annual_spend" (thousands) would completely
# dominate "age" (tens) in the distance calculation
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
model = KMeans(n_clusters=2, random_state=42, n_init=10)
model.fit(X_scaled) # NOW both features contribute fairly8. Summary & Next Steps
Key Takeaways
- K-Means partitions data into K clusters by repeatedly assigning points to their nearest centroid, then updating centroids to the average of their assigned points, until convergence.
- K must be chosen in advance — the elbow method (inertia vs K) and silhouette score (cluster cohesion/separation) both help guide this choice.
- K-Means assumes roughly spherical, similarly-sized clusters and is sensitive to random initialization (mitigated by
n_init, running multiple random starts) — it struggles badly with irregularly-shaped clusters. - Feature scaling is mandatory, since K-Means is entirely distance-based, exactly like KNN.
Concept Check
- Why does K-Means's inertia always decrease as K increases, and what does the "elbow" in an elbow plot actually indicate?
- Why does K-Means struggle with concentric ring-shaped clusters?
- Why does
n_init=10help address K-Means's sensitivity to initial centroid placement?
Next Chapter
→ Chapter 2: Hierarchical Clustering & DBSCAN
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index