Support Vector Machines

Jr Codex ML Notes

Level: Intermediate–Advanced Prerequisites: Chapter 3 Time to complete: ~30 minutes

Finding the Best Possible Boundary
The Maximum Margin Idea
Support Vectors — Where the Name Comes From
Soft Margins — Handling Imperfect Data
When Data Isn't Linearly Separable at All
The Kernel Trick
Common Kernels
Strengths, Weaknesses & When to Use
Summary & Next Steps

1. Finding the Best Possible Boundary

Chapter 1's logistic regression finds a linear decision boundary separating classes — but there are often infinitely many valid boundaries that separate the training data perfectly. Support Vector Machines (SVMs) ask a more specific question: which boundary is genuinely the best?

Many Valid Boundaries — Which Is Best?
─────────────────────────────────────────
      ●  ●              ●  ●              ●  ●
    ●   ●   ╱          ●   ●    |        ●   ●  ╲
        ●  ╱  ○ ○          ●    |  ○ ○         ●   ╲  ○ ○
          ╱   ○  ○           ‾‾‾‾  ○  ○            ╲   ○  ○

  All THREE lines perfectly separate ● from ○ — but they're
  clearly not equally good choices for classifying FUTURE,
  unseen points.
─────────────────────────────────────────

2. The Maximum Margin Idea

SVM's key insight: choose the boundary that is as far as possible from the closest points of either class — maximizing the "margin," the empty buffer zone around the decision boundary.

from sklearn.svm import SVC
import numpy as np
import matplotlib.pyplot as plt
 
np.random.seed(0)
class_0 = np.random.randn(20, 2) + [2, 2]
class_1 = np.random.randn(20, 2) + [6, 6]
X = np.vstack([class_0, class_1])
y = np.array([0]*20 + [1]*20)
 
model = SVC(kernel="linear", C=1000)      # large C = HARD margin (Section 4)
model.fit(X, y)
 
plt.scatter(X[:, 0], X[:, 1], c=y)
# The decision boundary AND the margin lines can be plotted directly
# from model.coef_ and model.intercept_ for a linear kernel

Why Maximum Margin Is a Good Idea
─────────────────────────────────────────
  A boundary that squeezes CLOSE to one class is fragile — a
  new, slightly different point from that class could easily
  fall on the WRONG side.

  A boundary with MAXIMUM margin gives the most "breathing room"
  on both sides — intuitively, the most ROBUST choice for
  classifying NEW, unseen points correctly, directly connecting
  to Module 1, Chapter 4's generalization concern.
─────────────────────────────────────────

3. Support Vectors — Where the Name Comes From

The maximum-margin boundary is determined entirely by the closest points from each class — these critical points are called support vectors, and every other training point could be removed without changing the boundary at all.

print(f"Number of support vectors: {model.n_support_}")      # e.g. [2, 2] — 2 per class
print(f"Support vector indices: {model.support_}")
print(f"The support vectors themselves:\n{model.support_vectors_}")

A Surprising, Genuinely Useful Property
─────────────────────────────────────────
  Only the support vectors (the points CLOSEST to the boundary)
  actually matter for defining it — every other training point,
  however many there are, could be deleted WITHOUT changing the
  learned boundary one bit.

  This is philosophically DIFFERENT from Chapter 2's KNN (which
  uses EVERY stored point for every prediction) and Chapter 1's
  logistic regression (which uses ALL training data to fit its
  coefficients, though the training data itself is then discarded).
─────────────────────────────────────────

4. Soft Margins — Handling Imperfect Data

Real data is rarely perfectly, cleanly separable — a few points often fall on the "wrong" side or awkwardly close to the boundary. The soft margin SVM tolerates some violations, controlled by the parameter C.

from sklearn.svm import SVC
 
model_strict = SVC(kernel="linear", C=1000)      # HIGH C — HARD margin, punishes violations HEAVILY
model_lenient = SVC(kernel="linear", C=0.1)         # LOW C — SOFT margin, tolerates more violations
 
# Both fit the SAME data, but produce DIFFERENT boundaries —
# model_lenient allows more points to sit inside/across the margin,
# in exchange for a potentially WIDER, more GENERALIZABLE margin overall

C — Directly the Bias-Variance Dial (Module 1, Ch.4)
─────────────────────────────────────────
  HIGH C:    tries HARD to classify every training point
               correctly, even outliers → NARROW margin,
               potential OVERFITTING (high variance)

  LOW C:        tolerates more misclassified/borderline
                  training points → WIDER margin, more
                  regularized, potential UNDERFITTING if too low
                  (echoes Module 3, Ch.2's regularization strength)
─────────────────────────────────────────

5. When Data Isn't Linearly Separable at All

Sometimes no straight line (or flat hyperplane) can separate the classes reasonably well, no matter how soft the margin — directly echoing Chapter 1's decision boundary limitation.

import numpy as np
import matplotlib.pyplot as plt
 
# Classic example: concentric circles — genuinely NOT linearly separable
np.random.seed(0)
theta_inner = np.random.uniform(0, 2*np.pi, 50)
theta_outer = np.random.uniform(0, 2*np.pi, 50)
 
inner_circle = np.column_stack([np.cos(theta_inner), np.sin(theta_inner)]) + np.random.normal(0, 0.1, (50, 2))
outer_circle = np.column_stack([3*np.cos(theta_outer), 3*np.sin(theta_outer)]) + np.random.normal(0, 0.1, (50, 2))
 
X = np.vstack([inner_circle, outer_circle])
y = np.array([0]*50 + [1]*50)
 
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("No straight line can separate these two classes")

6. The Kernel Trick

SVM's most powerful and famous idea: instead of giving up on a linear boundary, project the data into a higher-dimensional space where it does become linearly separable — without ever explicitly computing that higher-dimensional transformation.

The Kernel Trick, Conceptually
─────────────────────────────────────────
  In 2D, this data isn't linearly separable — but imagine
  ADDING a third dimension: height = distance from the center.

  Suddenly, the inner circle (small distance = LOW height) and
  outer circle (large distance = HIGH height) become perfectly
  separable by a FLAT plane in this new 3D space!

  A "kernel" computes the EFFECT of this higher-dimensional
  transformation directly, without ever actually constructing
  or storing the (potentially enormous, even infinite-dimensional)
  transformed feature space explicitly — a genuine mathematical
  and computational shortcut.
─────────────────────────────────────────

from sklearn.svm import SVC
 
model_rbf = SVC(kernel="rbf")      # the RBF kernel handles exactly this kind of problem
model_rbf.fit(X, y)
 
# Now the model can correctly separate the two circles,
# despite NO straight line being able to do so in the ORIGINAL 2D space

7. Common Kernels

from sklearn.svm import SVC
 
linear_svm = SVC(kernel="linear")           # no transformation — Sections 1-4's original case
poly_svm = SVC(kernel="poly", degree=3)        # polynomial boundary (echoes Module 3, Ch.3)
rbf_svm = SVC(kernel="rbf", gamma="scale")        # Radial Basis Function — handles complex, curved boundaries

Kernel	Best For
Linear	Data that's already linearly (or nearly linearly) separable — fastest, most interpretable
Polynomial	Curved boundaries with a known, moderate polynomial degree — directly connects to Module 3, Chapter 3
RBF (Radial Basis Function)	Complex, non-linear boundaries with no known shape — the most commonly used default for non-linear problems

RBF's Key Hyperparameter: gamma
─────────────────────────────────────────
  gamma controls how far a SINGLE training point's influence
  reaches:

  LOW gamma:    far-reaching influence → SMOOTHER boundary
                  (higher bias, lower variance)
  HIGH gamma:      very LOCALIZED influence → boundary can
                     wiggle tightly around individual points
                     (lower bias, higher variance — risk of
                     overfitting)

  This is ANOTHER direct instance of Module 1, Ch.4's tradeoff,
  tuned via Module 7's systematic search alongside C.
─────────────────────────────────────────

8. Strengths, Weaknesses & When to Use

Strengths
─────────────────────────────────────────
  - EFFECTIVE in high-dimensional spaces (even when features
    outnumber samples)
  - The kernel trick handles complex, non-linear boundaries
    elegantly, without manual feature engineering (Module 2,
    Ch.4; Module 3, Ch.3)
  - MEMORY efficient at prediction time — only support vectors
    (Section 3) are needed, not the entire training set (unlike
    Chapter 2's KNN)
─────────────────────────────────────────

Weaknesses
─────────────────────────────────────────
  - Training time scales POORLY to very large datasets
    (typically impractical beyond tens of thousands of examples
    without specialized techniques)
  - Requires CAREFUL hyperparameter tuning (C, gamma, kernel
    choice — Module 7's territory) to perform well
  - Less directly INTERPRETABLE than Chapter 1's logistic
    regression or Chapter 3's decision trees, especially with
    non-linear kernels (connects to the AI Notes' Explainable
    AI discussion)
  - Requires feature SCALING (Module 2, Ch.3) — like KNN, it's
    fundamentally distance/margin-based
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

SVMs find the linear boundary that maximizes the margin (buffer zone) between classes, rather than just any boundary that happens to separate them.
Support vectors are the specific closest points that define the boundary — every other training point is irrelevant to the final model, a genuinely distinctive property.
The parameter C controls the soft margin's tolerance for misclassified points, directly analogous to a regularization strength.
The kernel trick projects data into a higher-dimensional space (without explicitly computing it) to find a linear boundary where none existed in the original space — RBF is the standard default for non-linear problems.
SVMs need feature scaling, don't scale well to very large datasets, and require careful tuning of C, gamma, and kernel choice.

Concept Check

Why does maximizing the margin, rather than just finding any separating boundary, tend to generalize better to new data?
What's special about support vectors compared to the rest of the training data?
What problem does the kernel trick solve, and how does the RBF kernel's gamma parameter relate to the bias-variance tradeoff?

Next Chapter

→ Chapter 5: Naive Bayes for Classification

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index