Regularization: Ridge, Lasso & Elastic Net

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 1 Time to complete: ~25 minutes

Table of Contents

1. When Linear Regression Overfits
2. The Core Idea: Penalizing Complexity
3. Ridge Regression (L2 Regularization)
4. Lasso Regression (L1 Regularization)
5. Why Lasso Zeroes Out Coefficients but Ridge Doesn't
6. Elastic Net — the Best of Both
7. Choosing the Regularization Strength
8. Summary & Next Steps

1. When Linear Regression Overfits

Chapter 1's plain linear regression can still overfit, especially with many features relative to the amount of training data (Module 1, Chapter 4's bias-variance tradeoff) — coefficients can grow extremely large, fitting noise in the training data precisely.

import numpy as np
from sklearn.linear_model import LinearRegression
 
# Simulating a scenario with MANY features but LITTLE data — prone to overfitting
np.random.seed(0)
n_samples, n_features = 20, 15
X = np.random.randn(n_samples, n_features)
y = X[:, 0] * 3 + np.random.randn(n_samples) * 0.5      # only ONE feature genuinely matters
 
model = LinearRegression()
model.fit(X, y)
print(f"Coefficient magnitudes: {np.abs(model.coef_).round(2)}")
# Many coefficients end up SURPRISINGLY large, even for features
# that have NO real relationship to y — the model is fitting NOISE

Why This Happens
─────────────────────────────────────────
  With MORE features than the data can reliably support, plain
  OLS (Chapter 1) will happily assign LARGE coefficients to
  features that only APPEAR useful due to random chance in this
  particular sample — a direct, concrete instance of Module 1,
  Chapter 4's overfitting.
─────────────────────────────────────────

2. The Core Idea: Penalizing Complexity

Regularization adds a penalty term to the loss function that discourages large coefficients — trading a small amount of training-data fit for a model that generalizes better to new data.

The Regularized Loss Function, General Form
─────────────────────────────────────────
  Total Loss = Sum of Squared Errors  +  λ × (penalty on coefficients)
                     │                        │
              Chapter 1's usual              NEW — discourages
              OLS objective                    large coefficients

  λ (lambda, or "alpha" in scikit-learn) controls how STRONGLY
  this penalty is enforced.
─────────────────────────────────────────

The Trade-off This Creates
─────────────────────────────────────────
  λ = 0:      no penalty at all — identical to plain OLS
                (Chapter 1), full risk of overfitting

  λ large:       coefficients pushed HEAVILY toward zero —
                   risk of UNDERFITTING if too strong

  λ "just right":   the SWEET SPOT from Module 1, Chapter 4's
                       bias-variance diagram — found via
                       cross-validation (Module 1, Ch.5;
                       formalized further in Module 7)
─────────────────────────────────────────

3. Ridge Regression (L2 Regularization)

Adds a penalty proportional to the sum of squared coefficients.

Ridge's Penalty Term
─────────────────────────────────────────
  Loss = SSE + λ × Σ βᵢ²

  Large coefficients are penalized QUADRATICALLY — a
  coefficient of 10 contributes 100 to the penalty, twice as
  much as two separate coefficients of 5 (25 each, 50 total)
─────────────────────────────────────────

from sklearn.linear_model import Ridge
import numpy as np
 
np.random.seed(0)
n_samples, n_features = 20, 15
X = np.random.randn(n_samples, n_features)
y = X[:, 0] * 3 + np.random.randn(n_samples) * 0.5
 
plain_model = LinearRegression().fit(X, y)
ridge_model = Ridge(alpha=1.0).fit(X, y)      # alpha = λ, scikit-learn's naming
 
print("Plain OLS coefficients:", np.abs(plain_model.coef_).round(2))
print("Ridge coefficients:", np.abs(ridge_model.coef_).round(2))
# Ridge's coefficients are generally SMALLER and more STABLE —
# less influenced by noise in any single feature

Ridge shrinks coefficients toward zero but rarely all the way to exactly zero — every feature typically retains at least some small influence, even if it's genuinely unimportant (Section 5 explains precisely why).

4. Lasso Regression (L1 Regularization)

Adds a penalty proportional to the sum of absolute values of coefficients, rather than their squares.

Lasso's Penalty Term
─────────────────────────────────────────
  Loss = SSE + λ × Σ |βᵢ|

  Large coefficients are penalized LINEARLY, not quadratically
─────────────────────────────────────────

from sklearn.linear_model import Lasso
 
lasso_model = Lasso(alpha=0.5).fit(X, y)
print("Lasso coefficients:", lasso_model.coef_.round(3))
# Notice: MANY coefficients become EXACTLY 0.0 — Lasso performs
# AUTOMATIC FEATURE SELECTION as a side effect (Module 2, Ch.4's
# "embedded methods" preview, now explained fully)
 
selected_features = np.where(lasso_model.coef_ != 0)[0]
print(f"Features Lasso kept: {selected_features}")

5. Why Lasso Zeroes Out Coefficients but Ridge Doesn't

This is a genuinely important, often-asked conceptual question — the answer lies in the geometry of each penalty.

Geometric Intuition
─────────────────────────────────────────
  Ridge's penalty region (Σβᵢ² ≤ constant) forms a SMOOTH,
  ROUNDED shape (a circle/sphere in higher dimensions) — the
  optimal solution touching this boundary rarely lands EXACTLY
  on an axis (where a coefficient would be exactly zero).

  Lasso's penalty region (Σ|βᵢ| ≤ constant) forms a shape with
  SHARP CORNERS (a diamond in 2D) — these corners sit EXACTLY
  on the axes, and the optimal solution is disproportionately
  likely to land precisely on one of these corners, zeroing
  out one or more coefficients entirely.
─────────────────────────────────────────

Ridge (smooth circle)              Lasso (sharp diamond)
─────────────────────────────      ─────────────────────────────
        _____                              /\
      /       \                           /  \
     |    ×    |    ← optimal often      / ×  \    ← optimal often
      \       /       lands OFF-axis     \    /       lands ON a
        ‾‾‾‾‾                             \  /        CORNER (axis)
                                            \/
─────────────────────────────      ─────────────────────────────

Practical Consequence
─────────────────────────────────────────
  Ridge:    good when you believe MOST features contribute
              SOMEWHAT — shrinks everything, keeps everything
  Lasso:       good when you believe only a FEW features truly
                 matter — actively discards the rest, producing
                 a simpler, more interpretable final model
─────────────────────────────────────────

6. Elastic Net — the Best of Both

Combines Ridge's and Lasso's penalties together, controlled by a mixing parameter.

from sklearn.linear_model import ElasticNet
 
elastic_model = ElasticNet(alpha=0.5, l1_ratio=0.5)      # l1_ratio: 0=pure Ridge, 1=pure Lasso
elastic_model.fit(X, y)
print("Elastic Net coefficients:", elastic_model.coef_.round(3))

Elastic Net's Penalty Term
─────────────────────────────────────────
  Loss = SSE + λ × [ l1_ratio × Σ|βᵢ|  +  (1 - l1_ratio) × Σβᵢ² ]

  l1_ratio = 1.0   → pure Lasso
  l1_ratio = 0.0   → pure Ridge
  l1_ratio = 0.5   → an even BLEND of both penalties
─────────────────────────────────────────

When Elastic Net Is the Right Choice
─────────────────────────────────────────
  Particularly useful when features are HIGHLY CORRELATED
  with each other (Module 2, Ch.2's multicollinearity concern)
  — Lasso alone tends to arbitrarily pick JUST ONE feature from
  a correlated GROUP and zero out the rest, while Elastic Net's
  Ridge component tends to keep CORRELATED features together,
  shrinking them as a GROUP rather than making an arbitrary choice.
─────────────────────────────────────────

7. Choosing the Regularization Strength

Neither the choice of Ridge/Lasso/Elastic Net nor their strength (alpha) should be guessed — this is exactly what cross-validation (Module 1, Chapter 5) and Module 7's hyperparameter tuning are built to determine systematically.

from sklearn.linear_model import RidgeCV, LassoCV
 
# RidgeCV / LassoCV automatically try MULTIPLE alpha values and pick
# the best one via cross-validation — a direct preview of Module 7
ridge_cv = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0])
ridge_cv.fit(X, y)
print(f"Best alpha found via CV: {ridge_cv.alpha_}")

8. Summary & Next Steps

Key Takeaways

• Regularization adds a penalty on coefficient size to the loss function, trading a small amount of training fit for better generalization — directly addressing Module 1, Chapter 4's overfitting concern.
• Ridge (L2) penalizes squared coefficients, shrinking everything toward (but rarely exactly to) zero; Lasso (L1) penalizes absolute coefficients, driving some exactly to zero and performing automatic feature selection.
• The geometric difference between Ridge's smooth penalty region and Lasso's sharp-cornered one explains why only Lasso zeroes out coefficients.
• Elastic Net blends both penalties, particularly useful when features are highly correlated with each other.
• Regularization strength (alpha) should be chosen via cross-validation, not guessed — RidgeCV/LassoCV automate this search.

Concept Check

1. Why does regularization sometimes IMPROVE test-set performance despite making training-set fit slightly worse?
2. Why does Lasso tend to zero out coefficients entirely, while Ridge only shrinks them?
3. When would Elastic Net be preferable to pure Lasso?

Next Chapter

→ Chapter 3: Polynomial & Non-Linear Regression

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index