Machine Learning

Supervised Learning Regression

Polynomial & Non-Linear Regression

Chapter 1's residual plot diagnostic can reveal a curved pattern — direct evidence that the true relationship isn't linear (Chapter 1, Section 7's assumption).

JrCodex·6 min read

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 2 Time to complete: ~20 minutes

Table of Contents

  1. When a Straight Line Isn't Enough
  2. Polynomial Regression
  3. Choosing the Right Degree
  4. Polynomial Regression Is Still "Linear"
  5. Other Non-Linear Transformations
  6. When to Reach for a Fundamentally Different Algorithm
  7. Summary & Next Steps

1. When a Straight Line Isn't Enough

Chapter 1's residual plot diagnostic can reveal a curved pattern — direct evidence that the true relationship isn't linear (Chapter 1, Section 7's assumption). This chapter covers the most direct fix: letting the model curve.

import numpy as np
import matplotlib.pyplot as plt
 
np.random.seed(0)
X = np.linspace(-3, 3, 50).reshape(-1, 1)
y = X.ravel() ** 2 + np.random.normal(0, 1, 50)      # a genuinely QUADRATIC relationship
 
from sklearn.linear_model import LinearRegression
plain_model = LinearRegression().fit(X, y)
predictions = plain_model.predict(X)
 
plt.scatter(X, y, alpha=0.5)
plt.plot(X, predictions, color="red")
plt.title("A straight line CANNOT capture this curve — systematic underfitting")

2. Polynomial Regression

Directly reuses Module 2, Chapter 4's PolynomialFeatures — generating power terms of the original features, then fitting ordinary linear regression on top of them.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
 
poly_model = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    LinearRegression()
)
poly_model.fit(X, y)
 
predictions_poly = poly_model.predict(X)
 
plt.scatter(X, y, alpha=0.5)
plt.plot(np.sort(X, axis=0), poly_model.predict(np.sort(X, axis=0)), color="green")
plt.title("Degree-2 polynomial regression — correctly captures the curve")
What's Actually Happening
─────────────────────────────────────────
  Original model:      y = β₀ + β₁x                (a straight line)
  Polynomial model:       y = β₀ + β₁x + β₂x²          (a parabola)

  The model is fitting a straight line — but in a TRANSFORMED
  feature space that now includes x² as its own input.
─────────────────────────────────────────

3. Choosing the Right Degree

The polynomial degree is itself a hyperparameter (Module 7 covers this systematically) — too low underfits, too high overfits, directly reprising Module 1, Chapter 4's central tension.

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
 
for degree in [1, 2, 4, 10, 15]:
    model = make_pipeline(PolynomialFeatures(degree=degree, include_bias=False), LinearRegression())
    model.fit(X_train, y_train)
 
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse = mean_squared_error(y_test, model.predict(X_test))
    print(f"Degree {degree:2d}: train MSE={train_mse:8.2f}, test MSE={test_mse:8.2f}")
The Expected Pattern
─────────────────────────────────────────
  Degree 1:    HIGH train AND test error (underfitting —
                 can't capture the curve at all)
  Degree 2:      LOW train AND test error (just right — matches
                   the TRUE quadratic relationship)
  Degree 10+:       VERY LOW train error, but test error starts
                       CLIMBING again (overfitting — fitting
                       noise, Module 1 Ch.4's classic signature)
─────────────────────────────────────────

This experiment is a direct, hands-on demonstration of Module 1, Chapter 4's bias-variance curve — plotting train and test error against increasing polynomial degree traces out exactly that chapter's U-shaped total-error diagram.

4. Polynomial Regression Is Still "Linear"

A frequently confusing but important point: "linear regression" refers to the model being linear in its coefficients, not linear in its relationship to the original input.

Why "Polynomial Regression" Is Still a LINEAR Model
─────────────────────────────────────────
  y = β₀ + β₁x + β₂x²

  This is NOT linear in x (it curves) — but it IS linear in
  β₀, β₁, β₂ (each coefficient still just multiplies a fixed
  term and adds up). This is EXACTLY what "linear" means in
  "linear regression" — the model is a linear COMBINATION of
  (possibly transformed) features.

  This is why OLS (Chapter 1) and regularization (Chapter 2)
  apply UNCHANGED to polynomial regression — nothing about the
  underlying fitting method needed to change at all.
─────────────────────────────────────────

This distinction matters because it clarifies exactly what makes an algorithm "genuinely" non-linear (Section 6) — one where the relationship between coefficients and predictions itself curves, not just the relationship between features and target.

5. Other Non-Linear Transformations

Beyond polynomial terms, other transformations can linearize different kinds of relationships, letting the same linear regression machinery handle them.

import numpy as np
 
# Logarithmic relationships (common for phenomena that grow quickly then level off)
X_log_transformed = np.log(X_positive_only)      # requires X > 0
 
# Exponential relationships — transform the TARGET instead
y_log_transformed = np.log(y_positive_only)      # models y = e^(β₀ + β₁x) as log(y) = β₀ + β₁x
 
# Reciprocal relationships
X_reciprocal = 1 / X_nonzero
Choosing a Transformation — Guided by EDA
─────────────────────────────────────────
  This is exactly where the Data Science Notes' EDA workflow
  (Module 3, Ch.2's shape recognition) becomes directly useful
  for MODELING, not just understanding — plotting y against x
  and recognizing "this looks logarithmic" or "this looks like
  a power relationship" guides which transformation to try.
─────────────────────────────────────────

6. When to Reach for a Fundamentally Different Algorithm

Polynomial and transformed regression have real limits — sometimes the relationship is too complex, or involves interactions too intricate, for hand-picked transformations to capture reasonably.

Decision Guide
─────────────────────────────────────────
  Relationship is a KNOWN, SIMPLE curve shape (quadratic,
  logarithmic, exponential) — visible from EDA
      → Polynomial/transformed regression (this chapter) —
        keeps the interpretability advantage from Chapter 1

  Relationship is COMPLEX, involves many interacting features
  in ways that resist simple transformation, or you don't know
  the shape in advance
      → Move to Module 4's decision trees or Module 5's
        ensemble methods — these capture non-linearity and
        interactions AUTOMATICALLY, at some cost to
        interpretability (a direct trade-off, echoing the AI
        Notes' Explainable AI discussion)
─────────────────────────────────────────

A high polynomial degree is rarely the right way to capture genuinely complex, high-dimensional non-linearity — beyond a modest degree, the combinatorial explosion of interaction terms (Module 2, Chapter 4) and overfitting risk (Section 3) make tree-based methods (Module 4 onward) a much more practical choice for seriously non-linear problems.

7. Summary & Next Steps

Key Takeaways

  • Polynomial regression fits curves by adding power terms (x², x³, ...) as new features, then applying ordinary linear regression on top of them.
  • The polynomial degree is a hyperparameter with the same bias-variance tradeoff as any model complexity choice — too low underfits, too high overfits, visible directly by comparing train and test error across degrees.
  • Polynomial regression is still technically "linear" — linear in its coefficients, not in its relationship to the original feature — which is why OLS and regularization apply unchanged.
  • Logarithmic, exponential, and reciprocal transformations can linearize other common relationship shapes, guided by EDA's shape-recognition skills.
  • For genuinely complex, high-dimensional non-linearity, tree-based algorithms (Module 4 onward) are usually a more practical choice than an ever-higher polynomial degree.

Module 3 Complete (through Regression) — Next: Evaluation

You now understand the core regression algorithm family: linear regression, regularization, and handling non-linear relationships. Chapter 4 closes this module with the metrics needed to properly evaluate any regression model you build.

Concept Check

  1. Why is polynomial regression still considered a "linear" model despite fitting curves?
  2. What pattern would you expect to see plotting train/test error against increasing polynomial degree, and why?
  3. When would you reach for a tree-based algorithm (Module 4) instead of a higher polynomial degree?

Next Chapter

→ Chapter 4: Evaluating Regression Models

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index