Machine Learning

Supervised Learning Classification

Logistic Regression

A natural first instinct: why not just use Module 3's linear regression directly for a yes/no problem, treating "yes"=1 and "no"=0? This breaks down quickly.

JrCodex·8 min read

Jr Codex ML Notes

Level: Intermediate Prerequisites: Module 3: Supervised Learning — Regression Time to complete: ~30 minutes

Table of Contents

  1. Why Linear Regression Doesn't Work for Classification
  2. The Sigmoid Function
  3. The Logistic Regression Model
  4. From Probability to Class Prediction
  5. The Decision Boundary
  6. How Logistic Regression Is Trained
  7. Multi-Class Classification
  8. Interpreting Coefficients as Odds
  9. Summary & Next Steps

1. Why Linear Regression Doesn't Work for Classification

A natural first instinct: why not just use Module 3's linear regression directly for a yes/no problem, treating "yes"=1 and "no"=0? This breaks down quickly.

import numpy as np
from sklearn.linear_model import LinearRegression
 
# Trying to force linear regression onto a binary problem
hours_studied = np.array([[1], [2], [3], [8], [9], [10]])
passed_exam = np.array([0, 0, 0, 1, 1, 1])
 
model = LinearRegression().fit(hours_studied, passed_exam)
print(model.predict([[15]]))      # could easily predict something like 1.4 — NOT a valid probability!
print(model.predict([[-5]]))       # could predict something NEGATIVE — meaningless as a probability
The Core Problem
─────────────────────────────────────────
  Linear regression's output is UNBOUNDED — it can predict any
  number from -∞ to +∞. But a class probability MUST be between
  0 and 1. Linear regression has NO mechanism to enforce this,
  and will happily predict nonsensical values like 1.4 or -0.2
  "probability."
─────────────────────────────────────────

This is precisely why classification needs its own dedicated model — not because the underlying idea (weighted sum of features) is wrong, but because the output needs to be squeezed into a valid probability range.

2. The Sigmoid Function

The sigmoid function (also called the logistic function) takes any real number and squeezes it into the range (0, 1) — exactly what's needed to fix Section 1's problem.

import numpy as np
import matplotlib.pyplot as plt
 
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
 
z_values = np.linspace(-10, 10, 100)
sigmoid_values = sigmoid(z_values)
 
plt.plot(z_values, sigmoid_values)
plt.axhline(y=0.5, color="gray", linestyle="--")
plt.axvline(x=0, color="gray", linestyle="--")
plt.xlabel("z (linear combination of features)")
plt.ylabel("sigmoid(z) — always between 0 and 1")
plt.title("The Sigmoid Function")
 
print(sigmoid(0))       # 0.5 — exactly halfway
print(sigmoid(10))        # ~0.99995 — very close to 1
print(sigmoid(-10))         # ~0.00005 — very close to 0
Sigmoid's Shape
─────────────────────────────────────────
  1.0 ┤                    ______------
      │                ___/
  0.5 ┤            ___/
      │        ___/
  0.0 ┤___----
      └────────────────────────────────
        -10    -5     0     5     10
                      z

  As z → +∞, sigmoid(z) → 1
  As z → -∞, sigmoid(z) → 0
  At z = 0, sigmoid(z) = 0.5 exactly
─────────────────────────────────────────

3. The Logistic Regression Model

Combines Module 3's familiar linear combination of features with the sigmoid function wrapped around it.

The Model
─────────────────────────────────────────
  z = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ       (exactly Module 3's linear combination)

  P(y=1) = sigmoid(z) = 1 / (1 + e^(-z))    (squeezed into a valid probability)
─────────────────────────────────────────

from sklearn.linear_model import LogisticRegression
import numpy as np
 
hours_studied = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]])
passed_exam = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])
 
model = LogisticRegression()
model.fit(hours_studied, passed_exam)
 
# predict_proba returns the ACTUAL probability, not just a class
probabilities = model.predict_proba([[3], [6], [9]])
print(probabilities)
# [[0.85, 0.15],   ← 85% chance of "not passed", 15% chance of "passed"
#  [0.35, 0.65],   ← 35% / 65%
#  [0.05, 0.95]]    ← 5% / 95%

4. From Probability to Class Prediction

Once a probability is computed, it must be converted into a concrete class prediction using a threshold — by default, 0.5.

predictions = model.predict([[3], [6], [9]])
print(predictions)      # [0, 1, 1] — thresholded at 0.5 by default
 
# The threshold can be adjusted MANUALLY for different use cases
probabilities_class1 = model.predict_proba(hours_studied)[:, 1]
custom_predictions = (probabilities_class1 >= 0.3).astype(int)      # a LOWER threshold — more lenient
Why the Threshold Matters (Preview of Chapter 6)
─────────────────────────────────────────
  A LOWER threshold (e.g. 0.3) predicts the positive class MORE
  often — catching more true positives, but also more false
  positives.

  A HIGHER threshold (e.g. 0.7) predicts the positive class LESS
  often — fewer false positives, but also missing more true
  positives.

  This exact trade-off (precision vs recall) is covered fully
  in Chapter 6, and the threshold is a genuine, business-relevant
  choice, not always the default 0.5.
─────────────────────────────────────────

5. The Decision Boundary

The decision boundary is the line (or surface, with more features) where the model's predicted probability equals exactly 0.5 — everything on one side is classified as one class, everything on the other side as the other.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
 
np.random.seed(0)
class_0 = np.random.randn(50, 2) + [2, 2]
class_1 = np.random.randn(50, 2) + [5, 5]
X = np.vstack([class_0, class_1])
y = np.array([0]*50 + [1]*50)
 
model = LogisticRegression().fit(X, y)
 
xx, yy = np.meshgrid(np.linspace(0, 8, 100), np.linspace(0, 8, 100))
Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
 
plt.contourf(xx, yy, Z, alpha=0.3)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Logistic Regression's Decision Boundary")
A Key Limitation Worth Knowing Upfront
─────────────────────────────────────────
  Logistic regression's decision boundary is ALWAYS a straight
  line (or a flat plane/hyperplane, with more features) — it
  CANNOT naturally represent a curved or complex boundary
  between classes.

  When classes aren't LINEARLY separable, logistic regression
  will underfit (Module 1, Ch.4) — this is exactly why Chapters
  2-4 introduce KNN, decision trees, and SVMs, each capable of
  more flexible decision boundaries.
─────────────────────────────────────────

6. How Logistic Regression Is Trained

Unlike Module 3's linear regression, logistic regression has no closed-form solution — it's trained using gradient descent (Module 3, Chapter 1, Section 4), minimizing a different loss function suited to probabilities.

Log Loss (Binary Cross-Entropy) — the Loss Function Being Minimized
─────────────────────────────────────────
  Log Loss = -(1/n) × Σ [ y×log(p) + (1-y)×log(1-p) ]

  where p = predicted probability of the positive class

  Intuition: HEAVILY penalizes CONFIDENT, WRONG predictions
  (e.g. predicting 99% probability of "yes" when the true
  answer is "no") — far more harshly than a similarly-wrong
  but LESS confident prediction.
─────────────────────────────────────────

def log_loss(y_actual, y_predicted_proba):
    epsilon = 1e-15      # avoid log(0), which is undefined
    y_predicted_proba = np.clip(y_predicted_proba, epsilon, 1 - epsilon)
    return -np.mean(
        y_actual * np.log(y_predicted_proba) + (1 - y_actual) * np.log(1 - y_predicted_proba)
    )
 
# A confident WRONG prediction is penalized MUCH more than an unconfident one
print(log_loss(np.array([1]), np.array([0.01])))      # HUGE loss — very confident, very wrong
print(log_loss(np.array([1]), np.array([0.4])))          # smaller loss — wrong, but less confidently so

Gradient descent (introduced in Module 3, Chapter 1) iteratively adjusts the coefficients to minimize this log loss — the exact same "step downhill" process, just applied to a different loss surface.

7. Multi-Class Classification

Logistic regression naturally extends beyond binary (yes/no) to multiple classes using either the One-vs-Rest strategy or a direct multinomial extension (softmax regression).

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
 
iris = load_iris()
X, y = iris.data, iris.target      # 3 classes: setosa, versicolor, virginica
 
model = LogisticRegression(multi_class="multinomial", max_iter=200)
model.fit(X, y)
 
print(model.predict_proba(X[:1]))      # probability for EACH of the 3 classes, summing to 1
print(model.classes_)                     # [0, 1, 2]
One-vs-Rest vs Multinomial
─────────────────────────────────────────
  One-vs-Rest:      trains ONE binary classifier PER class
                       ("is it class A, or NOT class A?"),
                       repeated for every class, then picks
                       whichever classifier is most confident

  Multinomial (softmax):  a single, unified model directly
                             outputting a probability for EACH
                             class simultaneously, guaranteed to
                             sum to 1 — generally the more
                             principled, modern default
─────────────────────────────────────────

8. Interpreting Coefficients as Odds

Logistic regression retains much of Module 3's coefficient interpretability, but through odds rather than a direct additive effect.

import numpy as np
 
model = LogisticRegression().fit(hours_studied, passed_exam)
coefficient = model.coef_[0][0]
 
odds_ratio = np.exp(coefficient)
print(f"Coefficient: {coefficient:.3f}")
print(f"Odds ratio: {odds_ratio:.3f}")
# e.g. odds_ratio = 2.1 means: each ADDITIONAL hour studied
# roughly DOUBLES the ODDS of passing (not the raw probability —
# odds and probability are related but distinct concepts)
Why "Odds," Not Direct Probability
─────────────────────────────────────────
  odds = P(event) / (1 - P(event))

  Logistic regression's coefficients are linear in the LOG-ODDS
  (this is literally what "logistic" refers to), not in the
  probability itself — exponentiating a coefficient gives the
  MULTIPLICATIVE effect on ODDS, a genuinely useful, if less
  immediately intuitive, interpretation than Module 3's direct
  additive coefficients.
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

  • Linear regression can't be used directly for classification because its output is unbounded, while probabilities must fall between 0 and 1 — the sigmoid function fixes this.
  • Logistic regression combines a linear combination of features (like Module 3) with the sigmoid function, producing valid probabilities, then a threshold (default 0.5) converts probability into a class prediction.
  • The decision boundary is always linear — a real limitation when classes aren't linearly separable, motivating Chapters 2-4's more flexible algorithms.
  • Logistic regression is trained via gradient descent minimizing log loss, which heavily penalizes confident-but-wrong predictions.
  • Coefficients are interpreted through odds ratios (exponentiated coefficients), reflecting the model's "linear in log-odds" structure.

Concept Check

  1. Why can't linear regression's raw output be used directly as a class probability?
  2. What's a key limitation of logistic regression's decision boundary, and which later chapters address it?
  3. Why does log loss penalize a confident wrong prediction more harshly than an unconfident wrong one?

Next Chapter

→ Chapter 2: K-Nearest Neighbors

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index