Naive Bayes for Classification

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 4, AI Notes, Module 4, Ch.3: Naive Bayes & Probabilistic Inference Time to complete: ~20 minutes

A Different Philosophy: Probabilistic Classification
The Math, Briefly Recapped
Gaussian Naive Bayes — for Continuous Features
Multinomial Naive Bayes — for Text/Count Data
Bernoulli Naive Bayes — for Binary Features
Why Naive Bayes Trains So Fast
Strengths, Weaknesses & When to Use
Summary & Next Steps

1. A Different Philosophy: Probabilistic Classification

Chapters 1-4 covered geometric approaches to classification (boundaries, distances, margins). Naive Bayes takes a fundamentally different, probabilistic route — this is covered in full theoretical depth in the AI Notes, Module 4, Chapter 3, which built the algorithm from Bayes' theorem and the conditional independence assumption. This chapter assumes that foundation and focuses on applying it as a practical scikit-learn classifier.

Quick Recap from the AI Notes
─────────────────────────────────────────
  P(Class | Features) ∝ P(Class) × Π P(Featureᵢ | Class)

  "Naive" = assumes every feature is CONDITIONALLY INDEPENDENT
  of every other feature, GIVEN the class — technically almost
  always false, but a simplification that works remarkably
  well in practice (AI Notes explain exactly why)
─────────────────────────────────────────

2. The Math, Briefly Recapped

# The AI Notes built this from scratch; here's the same idea in scikit-learn
from sklearn.naive_bayes import GaussianNB
import numpy as np
 
X = np.array([[5.1, 3.5], [4.9, 3.0], [6.2, 3.4], [5.9, 3.0]])      # e.g. flower measurements
y = np.array([0, 0, 1, 1])      # two species
 
model = GaussianNB()
model.fit(X, y)
 
probabilities = model.predict_proba([[5.5, 3.2]])
print(probabilities)      # posterior probability for EACH class, summing to 1

If the underlying Bayes' theorem, priors, likelihoods, and conditional independence assumption feel unfamiliar, review the AI Notes chapter above before continuing — this chapter builds directly on that foundation without re-deriving it.

3. Gaussian Naive Bayes — for Continuous Features

Assumes each feature, within each class, follows a normal (Gaussian) distribution — directly reusing the Data Science Notes' probability distributions chapter.

from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=42
)
 
model = GaussianNB()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
 
print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
 
# Inspecting the LEARNED distribution parameters per class
print(f"Learned means per class:\n{model.theta_}")
print(f"Learned variances per class:\n{model.var_}")

What's Actually Learned
─────────────────────────────────────────
  For EACH feature, and EACH class, Gaussian NB learns just
  TWO numbers: the mean and variance of that feature's values
  WITHIN that class — an extremely compact, fast-to-learn model.
─────────────────────────────────────────

4. Multinomial Naive Bayes — for Text/Count Data

The classic choice for text classification — directly extending the AI Notes' spam filter example into scikit-learn's production-ready tooling.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
 
emails = [
    "win money now click here",
    "free prize click now",
    "meeting scheduled for tomorrow",
    "project report attached please review",
]
labels = ["spam", "spam", "ham", "ham"]
 
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(emails)      # converts text into WORD COUNT features
 
model = MultinomialNB()
model.fit(X, labels)
 
new_email = vectorizer.transform(["free money click now"])
prediction = model.predict(new_email)
print(prediction)      # ['spam']

Why "Multinomial" Fits Text Naturally
─────────────────────────────────────────
  Text, once vectorized (CountVectorizer, above), becomes WORD
  COUNTS per document — exactly the kind of DISCRETE COUNT data
  the multinomial distribution (Data Science Notes, Module 1,
  Ch.3) models directly, unlike Gaussian NB's continuous-value
  assumption.
─────────────────────────────────────────

This is the exact same underlying algorithm the AI Notes built from scratch, now applied via scikit-learn's production-quality implementation — the AI Notes' Laplace smoothing (avoiding zero probabilities for unseen words) is built into MultinomialNB automatically via its alpha parameter.

5. Bernoulli Naive Bayes — for Binary Features

A variant specifically for binary (present/absent, true/false) features, rather than counts.

from sklearn.naive_bayes import BernoulliNB
from sklearn.feature_extraction.text import CountVectorizer
 
vectorizer = CountVectorizer(binary=True)      # WORD PRESENCE (0/1), not counts
X_binary = vectorizer.fit_transform(emails)
 
model = BernoulliNB()
model.fit(X_binary, labels)

Multinomial vs Bernoulli — the Distinction
─────────────────────────────────────────
  Multinomial:    "how MANY times does 'money' appear?"
                    (uses the actual COUNT)

  Bernoulli:         "does 'money' appear AT ALL?" (yes/no only,
                        ignores repeat occurrences)

  In practice, Multinomial is more common for text since word
  FREQUENCY often carries genuine signal — but Bernoulli can
  work better for very short texts where presence/absence
  matters more than frequency.
─────────────────────────────────────────

6. Why Naive Bayes Trains So Fast

Directly connecting to Chapter 2's "lazy vs eager" distinction — Naive Bayes is about as "eager" and lightweight as classifiers get.

Why Training Is Nearly Instant
─────────────────────────────────────────
  Training Naive Bayes requires only COUNTING and computing
  simple statistics (means/variances for Gaussian, word
  frequencies for Multinomial) — NO iterative optimization,
  NO gradient descent (unlike Ch.1's logistic regression), NO
  searching for an optimal boundary (unlike Ch.4's SVM).

  This is EXACTLY why Naive Bayes remains a genuinely useful
  BASELINE model — even on large datasets, it trains in a
  fraction of the time other algorithms require.
─────────────────────────────────────────

7. Strengths, Weaknesses & When to Use

Strengths
─────────────────────────────────────────
  - EXTREMELY fast to train and predict
  - Works well even with RELATIVELY LITTLE training data
  - Genuinely strong, still-competitive baseline for TEXT
    classification specifically (spam, sentiment)
  - Naturally handles MULTI-CLASS problems without any
    special modification (unlike Ch.1's need for One-vs-Rest
    or multinomial extensions)
─────────────────────────────────────────

Weaknesses
─────────────────────────────────────────
  - The independence assumption is almost always technically
    FALSE (AI Notes explain why this often doesn't matter much
    in practice, but it CAN hurt performance when features are
    STRONGLY correlated)
  - Gaussian NB assumes features are normally distributed
    WITHIN each class — a poor fit if that assumption is badly
    violated
  - Generally less accurate than Chapters 3-4's algorithms
    (and Module 5's ensembles) on complex, non-text problems
    with intricate feature interactions
─────────────────────────────────────────

Practical guidance: reach for Naive Bayes as a fast first baseline, especially for text classification — if it performs adequately, its speed and simplicity are compelling; if not, its quick training time makes it cheap to rule out before investing in Module 4-5's more sophisticated (and slower-to-train) algorithms.

8. Summary & Next Steps

Key Takeaways

Naive Bayes classifies probabilistically using Bayes' theorem and a conditional independence assumption — fully derived in the AI Notes; this chapter applies it as a practical scikit-learn classifier.
Gaussian NB assumes normally-distributed continuous features per class; Multinomial NB is the standard choice for text/count data; Bernoulli NB handles binary presence/absence features.
Naive Bayes trains almost instantly, since it only requires counting and simple statistics, with no iterative optimization — making it a genuinely useful fast baseline.
Its independence assumption is almost always technically false but often doesn't hurt performance much in practice, particularly for text classification.

Module 4 Complete (Algorithms) — Next: Evaluation

You now have five distinct classification algorithms in your toolkit: logistic regression (linear, probabilistic), KNN (distance-based), decision trees (rule-based), SVM (margin-based), and Naive Bayes (probability-based). Chapter 6 closes this module with the metrics needed to properly compare and evaluate them.

Concept Check

Why does Naive Bayes train so much faster than logistic regression or SVM?
When would Multinomial Naive Bayes be preferred over Bernoulli Naive Bayes for text classification?
Why is Naive Bayes a good "first baseline" to try before more sophisticated algorithms?

Next Chapter

→ Chapter 6: Evaluating Classification Models

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index