Naive Bayes & Probabilistic Inference

Jr Codex AI Notes

Level: Intermediate Prerequisites: Chapter 2 Time to complete: ~25 minutes

Table of Contents

1. A Simpler, Special-Case Network
2. The Naive Bayes Assumption
3. The Naive Bayes Classifier — Formula
4. Building a Naive Bayes Spam Filter
5. Handling the Zero-Probability Problem
6. Why "Naive" Works Surprisingly Well Anyway
7. Naive Bayes' Place in the Bigger Picture
8. Summary & Next Steps

1. A Simpler, Special-Case Network

Chapter 2's Bayesian networks can represent arbitrarily complex dependency structures. Naive Bayes is a specific, deliberately simplified network structure — one that trades some accuracy for dramatic simplicity and speed, and turns out to be extremely useful in practice, especially for classification.

Naive Bayes as a Bayesian Network Structure
─────────────────────────────────────────
                [Class]
              /    |    \
             ▼     ▼      ▼
       [Feature1][Feature2][Feature3]

  ONE parent node (the class), with ALL features as its
  children — and CRITICALLY, no edges BETWEEN the features
  themselves.
─────────────────────────────────────────

Compare this to Chapter 2's burglary-alarm network, where Alarm had two parents and JohnCalls/MaryCalls both depended on Alarm — Naive Bayes flips the direction: here, everything depends on one shared cause (the class), and features never directly depend on each other.

2. The Naive Bayes Assumption

The "Naive" Assumption
─────────────────────────────────────────
  Given the class, ALL features are CONDITIONALLY INDEPENDENT
  of each other.

  P(Feature1, Feature2, ..., FeatureN | Class)
    = P(Feature1 | Class) × P(Feature2 | Class) × ... × P(FeatureN | Class)
─────────────────────────────────────────

# Example: classifying an email as spam or not, based on word presence
# The NAIVE assumption: whether the email contains "money" and whether it
# contains "click" are treated as INDEPENDENT, once we know it's spam
# (In reality, spam emails using "money" ALSO often use "click" — these
#  words are NOT truly independent, hence "naive")

Why call this assumption "naive"? Because it's almost always technically false in the real world — features usually do correlate with each other beyond just sharing a common cause. Section 6 explains why the classifier still works remarkably well despite this.

3. The Naive Bayes Classifier — Formula

Combining the naive independence assumption (Section 2) with Bayes' theorem (Chapter 1) gives a complete classification rule:

Naive Bayes Classification
─────────────────────────────────────────
  P(Class | Features) ∝ P(Class) × Π P(Featurei | Class)
                              │           │
                              │           └── the NAIVE product,
                              │                 from Section 2
                              └── the PRIOR probability of this class

  Predict whichever CLASS maximizes this product.
  (The "∝" means "proportional to" — we skip dividing by
  P(Features), since it's the SAME for every class we're
  comparing, so it doesn't affect which one is largest.)
─────────────────────────────────────────

def naive_bayes_predict(priors, likelihoods, features, classes):
    """
    priors: dict class -> P(class)
    likelihoods: dict (class, feature) -> P(feature | class)
    features: list of features present in this instance
    """
    scores = {}
    for c in classes:
        score = priors[c]
        for feature in features:
            score *= likelihoods.get((c, feature), 0.5)      # 0.5 = "no information" default
        scores[c] = score
 
    total = sum(scores.values())
    return {c: score / total for c, score in scores.items()}      # normalize into probabilities

4. Building a Naive Bayes Spam Filter

The single most classic real-world Naive Bayes application — directly extending Module 1, Chapter 1's rule-based spam filter example into a genuinely probabilistic, learned one:

from collections import defaultdict
 
class NaiveBayesSpamFilter:
    def __init__(self):
        self.class_counts = defaultdict(int)
        self.word_counts = defaultdict(lambda: defaultdict(int))
        self.vocab = set()
 
    def train(self, emails_with_labels):
        """emails_with_labels: list of (word_list, label) where label is 'spam' or 'ham'"""
        for words, label in emails_with_labels:
            self.class_counts[label] += 1
            for word in words:
                self.word_counts[label][word] += 1
                self.vocab.add(word)
 
    def word_probability(self, word, label, alpha=1):
        """P(word | label), with Laplace smoothing (Section 5)."""
        word_count = self.word_counts[label][word]
        total_words_in_class = sum(self.word_counts[label].values())
        return (word_count + alpha) / (total_words_in_class + alpha * len(self.vocab))
 
    def predict(self, words):
        total_emails = sum(self.class_counts.values())
        scores = {}
 
        for label in self.class_counts:
            prior = self.class_counts[label] / total_emails
            score = prior
            for word in words:
                score *= self.word_probability(word, label)
            scores[label] = score
 
        total = sum(scores.values())
        return {label: score / total for label, score in scores.items()}
 
 
filter = NaiveBayesSpamFilter()
training_data = [
    (["win", "money", "now"], "spam"),
    (["click", "here", "free", "money"], "spam"),
    (["meeting", "tomorrow", "afternoon"], "ham"),
    (["project", "update", "attached"], "ham"),
]
filter.train(training_data)
 
result = filter.predict(["free", "money", "click"])
print(result)      # {'spam': ~0.9x, 'ham': ~0.0x} — correctly flagged as likely spam

Compare this to Module 1, Chapter 1's rule-based spam filter: that version checked for exact hard-coded phrases. This version learns word probabilities from labeled training data — a genuine (if simple) machine learning classifier, built entirely from this module's probability foundations, no separate ML library required.

5. Handling the Zero-Probability Problem

A critical practical issue: if a word never appeared in the training data for a given class, its probability is 0 — and multiplying by 0 wipes out the entire product, regardless of every other word's evidence.

# THE PROBLEM, without smoothing:
# If "lottery" never appeared in "ham" training emails, P("lottery" | ham) = 0
# ANY email containing "lottery" gets classified P(ham) = 0, NO MATTER
# what else it contains — a single unseen word dominates the entire decision
 
# THE FIX — Laplace (add-one) smoothing, already used in Section 4's code:
def word_probability_smoothed(word_count, total_words, vocab_size, alpha=1):
    return (word_count + alpha) / (total_words + alpha * vocab_size)
 
# Even a word with word_count=0 now gets a small, NON-ZERO probability,
# proportional to how large the vocabulary is

Laplace Smoothing, Intuitively
─────────────────────────────────────────
  Instead of trusting the training data's counts LITERALLY
  (which can be zero due to simply not having seen enough
  examples yet), pretend EVERY word was seen alpha=1 extra
  time in every class — a small, principled hedge against
  overconfidence from limited training data.
─────────────────────────────────────────

6. Why "Naive" Works Surprisingly Well Anyway

Given that the independence assumption (Section 2) is almost always technically false, why does Naive Bayes remain a genuinely useful, widely-used classifier?

The Key Insight
─────────────────────────────────────────
  Naive Bayes only needs to get the RELATIVE ORDERING of class
  scores right (which class scores HIGHEST), not the exact
  probability VALUES.

  Even when features are correlated (violating the naive
  assumption), the DIRECTION of evidence each feature provides
  is usually still correct — "money" and "click" BOTH pointing
  toward "spam" still correctly pushes the spam score higher,
  even if the assumption technically double-counts the
  correlated evidence somewhat.
─────────────────────────────────────────

Practical Strengths That Explain Its Continued Use
─────────────────────────────────────────
  - Extremely FAST to train and predict — just counting,
    no iterative optimization needed
  - Works well even with RELATIVELY LITTLE training data
  - A strong, easy-to-implement BASELINE before trying more
    sophisticated models (a common practice covered further
    in the Machine Learning Notes)
  - Still genuinely competitive for TEXT classification tasks
    specifically (spam filtering, sentiment analysis) even
    decades after its introduction
─────────────────────────────────────────

7. Naive Bayes' Place in the Bigger Picture

Where This Fits Across the Curriculum
─────────────────────────────────────────
  THIS chapter:        Naive Bayes as a PROBABILISTIC REASONING
                          technique, built from first principles
                          (Bayes' theorem, conditional independence)

  Machine Learning Notes:  Naive Bayes reappears there as a
                              standard, off-the-shelf CLASSIFICATION
                              algorithm (scikit-learn's
                              MultinomialNB, GaussianNB, etc.) —
                              same underlying math, production-ready
                              tooling

  NLP/LLM Notes:                Naive Bayes was historically a
                                  major technique for text
                                  classification BEFORE deep
                                  learning — useful context for
                                  understanding NLP's own history
─────────────────────────────────────────

This is a good concrete example of Module 1, Chapter 5's point that AI, ML, and their applications overlap constantly — the same technique is "AI" (probabilistic reasoning, this module), "ML" (a classification algorithm), and "NLP" (a text-classification tool), depending on which lens you view it through.

8. Summary & Next Steps

Key Takeaways

• Naive Bayes is a special-case Bayesian network structure: one class node with all features as children, assuming features are conditionally independent given the class.
• The classification rule combines a class's prior probability with the product of each feature's likelihood given that class — predicting whichever class maximizes this product.
• Laplace (add-one) smoothing prevents a single unseen word/feature from zeroing out an entire class's score due to limited training data.
• Despite its independence assumption being almost always technically false, Naive Bayes remains genuinely useful because it usually gets the relative ranking between classes right, even when the exact probabilities are somewhat off.

Concept Check

1. What specific independence assumption does Naive Bayes make, and why is it called "naive"?
2. Why does a single word with zero training occurrences threaten to break the entire classifier without smoothing?
3. Why can Naive Bayes still classify correctly even when its independence assumption is technically violated by correlated features?

Next Chapter

→ Chapter 4: Markov Chains & Hidden Markov Models

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index