Probability for AI

Jr Codex AI Notes

Level: Intermediate Prerequisites: Module 3: Knowledge, Reasoning & Planning Time to complete: ~20 minutes

Why Logic Alone Isn't Enough
A Quick Probability Refresher
Conditional Probability, Revisited for AI
Bayes' Theorem — the Engine of This Module
Random Variables in AI
The Joint Probability Distribution
Why the Joint Distribution Doesn't Scale — Setting Up Chapter 2
Summary & Next Steps

1. Why Logic Alone Isn't Enough

Module 3's propositional and first-order logic dealt in absolutes: a sentence is true or false, an entailment holds or it doesn't. Real-world knowledge is rarely this clean:

Where Pure Logic Breaks Down
─────────────────────────────────────────
  "If a patient has a fever, they have the flu"
   — FALSE as a strict logical rule (many things cause fever)
   — but genuinely USEFUL as a probabilistic statement:
     "a fever makes flu MORE LIKELY"

  Module 3's expert systems (Ch.6) patched this with ad-hoc
  "certainty factors" — this module replaces that patch with
  actual, mathematically grounded PROBABILITY THEORY.
─────────────────────────────────────────

This chapter is a fast, AI-focused refresher — for deeper statistical foundations (distributions, sampling, hypothesis testing), see the Data Science Notes, Module 1, which this chapter deliberately doesn't repeat.

2. A Quick Probability Refresher

# P(event) = a number between 0 (impossible) and 1 (certain)
 
p_rain_tomorrow = 0.3           # 30% chance of rain
p_no_rain_tomorrow = 1 - p_rain_tomorrow      # complement rule: 0.7
 
# Independent events multiply
p_rain_and_traffic = 0.3 * 0.6      # if traffic is independent of rain (unrealistic, but illustrative)

Rule	Formula
Complement	`P(¬A) = 1 - P(A)`
Addition (mutually exclusive)	`P(A ∨ B) = P(A) + P(B)`
Independence	`P(A ∧ B) = P(A) × P(B)`

(Full derivations and worked examples are in the Data Science Notes — this chapter assumes familiarity and moves straight to AI-specific applications.)

3. Conditional Probability, Revisited for AI

P(A | B) = P(A ∧ B) / P(B)
─────────────────────────────────────────
  "Given that we KNOW B is true, what's the probability of A?"
─────────────────────────────────────────

# P(Flu | Fever) — given a patient has a fever, what's the probability they have flu?
p_fever_and_flu = 0.04       # 4% of ALL patients have both fever AND flu
p_fever = 0.20                  # 20% of ALL patients have a fever (many causes)
 
p_flu_given_fever = p_fever_and_flu / p_fever
print(p_flu_given_fever)      # 0.20 — 20% of feverish patients actually have flu

This is the single most important operation in this entire module — nearly every technique in Chapters 2-5 is fundamentally about computing or approximating a conditional probability like this one, just for much larger, more structured sets of variables.

4. Bayes' Theorem — the Engine of This Module

Bayes' Theorem
─────────────────────────────────────────
  P(A | B) = [ P(B | A) × P(A) ] / P(B)
─────────────────────────────────────────

# Medical diagnosis, framed the way AI systems actually use Bayes' theorem:
# we know P(symptom | disease) from medical studies, but we WANT P(disease | symptom)
 
p_fever_given_flu = 0.90        # 90% of flu patients have a fever (known from data)
p_flu_prior = 0.05                 # 5% of the general population has flu (base rate)
p_fever_overall = 0.20                # 20% of the population has a fever, for ANY reason
 
p_flu_given_fever = (p_fever_given_flu * p_flu_prior) / p_fever_overall
print(f"{p_flu_given_fever:.2%}")      # 22.5% — updated belief, given the fever evidence

Bayesian Terminology — Worth Knowing Precisely
─────────────────────────────────────────
  Prior:        P(A) — belief BEFORE seeing evidence (P(flu) = 5%)
  Likelihood:      P(B|A) — how likely the EVIDENCE is, if A is true
                     (P(fever|flu) = 90%)
  Evidence:           P(B) — overall probability of the evidence
                        (P(fever) = 20%, from ANY cause)
  Posterior:            P(A|B) — UPDATED belief AFTER seeing evidence
                          (P(flu|fever) = 22.5%)
─────────────────────────────────────────

This prior → posterior update, given new evidence, is the conceptual heartbeat of this entire module — Chapter 2's Bayesian networks are, at their core, a scalable way to chain many of these updates together across many interconnected variables.

5. Random Variables in AI

A random variable represents an uncertain quantity — in AI, these are typically the same predicates from Module 3, Chapter 3, but now allowed to take a probability rather than a strict true/false value.

# A random variable's DOMAIN — the values it can take
Weather = ["sunny", "rainy", "cloudy"]      # a discrete random variable
 
# A probability distribution over that domain — MUST sum to 1
weather_distribution = {"sunny": 0.6, "rainy": 0.25, "cloudy": 0.15}
print(sum(weather_distribution.values()))      # 1.0 — a valid distribution
 
# Boolean random variables (most common in AI examples) — just two outcomes
Fever = {"true": 0.20, "false": 0.80}

Module 3's Predicates vs This Module's Random Variables
─────────────────────────────────────────
  Module 3:    Fever(patient1) is EITHER true or false — no
                 middle ground, no degree of belief

  This module:    Fever is a RANDOM VARIABLE — we can say
                    "P(Fever=true) = 0.20" without ever needing
                    to resolve whether it's ACTUALLY true for
                    any specific patient until evidence arrives
─────────────────────────────────────────

6. The Joint Probability Distribution

The joint distribution specifies the probability of every possible combination of values across multiple random variables — in principle, it contains everything you could ever want to know.

# A joint distribution over TWO boolean variables: Fever and Flu
# Every row must be specified, and all rows must sum to 1
 
joint_distribution = {
    ("fever", "flu"):        0.036,
    ("fever", "no_flu"):       0.164,
    ("no_fever", "flu"):          0.014,
    ("no_fever", "no_flu"):          0.786,
}
print(sum(joint_distribution.values()))      # 1.0
 
# ANY probability question can be answered by summing the relevant rows —
# this is called "marginalization"
p_flu = joint_distribution[("fever", "flu")] + joint_distribution[("no_fever", "flu")]
print(p_flu)      # 0.05 — matches the prior from Section 4
 
p_fever_given_flu = joint_distribution[("fever", "flu")] / p_flu
print(p_fever_given_flu)      # 0.72

Marginalization — Summing Out a Variable
─────────────────────────────────────────
  To find P(Flu) alone, SUM every joint entry where Flu holds,
  regardless of Fever's value — "marginalizing out" Fever.

  This works for ANY question, given the FULL joint distribution.
─────────────────────────────────────────

7. Why the Joint Distribution Doesn't Scale — Setting Up Chapter 2

# With just 2 boolean variables: 2^2 = 4 entries needed (manageable)
# With 10 boolean variables: 2^10 = 1,024 entries needed
# With 30 boolean variables: 2^30 = over ONE BILLION entries needed!
 
n_variables = 30
n_entries_needed = 2 ** n_variables
print(f"{n_entries_needed:,} entries needed")      # 1,073,741,824

The Core Problem This Sets Up
─────────────────────────────────────────
  The full joint distribution answers EVERY possible question —
  but its SIZE grows EXPONENTIALLY with the number of variables,
  exactly like Chapter 2's "2^n possible worlds" problem for
  propositional logic model checking (Module 3, Chapter 2).

  Real AI applications (medical diagnosis, spam filtering,
  sensor fusion) easily involve dozens or hundreds of variables
  — storing the FULL joint distribution is completely infeasible.

  Chapter 2's Bayesian Networks solve this EXACT problem, using
  the same core idea Module 3 used for logic: exploit STRUCTURE
  (most variables aren't directly related to most others) to
  avoid needing the full, exponentially large representation.
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

Probability extends Module 3's strict true/false logic to handle genuine uncertainty — a fever doesn't logically guarantee flu, but it does make it more probable.
Conditional probability (P(A|B)) and Bayes' theorem (updating a prior belief into a posterior, given evidence) are the core operations this entire module builds on.
A random variable generalizes Module 3's predicates to take on probabilities across a domain of possible values, rather than a strict true/false.
The full joint probability distribution can answer any question via marginalization, but its size grows as 2^n for n boolean variables — completely infeasible beyond a handful of variables, exactly the problem Chapter 2's Bayesian networks are built to solve.

Concept Check

Why is "fever implies flu" a poor logical rule but a genuinely useful probabilistic one?
In Bayes' theorem, what's the difference between a prior and a posterior probability?
Why does storing the full joint probability distribution become infeasible as the number of variables grows?

Next Chapter

→ Chapter 2: Bayesian Networks

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index