Probability Fundamentals

Jr Codex Data Science Notes

Level: Beginner Prerequisites: Chapter 1 Time to complete: ~25 minutes

Table of Contents

1. What is Probability?
2. Sample Spaces & Events
3. The Rules of Probability
4. Independent vs Dependent Events
5. Conditional Probability
6. Bayes' Theorem
7. Simulating Probability with Python
8. Summary & Next Steps

1. What is Probability?

Probability quantifies how likely an event is, on a scale from 0 (impossible) to 1 (certain).

Probability Scale
─────────────────────────────────────────
  0.0 ─────────── 0.5 ─────────── 1.0
  Impossible     Even odds        Certain
  "will never      "coin flip"    "will definitely
   happen"                          happen"
─────────────────────────────────────────

# The basic formula, for equally likely outcomes:
# P(event) = number of favorable outcomes / total number of possible outcomes
 
p_heads = 1 / 2      # flipping a fair coin, landing heads
p_rolling_a_4 = 1 / 6   # rolling a fair six-sided die, landing on 4
print(p_heads, p_rolling_a_4)     # 0.5 0.16666666666666666

2. Sample Spaces & Events

The sample space is the set of all possible outcomes; an event is a specific subset of outcomes you care about.

# Sample space for rolling a die: {1, 2, 3, 4, 5, 6}
sample_space = {1, 2, 3, 4, 5, 6}      # a Python set (Python Notes, Module 1, Ch.9)
 
# Event: rolling an even number
even_event = {2, 4, 6}
 
p_even = len(even_event) / len(sample_space)
print(p_even)      # 0.5

Sample Space vs Event
─────────────────────────────────────────
  Sample Space (all possible rolls): {1, 2, 3, 4, 5, 6}
  Event (rolling even):               {2,    4,    6}
                                         └──── the EVENT is a subset ────┘
─────────────────────────────────────────

3. The Rules of Probability

# Rule 1: probabilities are always between 0 and 1
# Rule 2: the probabilities of ALL outcomes in a sample space sum to 1
die_probs = {1: 1/6, 2: 1/6, 3: 1/6, 4: 1/6, 5: 1/6, 6: 1/6}
print(sum(die_probs.values()))      # 1.0
 
# Rule 3: P(A or B) = P(A) + P(B) - P(A and B)   — the "addition rule"
# For MUTUALLY EXCLUSIVE events (can't both happen), P(A and B) = 0:
p_rolling_1_or_2 = 1/6 + 1/6            # mutually exclusive — can't roll both at once
print(p_rolling_1_or_2)                    # 0.333...
 
# Rule 4: P(not A) = 1 - P(A)   — the "complement rule"
p_not_rolling_6 = 1 - 1/6
print(p_not_rolling_6)      # 0.833...

4. Independent vs Dependent Events

Independent events: the outcome of one doesn't affect the other. Dependent events: it does.

# Independent: two coin flips — the first flip has NO effect on the second
p_two_heads = (1/2) * (1/2)
print(p_two_heads)      # 0.25 — multiply probabilities for independent events
 
# Dependent: drawing cards WITHOUT replacement
# P(first card is an Ace) = 4/52
# P(second card is an Ace | first WAS an Ace) = 3/51  ← changed, because one Ace is now gone!
p_first_ace = 4/52
p_second_ace_given_first = 3/51
p_both_aces = p_first_ace * p_second_ace_given_first
print(p_both_aces)      # ~0.0045

Independent                          Dependent
─────────────────────────────       ─────────────────────────────
  Coin flip #2 doesn't care            Drawing card #2 DOES depend
  what coin flip #1 was                 on what card #1 was
  P(A and B) = P(A) × P(B)              P(A and B) = P(A) × P(B|A)
─────────────────────────────       ─────────────────────────────

5. Conditional Probability

Conditional probability, written P(A | B) ("probability of A given B"), is the probability of an event, given that another event has already occurred.

# Example: a deck of 52 cards. What's P(card is a King | card is a face card)?
# Face cards: Jack, Queen, King × 4 suits = 12 cards
# Kings among face cards: 4
 
p_king_given_face = 4 / 12
print(p_king_given_face)      # 0.333...

P(A | B) = P(A and B) / P(B)
─────────────────────────────────────────
  "Given that we're already in the B world,
   what fraction of THAT world is also A?"
─────────────────────────────────────────

Real-world example, using a small dataset — exactly the kind of query you'll write with pandas filtering (Python Notes, Pandas Primer):

import pandas as pd
 
df = pd.DataFrame({
    "weather": ["rainy", "sunny", "sunny", "rainy", "sunny", "rainy", "sunny"],
    "delayed": [True, False, True, True, False, False, False],
})
 
# P(delayed | rainy) — filter to rainy days FIRST, then check the delay rate WITHIN that group
rainy_days = df[df["weather"] == "rainy"]
p_delayed_given_rainy = rainy_days["delayed"].mean()
print(p_delayed_given_rainy)      # 0.6667 — 2 of 3 rainy days were delayed

6. Bayes' Theorem

Bayes' theorem flips a conditional probability around — extremely useful when you know P(B | A) but actually want P(A | B).

Bayes' Theorem
─────────────────────────────────────────
  P(A | B) = [ P(B | A) × P(A) ] / P(B)
─────────────────────────────────────────

Classic example — medical testing. A disease affects 1% of the population. A test is 90% accurate for people who have the disease (true positive rate) and gives a false positive 5% of the time. If someone tests positive, what's the actual probability they have the disease?

p_disease = 0.01                     # P(A) — prior probability of having the disease
p_positive_given_disease = 0.90        # P(B|A) — test's true positive rate
p_positive_given_no_disease = 0.05       # false positive rate
 
p_no_disease = 1 - p_disease
# P(B) — total probability of testing positive, from EITHER group
p_positive = (p_positive_given_disease * p_disease) + (p_positive_given_no_disease * p_no_disease)
 
p_disease_given_positive = (p_positive_given_disease * p_disease) / p_positive
print(f"{p_disease_given_positive:.2%}")     # 15.38%

This result surprises almost everyone the first time: even with a 90%-accurate test, a positive result only means a 15.38% chance of actually having the disease — because the disease is so rare (1%) that false positives from the other 99% of the population dominate. This exact reasoning pattern — updating a belief given new evidence — underlies spam filters, medical diagnostics, and (much later in this curriculum) Naive Bayes classifiers and Bayesian networks in the AI notes.

7. Simulating Probability with Python

When the math gets complex, simulation (repeating a random experiment many times and counting outcomes) often gives an intuitive sanity check:

import numpy as np
 
np.random.seed(42)     # for reproducible results
 
# Simulate 100,000 coin flips, check what fraction are heads
flips = np.random.choice(["H", "T"], size=100_000)
p_heads_simulated = np.mean(flips == "H")
print(p_heads_simulated)      # ~0.50 — converges to the true probability
 
# Simulate the medical test scenario from Section 6
n_people = 1_000_000
has_disease = np.random.random(n_people) < 0.01                     # 1% have the disease
tests_positive = np.where(
    has_disease,
    np.random.random(n_people) < 0.90,        # 90% true positive rate
    np.random.random(n_people) < 0.05,          # 5% false positive rate
)
 
p_disease_given_positive_sim = has_disease[tests_positive].mean()
print(f"{p_disease_given_positive_sim:.2%}")     # ~15.4% — matches the math above!

This kind of simulation-based validation is a good habit whenever a probability result feels counter-intuitive — running the numbers is a strong sanity check against a math mistake.

8. Summary & Next Steps

Key Takeaways

• Probability ranges from 0 (impossible) to 1 (certain); the sample space is all possible outcomes, an event is a subset of interest.
• Independent events multiply directly (P(A) × P(B)); dependent events require conditioning on what already happened.
• Conditional probability P(A|B) asks "given we're in B's world, what fraction is also A?" — computed as P(A and B) / P(B).
• Bayes' theorem flips a conditional probability around and is essential whenever a rare event (like a disease) makes false positives dominate — a 90%-accurate test can still mean a <20% true-positive rate on a positive result.

Concept Check

1. Why does drawing two cards without replacement make the second draw's probability dependent on the first?
2. What's the difference between P(A and B) and P(A | B)?
3. In the medical test example, why is the actual probability of having the disease so much lower than the test's 90% accuracy might suggest?

Next Chapter

→ Chapter 3: Probability Distributions

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index