Data Science

Statistics And Probability Foundations

Hypothesis Testing Fundamentals

"I see a difference in my SAMPLE — but is there

JrCodex·8 min read

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 5 Time to complete: ~30 minutes

Table of Contents

  1. What is Hypothesis Testing?
  2. Null & Alternative Hypotheses
  3. The p-value
  4. Significance Level & Decision Rule
  5. The One-Sample t-test
  6. The Two-Sample t-test
  7. Type I & Type II Errors
  8. Summary & Next Steps

1. What is Hypothesis Testing?

Hypothesis testing is a formal procedure for deciding whether an observed effect in your data (a difference in means, a correlation, a conversion rate change) is likely a real pattern, or could plausibly have happened just by random chance, given everything Chapter 4 established about sampling variability.

The Core Question
─────────────────────────────────────────
  "I see a difference in my SAMPLE — but is there
   REALLY a difference in the POPULATION, or did I
   just get an unlucky/lucky sample?"
─────────────────────────────────────────

This chapter lays the statistical groundwork; Module 5 applies it directly to A/B testing.

2. Null & Alternative Hypotheses

Every hypothesis test starts by stating two competing claims:

Null Hypothesis (H₀)                  Alternative Hypothesis (H₁)
─────────────────────────────         ─────────────────────────────
  "There is NO effect/difference"       "There IS a real effect/difference"
  The BORING, default assumption          What you suspect might be true
  We assume this is true UNTIL             We only accept this if the
  the data gives strong evidence            data gives strong enough evidence
  against it

# Example: does a new website design increase average time-on-page?
# H0: new_design_avg_time == old_design_avg_time   (no real difference)
# H1: new_design_avg_time != old_design_avg_time     (there IS a difference)

Crucial mindset: a hypothesis test never proves H₀ true — it only tells you whether the data provides strong enough evidence to reject it. "Failing to reject H₀" is not the same as "proving there's no effect" — it might just mean your sample was too small to detect a real, smaller effect (see Type II errors, Section 7).

3. The p-value

The p-value answers a specific, often-misunderstood question:

What a p-value ACTUALLY Means
─────────────────────────────────────────
  "IF the null hypothesis were true, what's the probability
   of seeing a result AT LEAST as extreme as what we observed,
   purely by random chance?"
─────────────────────────────────────────

import numpy as np
from scipy import stats
 
np.random.seed(42)
 
# Simulating: is a coin fair? We flip it 100 times, get 62 heads. Suspicious?
n_flips = 100
observed_heads = 62
 
# Under H0 (fair coin), what's P(62 or MORE extreme heads out of 100 flips)?
result = stats.binomtest(observed_heads, n_flips, p=0.5, alternative="two-sided")
print(result.pvalue)      # ~0.0177

A small p-value (e.g. below 0.05) means: "if the coin really were fair, a result this extreme would be quite rare — so maybe the coin isn't actually fair." It does not mean "there's only a 1.77% chance the coin is fair" — that's a common, incorrect interpretation. The p-value is about the data given H₀, not about the probability of H₀ itself.

Common p-value Misinterpretation — Corrected
─────────────────────────────────────────
  ✗ WRONG: "p=0.03 means there's a 3% chance the null hypothesis is true"
  ✓ RIGHT: "p=0.03 means, IF H0 were true, we'd see data this extreme
             only 3% of the time — that's rare enough to be suspicious"
─────────────────────────────────────────

4. Significance Level & Decision Rule

The significance level (α) is a threshold you choose before looking at the data — the standard convention is α = 0.05.

alpha = 0.05
 
p_value = 0.0177    # from Section 3's coin example
 
if p_value < alpha:
    print("Reject H0 — reasonably strong evidence the coin is NOT fair")
else:
    print("Fail to reject H0 — not enough evidence to say the coin is unfair")
 
# Reject H0 — reasonably strong evidence the coin is NOT fair
Decision Rule
─────────────────────────────────────────
  p-value < α   →  REJECT the null hypothesis (statistically significant result)
  p-value ≥ α    →  FAIL TO REJECT the null hypothesis (not enough evidence)
─────────────────────────────────────────

α = 0.05 means: "I'm willing to accept a 5% chance of wrongly rejecting a TRUE null hypothesis" — this is a deliberate, field-standard trade-off, not a magic number, and Module 5 revisits how this choice interacts with sample size and business risk.

5. The One-Sample t-test

Tests whether a sample's mean differs significantly from some known or hypothesized value.

import numpy as np
from scipy import stats
 
# A factory claims its light bulbs last 1000 hours on average.
# We test a sample of 20 bulbs.
np.random.seed(1)
bulb_lifespans = np.random.normal(loc=970, scale=80, size=20)      # simulated sample
 
t_statistic, p_value = stats.ttest_1samp(bulb_lifespans, popmean=1000)
print(f"t-statistic: {t_statistic:.3f}")
print(f"p-value: {p_value:.3f}")
 
alpha = 0.05
if p_value < alpha:
    print("Reject H0 — bulb lifespan is significantly different from the claimed 1000 hours")
else:
    print("Fail to reject H0 — not enough evidence to dispute the claim")
One-Sample t-test — What It's Comparing
─────────────────────────────────────────
  H0: sample mean == hypothesized value (1000 hours)
  H1: sample mean != hypothesized value

  The t-statistic measures HOW FAR the sample mean is from the
  hypothesized value, in units of standard error (Chapter 4).
─────────────────────────────────────────

6. The Two-Sample t-test

Tests whether two independent groups have significantly different means — the exact statistical engine behind A/B testing (Module 5).

import numpy as np
from scipy import stats
 
np.random.seed(2)
 
# Group A: current website design — time on page, in seconds
group_a = np.random.normal(loc=45, scale=10, size=200)
 
# Group B: new website design — time on page, in seconds
group_b = np.random.normal(loc=48, scale=10, size=200)
 
t_statistic, p_value = stats.ttest_ind(group_a, group_b)
print(f"Group A mean: {np.mean(group_a):.2f}")
print(f"Group B mean: {np.mean(group_b):.2f}")
print(f"p-value: {p_value:.4f}")
 
alpha = 0.05
if p_value < alpha:
    print("Statistically significant difference — the new design likely has a real effect")
else:
    print("No statistically significant difference detected")
Two-Sample t-test — What It's Comparing
─────────────────────────────────────────
  H0: mean(Group A) == mean(Group B)     (no real difference between groups)
  H1: mean(Group A) != mean(Group B)       (there IS a real difference)
─────────────────────────────────────────

This exact pattern — two groups, one metric, a t-test comparing their means — is the statistical core of Module 5's A/B Testing Fundamentals chapter.

7. Type I & Type II Errors

No hypothesis test is ever 100% certain — two distinct kinds of mistakes are possible:

The Confusion Matrix of Hypothesis Testing
─────────────────────────────────────────────────────
                         H0 is ACTUALLY True    H0 is ACTUALLY False
  We REJECT H0            Type I Error ✗          Correct decision ✓
                          ("false positive")        ("true positive")

  We FAIL TO REJECT H0    Correct decision ✓       Type II Error ✗
                          ("true negative")         ("false negative")
─────────────────────────────────────────────────────

# Type I error: concluding the new website design works, when it actually doesn't
#   — controlled by your chosen significance level (alpha).
#   Lowering alpha (e.g. to 0.01) reduces Type I error risk.
 
# Type II error: concluding the new design does NOTHING, when it actually DOES help
#   — reduced primarily by INCREASING sample size (more statistical "power").
 
alpha = 0.05      # 5% chance of a Type I error, by design/convention
Consequence in Practice
Type I Error (false positive)You ship a change that doesn't actually help — wasted effort, possibly a worse product
Type II Error (false negative)You discard a change that actually would have helped — a missed opportunity

There's an inherent trade-off: making α smaller (stricter about Type I errors) makes Type II errors more likely for the same sample size, and vice versa. The only way to reduce both simultaneously is a larger sample — directly connecting back to Chapter 4's standard error, and setting up Module 5's discussion of experiment sample-size planning.

8. Summary & Next Steps

Key Takeaways

  • Hypothesis testing decides whether an observed effect is likely real or just sampling noise, starting from a null hypothesis (H₀: no effect) that's rejected only with sufficiently strong evidence.
  • A p-value is the probability of seeing data this extreme if H₀ were true — it is not the probability that H₀ itself is true, a very common misreading.
  • The decision rule compares the p-value to a pre-chosen significance level α (conventionally 0.05): p < α rejects H₀.
  • Type I errors (false positives) and Type II errors (false negatives) trade off against each other; only a larger sample size reduces both at once.

Module 1 Complete — Next Module

You now have the statistical vocabulary — descriptive stats, probability, distributions, sampling, correlation, and hypothesis testing — that every later module in this curriculum builds on, especially Module 5's A/B testing. Module 2 shifts to the practical side: getting real, messy data into a usable shape.

Concept Check

  1. Why is "p = 0.03" not the same as "there's a 3% chance the null hypothesis is true"?
  2. What's the difference between a Type I error and a Type II error, in plain terms?
  3. If you want to reduce BOTH Type I and Type II error risk at once, what's the one lever that helps with both?

Next Chapter

→ Module 2: Data Acquisition & Wrangling

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index