Statistical Significance & p-values in Practice

Jr Codex Data Science Notes

Level: Intermediate–Advanced Prerequisites: Chapter 1: A/B Testing Fundamentals Time to complete: ~25 minutes

Statistical Significance vs Practical Significance
Effect Size — the Number That's Often More Important Than p
One-Sided vs Two-Sided Tests
The Multiple Comparisons Problem
Bonferroni Correction
Reporting Results Honestly
Summary & Next Steps

1. Statistical Significance vs Practical Significance

Module 1, Chapter 6 defined statistical significance mechanically (p < α). This chapter's first, most important point: statistically significant does not automatically mean "worth acting on."

import numpy as np
from statsmodels.stats.proportion import proportions_ztest
 
# A MASSIVE sample can make a TRIVIAL difference "statistically significant"
np.random.seed(0)
n = 500_000      # a huge sample, e.g. a major tech company's traffic
 
control_conv = np.random.binomial(1, 0.1000, n)
treatment_conv = np.random.binomial(1, 0.1005, n)      # a difference of just 0.05 percentage points!
 
count = np.array([treatment_conv.sum(), control_conv.sum()])
nobs = np.array([n, n])
z_stat, p_value = proportions_ztest(count, nobs)
print(f"p-value: {p_value:.5f}")      # likely well below 0.05 — "statistically significant"!
print(f"Actual difference: {treatment_conv.mean() - control_conv.mean():.5f}")     # only 0.0005 (0.05pp)

Statistical Significance ≠ Practical Significance
─────────────────────────────────────────
  With enough data, almost ANY nonzero difference becomes
  "statistically significant" eventually — statistical significance
  just means "probably not exactly zero," not "big enough to matter."

  The BUSINESS question is always: is this effect size large enough
  to justify the cost of shipping the change?
─────────────────────────────────────────

2. Effect Size — the Number That's Often More Important Than p

Effect size quantifies how large a difference is, independent of sample size — this is what actually determines whether a result matters practically.

import numpy as np
 
control_mean, treatment_mean = 45.2, 48.7
pooled_std = 12.3
 
# Cohen's d — a standardized effect size for comparing two means
cohens_d = (treatment_mean - control_mean) / pooled_std
print(f"Cohen's d: {cohens_d:.3f}")      # 0.285

Cohen's d — Rough Interpretation Guide
─────────────────────────────────────────
  d ≈ 0.2   →  small effect
  d ≈ 0.5    →  medium effect
  d ≈ 0.8     →  large effect
  (These are rules of thumb, not laws — context always matters:
   even a "small" effect can be hugely valuable at massive scale,
   e.g. a major e-commerce platform's checkout flow.)
─────────────────────────────────────────

# For conversion rate differences, RELATIVE lift is often the more intuitive "effect size"
control_rate, treatment_rate = 0.10, 0.1005
relative_lift = (treatment_rate - control_rate) / control_rate * 100
print(f"Relative lift: {relative_lift:.2f}%")      # 0.5% — now the "significant" result from
                                                        # Section 1 looks appropriately unimpressive

Practical habit: always report effect size (absolute difference, relative lift, or Cohen's d) alongside a p-value — a p-value alone tells you "probably not zero," never "how much," and stakeholders almost always care more about "how much" than about statistical formalities.

3. One-Sided vs Two-Sided Tests

Module 1, Chapter 6 used two-sided tests by default. The choice matters and should be made before seeing results.

from scipy import stats
import numpy as np
 
np.random.seed(5)
control = np.random.normal(50, 10, 200)
treatment = np.random.normal(53, 10, 200)
 
# Two-sided: "is there ANY difference, in either direction?"
t_stat, p_two_sided = stats.ttest_ind(treatment, control, alternative="two-sided")
 
# One-sided: "is treatment SPECIFICALLY greater than control?"
t_stat, p_one_sided = stats.ttest_ind(treatment, control, alternative="greater")
 
print(f"Two-sided p-value: {p_two_sided:.4f}")
print(f"One-sided p-value: {p_one_sided:.4f}")      # roughly HALF the two-sided p-value

When Each Is Appropriate
─────────────────────────────────────────
  Two-sided: default choice — you genuinely don't know or don't
              care about DIRECTION in advance ("could go either way")

  One-sided:   ONLY appropriate when a result in the "wrong" direction
                is truly irrelevant to the decision, AND this was
                decided BEFORE seeing the data
─────────────────────────────────────────

Warning: switching to a one-sided test after seeing that a two-sided result narrowly missed significance is a form of p-hacking (Chapter 3) — the choice must be locked in during the experiment design stage, not after the fact.

4. The Multiple Comparisons Problem

Testing many metrics (or many variants) at once inflates the chance that something looks "significant" purely by chance — directly following from Module 1, Chapter 6's Type I error rate.

import numpy as np
from scipy import stats
 
np.random.seed(10)
 
# Simulate 20 metrics that have NO real effect (both groups from the SAME distribution)
n_metrics = 20
significant_count = 0
 
for i in range(n_metrics):
    control = np.random.normal(50, 10, 500)
    treatment = np.random.normal(50, 10, 500)      # SAME mean — no real difference exists!
    _, p_value = stats.ttest_ind(treatment, control)
    if p_value < 0.05:
        significant_count += 1
 
print(f"{significant_count} out of {n_metrics} metrics appeared 'significant' — purely by chance!")
# Often 1-2 out of 20, even though NONE of them have a real effect

Why This Happens
─────────────────────────────────────────
  At α = 0.05, EACH test has a 5% chance of a false positive
  (Type I error, Module 1, Ch.6), even with NO real effect.

  Testing 20 independent metrics: P(at least one false positive)
    = 1 - (0.95)^20 ≈ 64%!

  The more metrics you test, the more likely SOMETHING looks
  "significant" purely by chance.
─────────────────────────────────────────

This is exactly why "primary metric, chosen in advance" (Chapter 1) matters so much — testing dozens of secondary metrics and reporting whichever one came back significant is a direct path to false conclusions.

5. Bonferroni Correction

The simplest fix for the multiple comparisons problem: make the significance threshold stricter, proportional to how many tests you're running.

alpha = 0.05
n_tests = 20
 
bonferroni_alpha = alpha / n_tests
print(f"Adjusted significance threshold: {bonferroni_alpha:.4f}")      # 0.0025, instead of 0.05

import numpy as np
from scipy import stats
 
np.random.seed(10)
n_metrics = 20
alpha = 0.05
bonferroni_alpha = alpha / n_metrics
 
significant_count = 0
for i in range(n_metrics):
    control = np.random.normal(50, 10, 500)
    treatment = np.random.normal(50, 10, 500)
    _, p_value = stats.ttest_ind(treatment, control)
    if p_value < bonferroni_alpha:      # much stricter threshold now
        significant_count += 1
 
print(f"{significant_count} out of {n_metrics} metrics significant after Bonferroni correction")
# Typically 0 — correctly reflecting that none of these metrics have a REAL effect

Bonferroni Correction — the Trade-off
─────────────────────────────────────────
  Pro:  directly controls the overall false-positive rate across
         ALL tests, not just each one individually
  Con:   makes each individual test more CONSERVATIVE — increases
          the risk of Type II errors (missing REAL effects),
          especially with many tests
─────────────────────────────────────────

Practical recommendation: the best fix is prevention — define one primary metric in advance (Chapter 1) so this correction is rarely needed at all. Reserve Bonferroni (or related methods) for situations where testing several metrics is genuinely unavoidable.

6. Reporting Results Honestly

Putting this chapter's ideas together into what a defensible A/B test report actually looks like:

Weak Report (statistically naive)
─────────────────────────────────────────
  "The new checkout flow was significant (p=0.03), so we should ship it."
─────────────────────────────────────────

Strong Report (statistically honest)
─────────────────────────────────────────
  "The new checkout flow increased conversion from 10.0% to 10.4%
   (a 4% relative lift), 95% CI [0.1%, 0.7%], p=0.03. This was our
   single pre-registered primary metric. Guardrail metrics (refund
   rate, average order value) showed no significant change.

   While statistically significant, the absolute lift is modest —
   recommend a follow-up test on higher-traffic segments to confirm
   the effect size justifies the engineering cost of the change."
─────────────────────────────────────────

The strong version includes: the actual effect size (not just "significant"), a confidence interval, confirmation that this was the pre-registered primary metric (not one of many tried), guardrail metric results, and an honest practical-significance judgment — exactly the ingredients this chapter has built up.

7. Summary & Next Steps

Key Takeaways

Statistical significance only means "probably not exactly zero" — it says nothing about whether an effect is large enough to matter practically; always report effect size (absolute difference, relative lift, or Cohen's d) alongside any p-value.
One-sided vs two-sided tests must be chosen before seeing results — switching after the fact based on what looks favorable is a form of p-hacking.
Testing many metrics inflates the true false-positive rate well beyond your nominal α — with 20 metrics tested at α=0.05, there's roughly a 64% chance something looks "significant" by pure chance.
Bonferroni correction (dividing α by the number of tests) directly addresses this, at the cost of reduced power — prevention (one pre-registered primary metric) is the better long-term fix.

Concept Check

Why can a massive sample size make a practically meaningless difference "statistically significant"?
Why is choosing a one-sided test after seeing a near-significant two-sided result considered a statistical foul?
If you test 10 independent metrics at α=0.05 with no real effects present, roughly what's the chance at least one looks "significant" by chance?

Next Chapter

→ Chapter 3: Common Experiment Design Pitfalls

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index