Common Experiment Design Pitfalls

Jr Codex Data Science Notes

Level: Advanced Prerequisites: Chapter 2: Statistical Significance & p-values in Practice Time to complete: ~25 minutes

Table of Contents

1. Why Experienced Practitioners Still Get This Wrong
2. Peeking — Stopping Early When Results Look Good
3. The Novelty Effect
4. Sample Ratio Mismatch (SRM)
5. Network Effects & Interference
6. Survivorship Bias in Experiment Analysis
7. A Pre-Launch Checklist
8. Summary & Next Steps

1. Why Experienced Practitioners Still Get This Wrong

Chapters 1-2 covered the statistical mechanics correctly. This chapter covers the practical mistakes that undermine an otherwise well-designed test — these are the failure modes that catch even experienced data scientists, precisely because they don't show up as an obvious math error.

2. Peeking — Stopping Early When Results Look Good

Peeking means checking a test's results before the pre-calculated sample size (Chapter 1) is reached, and stopping the moment it looks significant — one of the single most damaging, common mistakes in practice.

import numpy as np
from scipy import stats
 
np.random.seed(3)
 
# Simulating a test with NO real effect, checking results EVERY DAY as data accumulates
control_rate, treatment_rate = 0.10, 0.10      # identical — no real effect exists
 
false_positive_found = False
for day in range(1, 31):
    n_so_far = day * 200      # 200 new users per group, per day
    control = np.random.binomial(1, control_rate, n_so_far)
    treatment = np.random.binomial(1, treatment_rate, n_so_far)
 
    _, p_value = stats.ttest_ind(treatment, control)
    if p_value < 0.05 and not false_positive_found:
        print(f"Day {day}: p={p_value:.3f} — 'significant'! (but there's NO real effect)")
        false_positive_found = True

Why Peeking Inflates False Positives
─────────────────────────────────────────
  A single test at n=6000 has a 5% false-positive rate (by design, α=0.05).

  But checking the SAME accumulating data 30 TIMES (once per day) gives
  the random noise 30 SEPARATE chances to cross the significance
  threshold at some point along the way — this is EXACTLY the multiple
  comparisons problem from Chapter 2, just applied across TIME instead
  of across metrics.

  Actual false-positive rate with repeated peeking can climb well
  above 20-30%, even though each individual check used α=0.05.
─────────────────────────────────────────

The fix: decide the sample size (Chapter 1) and/or test duration before starting, and only look at the result once that threshold is reached — or use a formal "sequential testing" statistical method specifically designed to allow safe early stopping (a more advanced topic beyond this primer).

3. The Novelty Effect

Users often react to any change — good or bad — simply because it's different, and that reaction fades over time. A test run for too short a period can mistake temporary novelty for a genuine, lasting improvement.

Novelty Effect, Visualized
─────────────────────────────────────────
  Treatment engagement
       │    ___
       │   /    \___
       │  /          \________________
       │ /                             (settles to the TRUE long-term effect,
       │/                                which might be smaller, zero, or even negative)
       └──────────────────────────────── time
         Week 1    Week 2   Week 3+
─────────────────────────────────────────

import pandas as pd
 
# Checking for a novelty effect: does the treatment's advantage SHRINK over the test's duration?
weekly_results = pd.DataFrame({
    "week": [1, 2, 3, 4],
    "control_rate": [0.100, 0.101, 0.099, 0.102],
    "treatment_rate": [0.135, 0.118, 0.109, 0.105],      # gap SHRINKING week over week
})
weekly_results["lift"] = weekly_results["treatment_rate"] - weekly_results["control_rate"]
print(weekly_results)
# A shrinking "lift" column over time is the classic novelty-effect signature

The fix: run tests long enough to observe whether an effect stabilizes (often at least 2-4 weeks for consumer-facing changes, though this varies by product), and examine week-over-week trends within the test, not just the final aggregate number.

4. Sample Ratio Mismatch (SRM)

If your intended split was 50/50 but the actual observed split is meaningfully different (e.g. 53/47), something is likely broken in the randomization or measurement pipeline — and any result from that test should be considered suspect until the cause is found.

from scipy.stats import chisquare
 
observed = [5320, 4680]      # actual counts in control vs treatment
expected_ratio = [0.5, 0.5]
expected = [sum(observed) * r for r in expected_ratio]
 
chi2_stat, p_value = chisquare(observed, expected)
print(f"Chi-square p-value: {p_value:.5f}")
 
if p_value < 0.01:      # a STRICTER threshold is standard for SRM checks specifically
    print("WARNING: Sample Ratio Mismatch detected — investigate before trusting results!")

Common Causes of SRM
─────────────────────────────────────────
  - A bug in the randomization/assignment code itself
  - One variant loading slower, causing some users to abandon
    before being counted (biasing WHO ends up measured)
  - Bot/crawler traffic being routed disproportionately to one group
  - Logging/analytics pipeline dropping events differently per group
─────────────────────────────────────────

SRM is a serious red flag, not a minor technicality — it means the two groups may no longer be comparable in ways beyond the intended treatment, which undermines the entire causal argument from Chapter 1. Always check for it before interpreting any other result.

5. Network Effects & Interference

A/B testing assumes each user's outcome is independent of every other user's group assignment — this assumption breaks down for anything social or shared (e.g. messaging features, marketplaces, social feeds).

Interference Example
─────────────────────────────────────────
  Testing a new "invite a friend" feature:
    - Treatment group users start inviting CONTROL group friends
    - Control group users are now ALSO affected by the treatment,
      indirectly — "contamination" between groups
    - The MEASURED difference between groups understates the
      TRUE effect, because control isn't a clean baseline anymore
─────────────────────────────────────────

Common mitigations (each with trade-offs, briefly noted for awareness): cluster randomization (randomize by friend-group, region, or market instead of by individual user), or switchback tests (alternate the whole system between variants over time instead of splitting users) — a deeper topic than this primer covers, but essential to recognize when standard per-user A/B testing isn't valid in the first place.

6. Survivorship Bias in Experiment Analysis

Analyzing only users who "survived" through a multi-step funnel can distort results — directly related to Module 2, Chapter 4's missing-data concerns, now applied to experiment analysis specifically.

import pandas as pd
 
# WRONG: analyzing conversion rate only among users who reached checkout
# — but if the treatment changes WHO reaches checkout, this comparison is biased
df = pd.DataFrame({
    "group": ["control"]*1000 + ["treatment"]*1000,
    "reached_checkout": [1]*400 + [0]*600 + [1]*250 + [0]*750,     # treatment: FEWER reach checkout
    "converted": [1]*150 + [0]*850 + [1]*120 + [0]*880,              # but treatment converts BETTER among those who do
})
 
# Biased comparison — only looking at those who reached checkout
reached = df[df["reached_checkout"] == 1]
print(reached.groupby("group")["converted"].mean())      # treatment looks BETTER here
 
# Correct comparison — measure against the FULL original population in each group
print(df.groupby("group")["converted"].mean())             # treatment might actually look WORSE overall!

The fix: always measure the primary metric against the full, originally-assigned population in each group — never only against a subset defined by a downstream behavior that the treatment itself might have influenced.

7. A Pre-Launch Checklist

Bringing every pitfall in this chapter together into a single, practical checklist:

Before Launching Any A/B Test
─────────────────────────────────────────
  ☐ Primary metric defined and written down BEFORE the test starts (Ch.1-2)
  ☐ Sample size calculated in advance, based on a real minimum
    detectable effect (Ch.1) — not "run it until it looks good"
  ☐ A decision made about test DURATION up front, resisting the
    temptation to peek and stop early (Section 2)
  ☐ A plan to check for novelty effects if the change is user-visible
    (Section 3)
  ☐ Sample Ratio Mismatch check planned as part of the analysis (Section 4)
  ☐ Considered whether users in this experiment can influence EACH
    OTHER (Section 5) — if so, standard per-user randomization may
    not be valid
  ☐ Metrics will be measured against the FULL assigned population,
    not a downstream-filtered subset (Section 6)
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

• Peeking at results early and stopping the moment they look favorable dramatically inflates the true false-positive rate — decide sample size/duration in advance and stick to it.
• The novelty effect means an early lift can fade — check whether an effect stabilizes over the test's duration, not just the final aggregate.
• A Sample Ratio Mismatch (observed group sizes deviating meaningfully from the intended split) is a serious red flag that should halt interpretation of any other result until resolved.
• Users influencing each other (network effects) violates the independence assumption standard A/B testing relies on; measuring only a downstream-filtered subset of users (survivorship bias) can flip a result's apparent direction entirely.

Module 5 Complete (through Chapter 3) — Continuing to Chapter 4

You now understand not just how to run an A/B test mechanically, but the practical failure modes that undermine even a statistically well-designed one. Chapter 4 shifts to a different but related applied-statistics skill: analyzing data that unfolds over time.

Concept Check

1. Why does checking a test's results daily and stopping at the first significant day inflate the false-positive rate, even if each individual check uses α=0.05?
2. What does a Sample Ratio Mismatch suggest might be wrong with an experiment, beyond just "bad luck"?
3. Why is it misleading to compare conversion rates only among users who reached a later funnel step, if the treatment itself affects who reaches that step?

Next Chapter

→ Chapter 4: Time Series Analysis Basics

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index