Data Science

Experimentation And Applied Statistics

A/B Testing Fundamentals

An A/B test is a controlled experiment: randomly split users into two (or more) groups, show each group a different version of something, and measure whether th

JrCodex·8 min read

Jr Codex Data Science Notes

Level: Intermediate–Advanced Prerequisites: Module 1, Chapter 6: Hypothesis Testing Time to complete: ~30 minutes

Table of Contents

  1. What is A/B Testing?
  2. Why Randomization Is Non-Negotiable
  3. Choosing a Metric
  4. Determining Sample Size
  5. Running the Test
  6. Analyzing the Results
  7. From Correlation Back to Causation
  8. Summary & Next Steps

1. What is A/B Testing?

An A/B test is a controlled experiment: randomly split users into two (or more) groups, show each group a different version of something, and measure whether the difference in outcomes is statistically real — this is the practical, industry-standard tool for establishing causation, directly answering Module 1, Chapter 5's warning that "correlation is not causation."

A/B Test Structure
─────────────────────────────────────────
  All eligible users
         │
    RANDOM split
         │
    ┌────┴────┐
    ▼         ▼
  Group A   Group B
  (Control)  (Treatment)
  old design  new design
    │         │
    ▼         ▼
  measure   measure
  metric     metric
    │         │
    └────┬────┘
         ▼
  Is the difference STATISTICALLY SIGNIFICANT? (Module 1, Ch.6)
─────────────────────────────────────────

2. Why Randomization Is Non-Negotiable

The entire causal claim of an A/B test rests on random assignment — it's what rules out confounding variables (Module 1, Chapter 5) as the real explanation for any observed difference.

Without Randomization (Confounded)              With Randomization
─────────────────────────────────────           ─────────────────────────────
  Letting users CHOOSE which version               Users assigned RANDOMLY,
  to see (e.g. opt-in to a beta)                     with no say in the matter

  Result: "beta users" convert more —                Result: any conversion difference
  but maybe beta users were already MORE                CANNOT be explained by user
  engaged/tech-savvy BEFORE seeing the beta               differences — groups are
  → confounded, NOT proof the beta caused it              statistically equivalent
                                                            except for the change itself
─────────────────────────────────────────────   ─────────────────────────────

import numpy as np
import pandas as pd
 
np.random.seed(42)
 
n_users = 10000
user_ids = range(1, n_users + 1)
 
# Proper random assignment — every user has an EQUAL, INDEPENDENT chance of either group
assignment = np.random.choice(["control", "treatment"], size=n_users, p=[0.5, 0.5])
 
df = pd.DataFrame({"user_id": user_ids, "group": assignment})
print(df["group"].value_counts())      # should be roughly 50/50

A "sample ratio mismatch" — the groups ending up notably different from the intended split (e.g. 55/45 instead of 50/50) — is itself a warning sign of a broken randomization process, and Chapter 3 covers how to detect it.

3. Choosing a Metric

Before running anything, define one primary metric that determines success — chosen before seeing any results, to avoid the temptation to cherry-pick a metric that happened to look good afterward.

# Common metric types
# Binary/conversion metric: did the user complete X? (signup, purchase, click)
conversions = {"control": 245, "treatment": 289}       # out of some number of users each
 
# Continuous metric: average value per user (revenue, time on page)
avg_revenue = {"control": 12.40, "treatment": 13.85}
Good Metric Criteria
─────────────────────────────────────────
  1. Directly tied to the actual business question
     ("does this button color increase SIGNUPS?", not
      just "does it increase CLICKS?" if clicks aren't the real goal)
  2. Measurable RELIABLY and CONSISTENTLY for every user in both groups
  3. Chosen and written down BEFORE the test starts
─────────────────────────────────────────

Also worth defining upfront: guardrail metrics — things that shouldn't get worse even if the primary metric improves (e.g. a new checkout flow shouldn't increase conversion at the cost of a much higher return rate).

4. Determining Sample Size

Directly connects to Module 1, Chapter 4's standard error and Chapter 6's Type I/II errors — running a test with too few users risks a Type II error (missing a real effect).

from scipy.stats import norm
import numpy as np
 
def required_sample_size(baseline_rate, minimum_detectable_effect, alpha=0.05, power=0.8):
    """
    Rough sample size needed PER GROUP for a two-proportion test.
    baseline_rate: current conversion rate (e.g. 0.10 for 10%)
    minimum_detectable_effect: smallest absolute difference worth detecting (e.g. 0.02 for +2pp)
    """
    p1 = baseline_rate
    p2 = baseline_rate + minimum_detectable_effect
 
    z_alpha = norm.ppf(1 - alpha / 2)      # two-sided significance threshold
    z_beta = norm.ppf(power)                  # power = 1 - Type II error rate (Module 1, Ch.6)
 
    pooled_p = (p1 + p2) / 2
    numerator = (z_alpha * np.sqrt(2 * pooled_p * (1 - pooled_p)) +
                 z_beta * np.sqrt(p1*(1-p1) + p2*(1-p2))) ** 2
    denominator = (p2 - p1) ** 2
 
    return int(np.ceil(numerator / denominator))
 
n_needed = required_sample_size(baseline_rate=0.10, minimum_detectable_effect=0.02)
print(f"Need approximately {n_needed} users PER GROUP")      # e.g. ~3,800
The Trade-off This Formula Reflects
─────────────────────────────────────────
  Smaller effect to detect     → need MORE users (harder to distinguish
                                   from noise, per Module 1's standard error)
  Lower baseline conversion       → need MORE users (rarer events need
                                     bigger samples to see enough of them)
  Wanting MORE statistical power    → need MORE users (lower Type II
  (fewer missed real effects)         error risk, Module 1, Ch.6)
─────────────────────────────────────────

Practical implication: running a test with "however many users show up this week" instead of a pre-calculated sample size risks either wasting time on an underpowered test, or running it needlessly long past when a real effect would already be clearly detectable.

5. Running the Test

import numpy as np
import pandas as pd
 
np.random.seed(1)
 
n_per_group = 4000
 
# Simulating: treatment has a genuinely higher conversion rate (12% vs 10%)
control_conversions = np.random.binomial(1, 0.10, n_per_group)
treatment_conversions = np.random.binomial(1, 0.12, n_per_group)
 
df = pd.DataFrame({
    "user_id": range(1, 2 * n_per_group + 1),
    "group": ["control"] * n_per_group + ["treatment"] * n_per_group,
    "converted": np.concatenate([control_conversions, treatment_conversions]),
})
 
print(df.groupby("group")["converted"].agg(["count", "sum", "mean"]))
#            count  sum   mean
# control     4000   398  0.0995
# treatment   4000   485  0.1213

During the test, resist checking results and stopping early the moment they look favorable — this is "peeking," one of the most common and damaging mistakes in practice, covered fully in Chapter 3.

6. Analyzing the Results

This is Module 1, Chapter 6's two-sample test, applied directly to a real conversion-rate comparison:

from scipy import stats
import numpy as np
 
control = df[df["group"] == "control"]["converted"]
treatment = df[df["group"] == "treatment"]["converted"]
 
# For conversion rates (binary outcomes), a two-proportion z-test is standard
from statsmodels.stats.proportion import proportions_ztest
 
count = np.array([treatment.sum(), control.sum()])
nobs = np.array([len(treatment), len(control)])
 
z_stat, p_value = proportions_ztest(count, nobs)
print(f"z-statistic: {z_stat:.3f}")
print(f"p-value: {p_value:.4f}")
 
alpha = 0.05
if p_value < alpha:
    lift = (treatment.mean() - control.mean()) / control.mean() * 100
    print(f"Statistically significant! Treatment shows a {lift:.1f}% relative lift.")
else:
    print("Not statistically significant — cannot conclude the treatment had an effect.")
# Reporting a confidence interval alongside the point estimate — more informative than p-value alone
from statsmodels.stats.proportion import confint_proportions_2indep
 
ci_low, ci_upp = confint_proportions_2indep(
    count1=treatment.sum(), nobs1=len(treatment),
    count2=control.sum(), nobs2=len(control),
)
print(f"95% CI for the difference in conversion rate: [{ci_low:.4f}, {ci_upp:.4f}]")

A confidence interval communicates more than a p-value alone — "the treatment lifted conversion by somewhere between 0.5 and 3.9 percentage points, with 95% confidence" gives a stakeholder both the direction and a sense of the plausible magnitude, not just a binary "significant/not significant" verdict.

7. From Correlation Back to Causation

This is the payoff Module 1, Chapter 5 promised: an A/B test, done correctly, is the tool that turns an observed association into a defensible causal claim.

Why Randomization Enables a Causal Claim
─────────────────────────────────────────
  Observational correlation (Module 1, Ch.5):
    "users who saw the new design converted more"
    → could be confounded by WHO happened to see it

  A/B test (this chapter):
    "users were RANDOMLY assigned, and the treatment
     group converted significantly more"
    → randomization means the ONLY systematic difference
      between groups is the treatment itself — so it's the
      most defensible explanation for the observed difference
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

  • An A/B test randomly assigns users to control/treatment groups, which is precisely what allows a defensible causal claim — rather than the mere correlation Module 1 warned against.
  • Choose one primary metric (plus guardrail metrics) before the test starts, to avoid post-hoc cherry-picking.
  • Calculate required sample size in advance, based on baseline rate, minimum detectable effect, and desired statistical power — running "whatever traffic shows up" risks an underpowered, inconclusive test.
  • Analyze results with the appropriate test (a two-proportion z-test for conversion rates) and report a confidence interval alongside the p-value, not just a significant/not-significant verdict.

Concept Check

  1. Why does random assignment matter more than simply comparing "users who opted into the new feature" vs those who didn't?
  2. What two factors, besides sample size, feed directly into how large a sample you need for an A/B test?
  3. Why is a confidence interval more informative to a stakeholder than a p-value alone?

Next Chapter

→ Chapter 2: Statistical Significance & p-values in Practice

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index