Sampling & the Central Limit Theorem

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~25 minutes

Table of Contents

1. Population vs Sample
2. Why We Sample At All
3. Sampling Methods
4. Sampling Error
5. The Sampling Distribution of the Mean
6. The Central Limit Theorem
7. Standard Error
8. Why This Matters for Everything After
9. Summary & Next Steps

1. Population vs Sample

Population vs Sample
─────────────────────────────────────────
  Population: EVERY member of the group you care about
              (e.g. all 8 billion humans, all website visitors ever)

  Sample:     a SUBSET of the population you actually collect data on
              (e.g. 1,000 survey respondents)
─────────────────────────────────────────

import numpy as np
 
# In practice, you almost never have the full population — you work with a sample
# and try to infer things ABOUT the population from it.
np.random.seed(42)
 
# Simulating a "true" population for demonstration (in reality, you'd never know this!)
population = np.random.normal(loc=100, scale=15, size=1_000_000)
print(np.mean(population))      # ~100.0 — the TRUE population mean (usually unknowable in real life)
 
# A sample: what you'd actually collect
sample = np.random.choice(population, size=100)
print(np.mean(sample))            # close to 100, but NOT exactly — this gap is "sampling error" (Section 4)

2. Why We Sample At All

Surveying every visitor, testing every product off an assembly line, or measuring every star in the galaxy is usually impossible, too expensive, or too slow. Statistical inference is the entire discipline of drawing reliable conclusions about a population from a properly chosen sample — and it only works if the sample is representative.

Population Too Large/Costly to Fully Measure
─────────────────────────────────────────
  All website visitors (millions)  →  survey a representative 1,000
  All light bulbs off a production line → test a batch of 50
  All voters in a country            → poll 1,500 likely voters
─────────────────────────────────────────

3. Sampling Methods

The how of sampling matters enormously — a biased sampling method produces biased conclusions, no matter how sophisticated the analysis afterward.

import numpy as np
import pandas as pd
 
df = pd.DataFrame({
    "customer_id": range(1, 1001),
    "region": (["North"] * 400 + ["South"] * 350 + ["East"] * 250),
    "spend": np.random.normal(500, 100, 1000),
})
 
# Simple random sampling — every row has an equal chance of selection
simple_sample = df.sample(n=100, random_state=42)
 
# Stratified sampling — preserve each group's PROPORTION in the sample
stratified_sample = df.groupby("region", group_keys=False).apply(
    lambda group: group.sample(frac=0.1, random_state=42)
)
print(stratified_sample["region"].value_counts(normalize=True))
print(df["region"].value_counts(normalize=True))       # should closely match the full population's proportions

The classic cautionary tale: a 1936 US presidential poll surveyed 2 million people (a huge sample!) but sourced them from telephone directories and car registrations — during the Depression, this over-represented wealthier respondents, and the poll's prediction was completely wrong. Sample size doesn't fix a biased sampling method.

4. Sampling Error

Even with a perfectly unbiased sampling method, a sample's statistics (mean, proportion, etc.) will differ slightly from the true population value, purely due to chance — this is sampling error, and it's unavoidable, only manageable.

import numpy as np
 
np.random.seed(1)
population = np.random.normal(loc=50, scale=10, size=1_000_000)
true_mean = np.mean(population)
 
sample_means = []
for _ in range(1000):
    sample = np.random.choice(population, size=30)
    sample_means.append(np.mean(sample))
 
print(f"True population mean: {true_mean:.2f}")
print(f"Average of 1000 sample means: {np.mean(sample_means):.2f}")     # very close to true_mean
print(f"But any ONE sample mean varies: {sample_means[0]:.2f}, {sample_means[1]:.2f}, ...")

Larger samples reduce sampling error (bring individual sample means closer to the true value, on average) but never eliminate it entirely — this trade-off is exactly why sample size calculations matter in Module 5's A/B testing.

5. The Sampling Distribution of the Mean

If you repeatedly took samples from a population and computed each sample's mean, those sample means themselves form a distribution — the sampling distribution of the mean. This is the concept the code in Section 4 was actually demonstrating.

import matplotlib.pyplot as plt
 
plt.hist(sample_means, bins=30)
# The resulting histogram is itself bell-shaped, CENTERED on the true population mean —
# even before we get to the Central Limit Theorem's full statement in Section 6.

6. The Central Limit Theorem

The Central Limit Theorem (CLT) is one of the most important results in all of statistics:

The Central Limit Theorem
─────────────────────────────────────────
  As sample size increases, the sampling distribution of the mean
  approaches a NORMAL distribution — REGARDLESS of the shape of
  the original population's distribution.
─────────────────────────────────────────

import numpy as np
import matplotlib.pyplot as plt
 
np.random.seed(0)
 
# Start with a HEAVILY skewed population — nothing like a normal distribution
skewed_population = np.random.exponential(scale=2, size=1_000_000)
 
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
 
# Original population — skewed, not remotely bell-shaped
axes[0].hist(skewed_population, bins=50)
axes[0].set_title("Original population (skewed)")
 
# Sampling distribution of the mean, sample size = 5
means_n5 = [np.mean(np.random.choice(skewed_population, 5)) for _ in range(2000)]
axes[1].hist(means_n5, bins=50)
axes[1].set_title("Sample means, n=5 (starting to smooth out)")
 
# Sampling distribution of the mean, sample size = 50
means_n50 = [np.mean(np.random.choice(skewed_population, 50)) for _ in range(2000)]
axes[2].hist(means_n50, bins=50)
axes[2].set_title("Sample means, n=50 (clearly bell-shaped!)")

CLT in Action
─────────────────────────────────────────
  Original data:       skewed, long right tail
                             /\_____
                          __/       \_____________

  Sample means (n=50):     bell-shaped, symmetric — EVEN THOUGH
                                 __                the original data
                              _-'  '-_               was skewed!
                            _/        \_
─────────────────────────────────────────

Why this matters so much: it means you can apply normal-distribution-based tools (confidence intervals, hypothesis tests — Chapter 6, and A/B testing in Module 5) to sample means, without needing the underlying data itself to be normally distributed — as long as the sample size is reasonably large (a common rule of thumb is n ≥ 30).

7. Standard Error

The standard error (SE) is the standard deviation of the sampling distribution of the mean — it measures how much sample means typically vary from the true population mean.

import numpy as np
 
population_std = 10          # standard deviation of the ORIGINAL population
sample_size = 30
 
standard_error = population_std / np.sqrt(sample_size)
print(standard_error)      # ~1.83

Standard Error Formula
─────────────────────────────────────────
  SE = population standard deviation / √(sample size)

  Larger sample size (n) → smaller SE → sample means cluster
  MORE tightly around the true population mean
─────────────────────────────────────────

# Demonstrating: SE shrinks as sample size grows
for n in [10, 100, 1000]:
    se = population_std / np.sqrt(n)
    print(f"n={n}: standard error = {se:.3f}")
# n=10: standard error = 3.162
# n=100: standard error = 1.000
# n=1000: standard error = 0.316

This is the mathematical reason "bigger samples give more reliable estimates" — and it's also why the rate of improvement slows down: quadrupling your sample size only halves your standard error (since it depends on the square root of n), a fact that matters directly for planning experiment sizes in Module 5.

8. Why This Matters for Everything After

The Chain of Reasoning This Chapter Builds
─────────────────────────────────────────
  1. We can rarely measure a whole population → we sample instead (Sections 1-3)
  2. Any single sample has some error vs the true population value (Section 4)
  3. But sample MEANS, across many samples, form a predictable,
     NORMAL distribution — regardless of the original data's shape (Section 6)
  4. This lets us quantify uncertainty (standard error, Section 7)
     and build the hypothesis tests in Chapter 6 and the A/B tests in Module 5
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

• A sample is a subset used to infer things about a population — sampling method matters more than sample size for avoiding bias (the 1936 poll disaster).
• Sampling error is the unavoidable difference between a sample's statistic and the true population value; larger samples reduce it but never eliminate it.
• The Central Limit Theorem guarantees that sample means become approximately normally distributed as sample size grows, regardless of the original population's shape — the foundation for most classical statistical inference.
• Standard error (population std / √n) quantifies how much sample means vary — it shrinks with larger samples, but only at the rate of the square root of n.

Concept Check

1. Why did a 2-million-person poll in 1936 still produce a badly wrong prediction?
2. What does the Central Limit Theorem guarantee, even when the original population data is heavily skewed?
3. If you quadruple your sample size, what happens to the standard error?

Next Chapter

→ Chapter 5: Correlation & Covariance

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index