Experimentation And Applied Statistics
Storytelling with Data
Every technique in this curriculum — statistics (Module 1), wrangling (Module 2), EDA (Module 3), visualization (Module 4), experimentation (this module) — ulti
Jr Codex Data Science Notes
Level: Intermediate Prerequisites: Chapter 4: Time Series Analysis Basics Time to complete: ~20 minutes
Table of Contents
- Analysis Is Not the Deliverable
- Structuring a Data Narrative
- One Chart, One Message
- Leading with the Answer
- Honest Storytelling — Not Manipulative Storytelling
- Adapting the Same Analysis for Different Audiences
- A Complete Example
- Summary & Next Steps
1. Analysis Is Not the Deliverable
Every technique in this curriculum — statistics (Module 1), wrangling (Module 2), EDA (Module 3), visualization (Module 4), experimentation (this module) — ultimately serves one goal: helping someone else make a better decision. An analysis that never gets understood or acted on, no matter how rigorous, has failed at its actual job.
The Gap This Chapter Closes
─────────────────────────────────────────
"I ran a rigorous A/B test, computed p-values, effect sizes,
and confidence intervals..."
↓
"...but the stakeholder meeting glazed over at the word
'confidence interval' and nothing got decided."
─────────────────────────────────────────
2. Structuring a Data Narrative
A reliable structure, directly connecting Chapter 1's A/B testing rigor to Module 4's audience-awareness principle:
The Standard Narrative Arc
─────────────────────────────────────────
1. CONTEXT — what question were we trying to answer, and why does it matter?
2. HEADLINE — the single most important finding, stated in plain language, up front
3. EVIDENCE — the chart(s)/numbers that support the headline
4. NUANCE/CAVEATS — what this does and doesn't prove; confidence, limitations
5. RECOMMENDATION — the specific action this finding suggests
─────────────────────────────────────────
Weak Structure (analysis-first) Strong Structure (answer-first)
───────────────────────────────────── ─────────────────────────────────
"We ran a two-sample t-test on "The new checkout flow increases
conversion rates across 8,000 users revenue by an estimated 4%.
per group over a 3-week period, Here's how we know, and what
using a 95% confidence level..." we recommend doing next."
───────────────────────────────────── ─────────────────────────────────
3. One Chart, One Message
Directly extending Module 4, Chapter 1's principle — each chart in a stakeholder-facing narrative should support exactly one point, titled with the conclusion, not just a description of the axes.
import matplotlib.pyplot as plt
import pandas as pd
df = pd.DataFrame({
"group": ["Control", "Treatment"],
"conversion_rate": [10.0, 10.4],
})
fig, ax = plt.subplots(figsize=(6, 4))
bars = ax.bar(df["group"], df["conversion_rate"], color=["gray", "steelblue"])
# The title states the CONCLUSION, not just "Conversion Rate by Group"
ax.set_title("New Checkout Flow Increased Conversion by 4%", fontsize=13, fontweight="bold")
ax.set_ylabel("Conversion Rate (%)")
# Direct annotation — the reader doesn't have to compute the difference themselves
for bar, value in zip(bars, df["conversion_rate"]):
ax.text(bar.get_x() + bar.get_width()/2, value + 0.1, f"{value}%",
ha="center", fontweight="bold")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)Descriptive Title Conclusion Title
───────────────────────────── ─────────────────────────────
"Conversion Rate by Group" "New Checkout Flow Increased
(forces the reader to interpret Conversion by 4%"
the chart themselves) (states the takeaway directly —
the chart now just PROVES it)
───────────────────────────── ─────────────────────────────
4. Leading with the Answer
Technical writing (a paper, documentation) often builds up to a conclusion. Business communication should usually do the reverse — state the conclusion first, then support it, since most readers (especially executives) decide how much attention to give something within the first sentence.
Weak Opening
─────────────────────────────────────────
"We analyzed 90 days of transaction data across four regions,
controlling for seasonality and applying a Bonferroni correction
for the twelve metrics tested, and found that..."
─────────────────────────────────────────
Strong Opening
─────────────────────────────────────────
"The West region's customers spend 60% more per order than any
other region — and this holds even after controlling for order
frequency. Recommend investigating West's marketing channel mix
as a candidate for expansion. Full methodology below."
─────────────────────────────────────────
This mirrors exactly the EDA case study's findings write-up from Module 3, Chapter 5 — lead with the finding, follow with the support, and put full methodological detail at the end (or in an appendix) for whoever wants to dig deeper.
5. Honest Storytelling — Not Manipulative Storytelling
Every technique in this chapter is about clarity, never about making a weak result look stronger than it is — this is where storytelling connects directly back to Module 4, Chapter 1's warnings about misleading charts, and Chapter 2's insistence on reporting effect size honestly.
Where the Line Is
─────────────────────────────────────────
Legitimate: choosing the clearest chart type, leading with the
headline, annotating the key number directly
NOT legitimate: truncating a y-axis to exaggerate a tiny difference,
cherry-picking a time window that flatters the result,
omitting a guardrail metric that moved in a bad direction
─────────────────────────────────────────
A good storytelling practice is a discipline, not a decoration: if honestly presenting a result means the headline is "the effect was small and we recommend not shipping this," that's exactly the story to tell — the goal is the clearest path to the truth, not the most flattering one.
6. Adapting the Same Analysis for Different Audiences
Directly following up on Module 4, Chapter 1's "know your audience" principle — the underlying analysis stays the same, but its presentation should adapt:
Same Finding, Three Audiences
─────────────────────────────────────────
Executive summary (1 sentence + 1 chart):
"New checkout flow lifts revenue ~4%; recommend full rollout."
Product team (paragraph + a few charts):
Includes the effect size, confidence interval, guardrail metric
results, and a specific recommended next step.
Data science peer review (full report):
Includes methodology, sample size calculation, statistical test
used, any pitfalls checked (Chapter 3: SRM, peeking, novelty
effect), and raw numbers for independent verification.
─────────────────────────────────────────
The underlying rigor (Chapters 1-4) should be identical across all three — what changes is how much of that rigor is shown up front versus available on request.
7. A Complete Example
Bringing every chapter in this module together into one realistic stakeholder-facing summary:
Subject: Checkout Redesign — A/B Test Results
HEADLINE
The redesigned checkout flow increased conversion from 10.0% to
10.4% (a 4% relative lift), and we recommend a full rollout.
EVIDENCE
[Bar chart: Control vs Treatment conversion rate, annotated with %]
- 95% confidence interval: [0.1%, 0.7%] — a real, positive effect
- Sample size: 4,000 users per group, pre-calculated for 80% power
to detect this effect (Chapter 1)
- Guardrail metrics (refund rate, avg order value): no significant
change (Chapter 1)
CAVEATS
- Test ran for 3 weeks; a week-over-week check (Chapter 3) showed
the lift was STABLE, not a fading novelty effect
- Sample Ratio Mismatch check passed (49.8% / 50.2% split, expected)
- This is a MODEST but genuine effect — not transformative, but the
engineering cost of this change was also low
RECOMMENDATION
Ship to 100% of users. Suggest a follow-up test on the higher-value
enterprise segment specifically, where a similar % lift would
represent a larger absolute revenue impact.
Every claim in this summary traces back to a specific technique from Modules 1-5 — the statistics justify the confidence, but the structure (headline first, evidence second, honest caveats, clear recommendation) is what actually makes it usable by someone who wasn't in the room for the analysis itself.
8. Summary & Next Steps
Key Takeaways
- An analysis only succeeds if it changes a decision — structure communication around context, headline, evidence, caveats, and recommendation, in that order.
- Each chart should support exactly one message, with a title stating the conclusion rather than just describing the axes.
- Lead with the answer, not the methodology — put full technical detail at the end or in an appendix for those who want it.
- Honest storytelling means presenting a weak or null result just as clearly as a strong one — the goal is clarity toward the truth, not the most flattering framing.
- The same underlying analysis should adapt its level of detail to its audience, while keeping the underlying rigor identical across every version.
Module 5 Complete — Next Module
You now have the complete experimentation and applied-statistics toolkit — A/B testing, significance in practice, design pitfalls, time series basics, and how to communicate all of it honestly and persuasively. Module 6 closes out this entire curriculum with the broader data science workflow and a full capstone project.
Concept Check
- Why should a stakeholder-facing chart's title state the conclusion rather than just describing what's plotted?
- What's the difference between adapting an analysis's presentation for different audiences and being dishonest about the results?
- In the "leading with the answer" principle, where should detailed methodology go instead of the opening?
Next Chapter
→ Module 6: Data Science in Practice
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index