Correlation & Covariance

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 4 Time to complete: ~20 minutes

Table of Contents

1. What is Covariance?
2. From Covariance to Correlation
3. Pearson Correlation in Practice
4. Visualizing Correlation with Scatter Plots
5. Spearman Correlation — for Non-Linear Relationships
6. Correlation Matrices
7. Correlation Is NOT Causation
8. Summary & Next Steps

1. What is Covariance?

Covariance measures whether two variables tend to move together — when one goes up, does the other tend to go up too (positive covariance), go down (negative), or show no consistent pattern (near zero)?

import numpy as np
 
hours_studied = np.array([1, 2, 3, 4, 5])
exam_scores = np.array([50, 55, 65, 70, 80])
 
covariance_matrix = np.cov(hours_studied, exam_scores)
print(covariance_matrix)
# [[ 2.5   17.5]
#  [17.5  132.5]]
print(covariance_matrix[0, 1])      # 17.5 — the covariance BETWEEN the two variables

Reading a Covariance Matrix
─────────────────────────────────────────
  [[ var(hours),        cov(hours, scores) ]
   [ cov(hours, scores),  var(scores)      ]]

  The diagonal = each variable's own VARIANCE (Chapter 1)
  The off-diagonal = the COVARIANCE between them
─────────────────────────────────────────

The problem with raw covariance: its scale depends on the units of the original variables, making it hard to interpret or compare. 17.5 — is that a strong relationship or a weak one? You can't tell without more context. This is exactly what correlation fixes.

2. From Covariance to Correlation

Correlation is covariance, standardized to always fall between -1 and +1 — making it directly interpretable and comparable across any pair of variables, regardless of their original units.

Pearson Correlation Coefficient (r)
─────────────────────────────────────────
  r = covariance(X, Y) / (std_dev(X) × std_dev(Y))

  r = +1   →  perfect POSITIVE linear relationship
  r =  0    →  NO linear relationship
  r = -1     →  perfect NEGATIVE linear relationship
─────────────────────────────────────────

import numpy as np
 
hours_studied = np.array([1, 2, 3, 4, 5])
exam_scores = np.array([50, 55, 65, 70, 80])
 
correlation_matrix = np.corrcoef(hours_studied, exam_scores)
print(correlation_matrix[0, 1])      # ~0.99 — a very strong positive linear relationship

3. Pearson Correlation in Practice

import pandas as pd
 
df = pd.DataFrame({
    "hours_studied": [1, 2, 3, 4, 5, 6, 7, 8],
    "exam_score": [50, 55, 65, 70, 80, 82, 88, 95],
    "hours_slept": [8, 7, 8, 6, 7, 5, 6, 8],       # deliberately close to unrelated
})
 
print(df["hours_studied"].corr(df["exam_score"]))      # ~0.98 — strong positive
print(df["hours_studied"].corr(df["hours_slept"]))        # near 0 — weak/no linear relationship

These thresholds are rules of thumb, not hard rules — what counts as "strong" varies by field (a 0.3 correlation might be notable in social science, unremarkable in physics).

4. Visualizing Correlation with Scatter Plots

Correlation is a single number — but always look at the actual data too, since very different patterns can produce identical correlation values (previewed here, covered fully in Module 4's visualization chapters).

import matplotlib.pyplot as plt
 
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
 
axes[0].scatter(df["hours_studied"], df["exam_score"])
axes[0].set_title(f"r = {df['hours_studied'].corr(df['exam_score']):.2f} (strong, linear)")
 
axes[1].scatter(df["hours_studied"], df["hours_slept"])
axes[1].set_title(f"r = {df['hours_studied'].corr(df['hours_slept']):.2f} (weak/none)")
 
# A famous cautionary case: a strong NON-linear relationship can still show LOW Pearson correlation
import numpy as np
x = np.linspace(-10, 10, 100)
y = x ** 2                            # a clean parabola — a very real, strong relationship
axes[2].scatter(x, y)
axes[2].set_title(f"r = {np.corrcoef(x, y)[0,1]:.2f} (near zero, despite an obvious pattern!)")

Why the Parabola "Fools" Pearson Correlation
─────────────────────────────────────────
  Pearson correlation ONLY detects LINEAR relationships.
  y = x² is a perfectly predictable, strong relationship —
  but it curves, so Pearson's r comes out near 0.

  Lesson: ALWAYS plot your data (Module 4) — never trust
  a correlation number alone.
─────────────────────────────────────────

5. Spearman Correlation — for Non-Linear Relationships

Spearman correlation measures whether two variables have a consistent monotonic relationship (as one increases, does the other consistently increase or decrease — even if not in a straight line), by correlating each variable's ranks instead of its raw values.

import pandas as pd
 
df = pd.DataFrame({
    "x": [1, 2, 3, 4, 5],
    "y_exponential": [2, 8, 27, 90, 300],      # non-linear, but CONSISTENTLY increasing
})
 
pearson_r = df["x"].corr(df["y_exponential"], method="pearson")
spearman_r = df["x"].corr(df["y_exponential"], method="spearman")
 
print(f"Pearson: {pearson_r:.3f}")      # ~0.88 — good, but understates the relationship's consistency
print(f"Spearman: {spearman_r:.3f}")      # 1.000 — PERFECT, because y always increases as x increases

6. Correlation Matrices

For datasets with many numeric columns, compute all pairwise correlations at once:

import pandas as pd
import numpy as np
 
np.random.seed(0)
df = pd.DataFrame({
    "age": np.random.randint(20, 60, 100),
    "income": np.random.randint(30000, 120000, 100),
    "hours_worked": np.random.randint(30, 60, 100),
})
df["income"] = df["income"] + df["age"] * 500      # deliberately correlate income with age
 
corr_matrix = df.corr(numeric_only=True)
print(corr_matrix)
#                    age    income  hours_worked
# age           1.000000  0.522891     -0.034561
# income        0.522891  1.000000      0.012098
# hours_worked -0.034561  0.012098      1.000000

This is almost always the first thing worth visualizing as a heatmap in EDA (Module 3) — a quick way to spot which variable pairs are worth investigating further, and to catch multicollinearity (two features that are highly correlated with each other) before feeding data into a model in the Machine Learning notes.

7. Correlation Is NOT Causation

The single most important, most frequently ignored warning in all of applied statistics.

Classic Spurious Correlation Examples
─────────────────────────────────────────
  Ice cream sales  ↔  Drowning deaths        (r is HIGH, positive)
    Real cause: BOTH increase in summer heat — ice cream doesn't cause drowning!

  Number of firefighters at a fire ↔ Damage caused
    Real cause: BIGGER fires get MORE firefighters AND cause more damage —
    firefighters aren't causing the damage.
─────────────────────────────────────────

Three Ways Two Variables Can Correlate Without A Causing B
─────────────────────────────────────────
  1. B actually causes A (reverse causation)
  2. A confounding variable C causes BOTH A and B (the ice cream/drowning case — heat is C)
  3. Pure coincidence (especially likely when testing MANY variable pairs at once)
─────────────────────────────────────────

Practical rule: a strong correlation is a reason to investigate further (design an experiment, control for confounders), never a conclusion on its own. Module 5's A/B testing chapter covers the proper tool for establishing actual causation — controlled experiments, not observed correlation.

8. Summary & Next Steps

Key Takeaways

• Covariance shows the direction two variables move together, but its scale is uninterpretable on its own; correlation standardizes it to a comparable -1 to +1 range.
• Pearson correlation only detects linear relationships — a strong non-linear relationship (like a parabola) can show a near-zero Pearson r, which is why you always plot the data too.
• Spearman correlation, based on ranks, catches any consistent monotonic relationship, linear or not.
• Correlation, no matter how strong, never proves causation — always consider reverse causation and confounding variables before concluding "A causes B."

Concept Check

1. Why is a raw covariance value of "17.5" hard to interpret on its own, and what does correlation fix?
2. Why can y = x² produce a Pearson correlation near 0, despite being a very predictable relationship?
3. In the ice cream/drowning example, what's the actual confounding variable causing both to rise together?

Next Chapter

→ Chapter 6: Hypothesis Testing Fundamentals

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index