Outlier Detection

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~20 minutes

Table of Contents

1. What is an Outlier?
2. The IQR Method
3. The Z-Score Method
4. IQR vs Z-Score — Which to Use
5. Visualizing Outliers
6. Deciding What To Do With an Outlier
7. Multivariate Outliers
8. Summary & Next Steps

1. What is an Outlier?

An outlier is a data point that differs substantially from the rest of the dataset. Outliers matter because they can be either genuine, important signal (a fraud transaction, a record-breaking sale) or data errors (a mistyped age of 999) — and treating one as the other leads to real mistakes.

Two Very Different Kinds of Outlier
─────────────────────────────────────────
  Genuine outlier:  a customer who spent $50,000 in one order —
                     REAL, and possibly the most important row
                     in the entire dataset (a whale customer, or fraud).

  Error outlier:     age = 999, or a negative price —
                      almost certainly a DATA ENTRY MISTAKE,
                      should usually be corrected or removed.
─────────────────────────────────────────

This chapter's tools detect candidates — deciding which kind you're looking at always requires judgment and, ideally, domain knowledge (Section 6).

2. The IQR Method

Introduced in Module 1, Chapter 1 — now applied as a full outlier-detection routine.

import pandas as pd
 
df = pd.read_csv("customer_data.csv")
 
q1 = df["purchase_amount"].quantile(0.25)
q3 = df["purchase_amount"].quantile(0.75)
iqr = q3 - q1
 
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr
 
outliers = df[(df["purchase_amount"] < lower_bound) | (df["purchase_amount"] > upper_bound)]
print(f"Found {len(outliers)} outliers out of {len(df)} rows")
print(f"Bounds: [{lower_bound:.2f}, {upper_bound:.2f}]")

The 1.5×IQR Rule, Visualized
─────────────────────────────────────────
  outlier         outlier
    zone     ┌─────┬─────┐    zone
  ●  ●  ●────┤ Q1  │ Q3  ├────●    ●
             └─────┴─────┘
  lower_bound         upper_bound
  = Q1 - 1.5×IQR      = Q3 + 1.5×IQR

  (this is EXACTLY what a box plot's whiskers and dots visualize, Chapter 2)
─────────────────────────────────────────

Why 1.5? It's a widely-used convention (not a law of nature) — under a normal distribution, it flags roughly the most extreme 0.7% of values. Some analyses use 3.0 instead of 1.5 for a stricter "extreme outliers only" definition.

3. The Z-Score Method

Measures how many standard deviations a value is from the mean — directly using Module 1's normal distribution and empirical rule.

import numpy as np
import pandas as pd
from scipy import stats
 
df = pd.read_csv("customer_data.csv")
 
z_scores = stats.zscore(df["purchase_amount"].dropna())
outlier_mask = np.abs(z_scores) > 3        # flag anything more than 3 std devs from the mean
 
outliers_z = df.loc[df["purchase_amount"].dropna().index[outlier_mask]]
print(f"Found {len(outliers_z)} outliers using the Z-score method")

Z-Score Formula
─────────────────────────────────────────
  z = (value - mean) / std_dev

  |z| > 3   →  more than 3 standard deviations away —
               under the empirical rule (Module 1, Ch.3), this
               covers only ~0.3% of a normal distribution
─────────────────────────────────────────

4. IQR vs Z-Score — Which to Use

import pandas as pd
 
# The critical difference shows up on SKEWED data — compare on a right-skewed column
df = pd.read_csv("customer_data.csv")
print(f"Skewness: {df['purchase_amount'].skew():.2f}")     # likely notably positive (Module 1, Ch.1)

Practical rule of thumb: default to the IQR method for most real-world business data (which is often right-skewed) — reserve the Z-score method for data you've already confirmed is roughly normally distributed (a quick check: does the skewness value from Chapter 2 sit close to 0?).

5. Visualizing Outliers

import matplotlib.pyplot as plt
 
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
 
axes[0].boxplot(df["purchase_amount"].dropna())
axes[0].set_title("Box Plot — outliers shown as individual dots")
 
axes[1].scatter(range(len(df)), df["purchase_amount"], alpha=0.3, s=10)
axes[1].axhline(upper_bound, color="red", linestyle="--", label="Upper bound")
axes[1].axhline(lower_bound, color="red", linestyle="--", label="Lower bound")
axes[1].set_title("Scatter with IQR bounds")
axes[1].legend()

A box plot is the fastest single view; a scatter plot against row index (or against a date, if there is one) can reveal whether outliers are scattered randomly or clustered in a specific time period — which is itself a useful diagnostic clue about why they occurred.

6. Deciding What To Do With an Outlier

Detection is mechanical; the decision is not. Every flagged outlier deserves a specific judgment call, not an automatic rule.

A Decision Framework
─────────────────────────────────────────
  1. Is it PHYSICALLY/LOGICALLY possible?
     age = 300  → impossible → almost certainly an ERROR → fix or remove

  2. Is it possible, but IMPLAUSIBLE for this dataset's context?
     A $50,000 order in a dataset of typical $50 orders → investigate:
       - Check for a data entry error (extra zero?)
       - Check if it's a legitimate bulk/wholesale order
     → the ANSWER determines whether to keep, cap, or remove it

  3. Is it genuinely extreme, but clearly REAL and IMPORTANT?
     A record company sales day, a viral event → KEEP —
     removing it would hide the exact signal you might care about most
─────────────────────────────────────────

import pandas as pd
import numpy as np
 
# Capping (also called "winsorizing") — an alternative to removing outliers entirely
lower_cap = df["purchase_amount"].quantile(0.01)
upper_cap = df["purchase_amount"].quantile(0.99)
df["purchase_amount_capped"] = df["purchase_amount"].clip(lower=lower_cap, upper=upper_cap)
 
# This PRESERVES the row (and its other column values) while limiting the
# extreme value's influence on statistics like the mean.

Never silently drop outliers without documenting why — Chapter 1's EDA-findings habit applies directly here: "removed 12 rows where purchase_amount > $10,000, confirmed via [data source] to be data entry errors" belongs in your notes, both for reproducibility and so a collaborator (or your future self) can question the decision if needed.

7. Multivariate Outliers

A value can look completely normal on its own, yet be an outlier in combination with another variable — something univariate methods (Sections 2-3) will never catch.

import matplotlib.pyplot as plt
 
# A 25-year-old is normal. An income of $200,000 is unusual but not impossible.
# A 25-year-old WITH an income of $200,000 might be a genuine multivariate outlier.
plt.scatter(df["age"], df["income"], alpha=0.3)
plt.xlabel("Age")
plt.ylabel("Income")
# Look for points that sit AWAY from the main cloud, even if each axis alone looks unremarkable

This is exactly why Chapter 3's bivariate scatter plots double as an outlier-detection tool — a point isolated from the main cluster, even with individually unremarkable values on each axis, is worth flagging for the same judgment process as Section 6.

8. Summary & Next Steps

Key Takeaways

• Outliers can be genuine, important signal or data errors — detection is mechanical, but deciding which one you're looking at always requires judgment and context.
• The IQR method (1.5×IQR beyond Q1/Q3) is robust to skew and generally the safer default for real-world business data; the Z-score method assumes a roughly normal distribution.
• Never automatically drop flagged outliers — investigate each one, and document the decision and reasoning.
• Multivariate outliers (unusual combinations of otherwise-normal values) require looking at variables together, not one at a time — Chapter 3's scatter plots double as an outlier-detection tool here.

Module 3 Complete — Next Module

You now have a complete EDA toolkit: the overall workflow, univariate analysis, bivariate/multivariate relationships, and outlier detection. Chapter 5's case study ties all of it together on one realistic dataset from start to finish. From there, Module 4 goes deep on the visualization tools used throughout this module.

Concept Check

1. Why is the IQR method generally preferred over the Z-score method for skewed, real-world data?
2. What's the difference between capping (winsorizing) an outlier and simply deleting its row?
3. Why can't univariate methods (IQR, Z-score on one column) catch a multivariate outlier?

Next Chapter

→ Chapter 5: EDA Case Study

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index