Handling Missing & Duplicate Data

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~25 minutes

Table of Contents

1. Why Missing Data Deserves Its Own Chapter
2. Detecting Missing Data
3. Why Data Is Missing — MCAR, MAR, MNAR
4. Strategy 1: Deletion
5. Strategy 2: Imputation
6. Strategy 3: Flagging "Missingness" as Information
7. Detecting & Handling Duplicates
8. Summary & Next Steps

1. Why Missing Data Deserves Its Own Chapter

The Python Notes' Pandas Primer showed .dropna() and .fillna() as quick mechanical tools. In real data science work, the choice between them (and everything in between) has real consequences for any analysis or model built downstream — this chapter is about making that choice deliberately.

2. Detecting Missing Data

import pandas as pd
import numpy as np
 
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "age": [25, np.nan, 35, np.nan, 28],
    "income": [50000, 62000, np.nan, 71000, np.nan],
    "city": ["Boston", "NYC", "LA", None, "Boston"],
})
 
print(df.isna().sum())
# name      0
# age       2
# income    2
# city      1
 
print(df.isna().mean() * 100)      # percentage missing per column
# age       40.0
# income    40.0
# city      20.0

import matplotlib.pyplot as plt
 
# A quick visual of WHERE data is missing — invaluable for spotting patterns (Section 3)
plt.figure(figsize=(8, 4))
plt.imshow(df.isna(), aspect="auto", cmap="viridis")
plt.xticks(range(len(df.columns)), df.columns, rotation=45)
plt.title("Missing Data Pattern")

First question to ask, always: what percentage of each column is missing? A column missing 2% of values is a very different problem than one missing 60%.

3. Why Data Is Missing — MCAR, MAR, MNAR

Understanding why data is missing determines whether a cleaning strategy will introduce bias:

Three Categories of Missingness
─────────────────────────────────────────
  MCAR (Missing Completely At Random)
    The fact that it's missing has NOTHING to do with any value —
    e.g. a random sensor glitch. SAFEST to handle — dropping or
    imputing rarely introduces bias.

  MAR (Missing At Random)
    Missingness relates to OTHER observed columns, not the missing
    value itself — e.g. older survey respondents more often skip
    the "income" question. Can be handled well IF you use the
    related columns during imputation.

  MNAR (Missing Not At Random)
    Missingness relates to the missing value ITSELF — e.g. people
    with VERY HIGH incomes are more likely to decline answering the
    income question. The HARDEST case — naive imputation can badly
    bias results.
─────────────────────────────────────────

# A quick, practical check for MAR: does missingness in one column correlate with another column's values?
df["income_missing"] = df["income"].isna()
print(df.groupby("income_missing")["age"].mean())
# If average age differs a lot between the two groups, missingness likely relates to age (MAR)

There's no code that automatically tells you which category you're in — it requires domain knowledge about how the data was collected. But this judgment should always come before choosing a strategy in Sections 4-6.

4. Strategy 1: Deletion

# Drop rows with ANY missing value
df_dropped_any = df.dropna()
 
# Drop rows only if a SPECIFIC column is missing
df_dropped_income = df.dropna(subset=["income"])
 
# Drop columns that are mostly missing (e.g. more than 50%)
threshold = len(df) * 0.5
df_dropped_cols = df.dropna(axis=1, thresh=threshold)

Risk: if missingness is MAR or MNAR (Section 3), dropping rows systematically removes a specific kind of record — e.g. dropping every row missing income might disproportionately remove high earners, quietly skewing every downstream statistic.

5. Strategy 2: Imputation

Imputation fills in a reasonable estimate instead of removing the record entirely — preserving sample size at the cost of introducing an estimate rather than a real observation.

import pandas as pd
 
# Simple imputation — mean, median, or a constant
df["age_filled"] = df["age"].fillna(df["age"].mean())
df["age_filled_median"] = df["age"].fillna(df["age"].median())      # more robust to outliers (Module 1, Ch.1)
 
# For categorical data, the MODE (most frequent value) is the equivalent choice
df["city_filled"] = df["city"].fillna(df["city"].mode()[0])
 
# Forward/backward fill — useful for TIME-ORDERED data (e.g. daily stock prices)
time_series = pd.Series([100, np.nan, np.nan, 103, 105])
print(time_series.ffill())      # [100, 100, 100, 103, 105] — carries the last known value forward
print(time_series.bfill())        # [100, 103, 103, 103, 105] — carries the NEXT known value backward

# Group-aware imputation — often far better than a single global mean
df["income_filled"] = df.groupby("city")["income"].transform(
    lambda x: x.fillna(x.mean())
)
# Fills each city's missing income with THAT CITY's average, not the overall average —
# directly connects back to why MAR-aware imputation (Section 3) matters

Imputation Trade-off
─────────────────────────────────────────
  Simple (single mean/median):  fast, but IGNORES any relationship
                                  between the missing value and other columns

  Group-aware / model-based:      more accurate, but more complex —
                                    worth it once missingness looks like MAR
─────────────────────────────────────────

6. Strategy 3: Flagging "Missingness" as Information

Sometimes the fact that a value is missing is itself a useful signal — worth preserving explicitly rather than hiding it.

import pandas as pd
 
# Add a binary indicator column BEFORE imputing — preserves the "was this missing?" signal
df["income_was_missing"] = df["income"].isna().astype(int)
df["income"] = df["income"].fillna(df["income"].median())
 
# Now a downstream model (Machine Learning notes) can learn from BOTH:
#   - the imputed income value itself
#   - whether that value was originally missing (which might correlate with the target)

This pattern is common in practice precisely because MNAR missingness (Section 3) means the absence of data can carry real signal — for example, "declined to state income" might itself correlate with something you care about predicting.

7. Detecting & Handling Duplicates

A separate, related problem — the same record appearing more than once, usually from data entry errors or merging multiple sources.

import pandas as pd
 
df = pd.DataFrame({
    "customer_id": [1, 2, 3, 2, 4],
    "name": ["Alice", "Bob", "Charlie", "Bob", "Diana"],
    "email": ["a@x.com", "b@x.com", "c@x.com", "b@x.com", "d@x.com"],
})
 
print(df.duplicated())            # boolean Series — True where a FULL row repeats an earlier one
print(df.duplicated().sum())         # 1 — one exact duplicate row
 
# Duplicates based on a SUBSET of columns (e.g. same customer_id, even if other fields differ slightly)
print(df.duplicated(subset=["customer_id"]))
 
df_deduped = df.drop_duplicates()                              # drop exact duplicate rows
df_deduped_by_id = df.drop_duplicates(subset=["customer_id"], keep="first")   # keep first occurrence only

Watch for "fuzzy" duplicates too — the same real-world entity recorded slightly differently ("Bob Smith" vs "bob smith" vs "B. Smith"), which drop_duplicates() won't catch on its own; this is exactly why Chapter 3's text-standardization step (.str.strip().str.lower()) should generally run before deduplication, not after.

8. Summary & Next Steps

Key Takeaways

• Always quantify missingness (.isna().mean()) per column before deciding what to do — 2% missing and 60% missing call for very different strategies.
• Understanding why data is missing (MCAR/MAR/MNAR) determines whether dropping or naive imputation will introduce bias — MNAR is the hardest and riskiest case.
• Deletion is simplest but risks losing systematic information; imputation preserves sample size but requires judgment (mean/median, group-aware, or forward/backward fill for time series).
• Adding a "was this missing?" indicator column before imputing preserves potentially useful signal that a blanket fill would otherwise erase.
• Deduplicate after standardizing text formatting (Chapter 3) — otherwise near-duplicates with inconsistent casing/whitespace slip through undetected.

Concept Check

1. Why does dropping rows with missing income risk biasing your dataset if the missingness is MNAR?
2. When would group-aware imputation (e.g. filling by city) be preferable to a single global mean?
3. Why should text cleaning (Chapter 3) typically happen before checking for duplicates?

Next Chapter

→ Chapter 5: Feature Engineering Basics

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index