Bias & Fairness in AI

Jr Codex AI Notes

Level: Intermediate Prerequisites: Module 5: Agents, Multi-Agent Systems & RL Time to complete: ~25 minutes

Table of Contents

1. Where Bias Comes From
2. Historical Bias in Training Data
3. Representation Bias
4. Measurement Bias
5. Defining Fairness — Multiple, Conflicting Definitions
6. The Impossibility of Satisfying All Fairness Definitions
7. Practical Mitigation Strategies
8. Summary & Next Steps

1. Where Bias Comes From

Bias in an AI system means it systematically produces worse or unfair outcomes for certain groups — not because anyone deliberately programmed unfairness, but because bias enters through the data, the problem framing, or the deployment context. This chapter's concern is squarely with learned systems (Machine Learning, Deep Learning, NLP/LLM notes), since bias enters differently than in the rule-based systems of Module 3.

Where Bias Enters an AI Pipeline
─────────────────────────────────────────
  Historical Bias:     the WORLD itself, reflected in the data,
                          already contains unfair patterns (Section 2)
  Representation Bias:    the TRAINING DATA doesn't represent
                             everyone the system will affect (Section 3)
  Measurement Bias:          the FEATURES/LABELS used are flawed
                               proxies for what's actually intended (Section 4)
─────────────────────────────────────────

2. Historical Bias in Training Data

Even perfectly accurate historical data can encode unfair patterns from society itself — a model trained on it will faithfully learn and reproduce those patterns.

# A hypothetical hiring model trained on a company's PAST hiring decisions
# If the company historically hired disproportionately from one demographic
# (for whatever historical reason — biased past decisions, unequal access
# to opportunity, etc.), a model trained to predict "who gets hired" will
# LEARN and REPRODUCE that same historical pattern, even with no
# discriminatory INTENT anywhere in the modeling process itself.
 
# This is precisely why "the data doesn't lie" is a dangerously incomplete
# defense — the data can accurately reflect a PAST that was itself unfair.

Real-World Example (Widely Reported)
─────────────────────────────────────────
  A major tech company scrapped an internal recruiting tool
  after discovering it had learned to penalize resumes
  containing the word "women's" (as in "women's chess club
  captain") — because the training data (past successful
  hires) skewed heavily male, a pattern from the tech
  industry's historical hiring, not something anyone
  intentionally coded.
─────────────────────────────────────────

3. Representation Bias

Occurs when the training data doesn't adequately cover all the groups the system will actually be used on.

# A simplified illustration: a facial recognition system trained on a
# dataset that is 80% one skin tone, 20% others
training_data_composition = {"light_skin": 0.80, "dark_skin": 0.20}
 
# The model has FAR more examples to learn from for the majority group —
# performance on the underrepresented group predictably suffers,
# not because of any explicit discriminatory rule, but because there
# simply wasn't enough data to learn those patterns as well

Well-Documented Real-World Finding
─────────────────────────────────────────
  Independent audits of several commercial facial recognition
  systems found dramatically higher error rates for women
  with darker skin tones compared to men with lighter skin
  tones — directly traceable to training datasets that
  underrepresented the former group.
─────────────────────────────────────────

This connects directly to the Data Science Notes' sampling chapter — representation bias is fundamentally a sampling problem: if your training data isn't a representative sample of the population the system will actually serve, performance will predictably differ across groups.

4. Measurement Bias

Occurs when the features or labels used as a proxy for what you actually care about are themselves systematically flawed for certain groups.

# A hypothetical example: using "arrests" as a proxy label for "crime committed"
# If policing intensity historically differed across neighborhoods
# (for reasons unrelated to actual crime rates), "arrests" becomes a
# BIASED proxy — a model trained to predict "arrests" will learn
# patterns reflecting POLICING PATTERNS, not the underlying reality
# it was actually intended to predict

The General Pattern
─────────────────────────────────────────
  What you WANT to measure:    "who is likely to commit a crime"
  What you ACTUALLY measure:      "who was arrested" (a FLAWED PROXY,
                                     shaped by historical policing
                                     patterns, not just the underlying
                                     reality)

  This gap between INTENDED measurement and ACTUAL measurement
  is measurement bias — and it's often much harder to spot than
  representation bias, since the data can look complete and
  well-sampled while still measuring the wrong thing.
─────────────────────────────────────────

5. Defining Fairness — Multiple, Conflicting Definitions

A common misconception: "fairness" has one clear, agreed-upon definition. In reality, there are several mathematically precise, individually reasonable definitions of fairness — that can directly conflict with each other.

# Illustrating with a simplified loan-approval scenario
 
def demographic_parity(approval_rates_by_group):
    """Fairness definition 1: equal APPROVAL RATES across groups."""
    rates = list(approval_rates_by_group.values())
    return max(rates) - min(rates) < 0.05      # roughly equal approval rates
 
def equalized_odds(true_positive_rates_by_group, false_positive_rates_by_group):
    """Fairness definition 2: equal TRUE POSITIVE and FALSE POSITIVE rates across groups."""
    tpr_gap = max(true_positive_rates_by_group.values()) - min(true_positive_rates_by_group.values())
    fpr_gap = max(false_positive_rates_by_group.values()) - min(false_positive_rates_by_group.values())
    return tpr_gap < 0.05 and fpr_gap < 0.05
 
def predictive_parity(precision_by_group):
    """Fairness definition 3: equal PRECISION (accuracy of positive predictions) across groups."""
    precisions = list(precision_by_group.values())
    return max(precisions) - min(precisions) < 0.05

6. The Impossibility of Satisfying All Fairness Definitions

A landmark, mathematically proven result: except in special cases (like a perfectly accurate model, or identical base rates across groups), it's mathematically impossible to satisfy demographic parity, equalized odds, and predictive parity simultaneously.

Why This Happens, Intuitively
─────────────────────────────────────────
  If two groups have DIFFERENT underlying base rates (e.g.
  different actual default rates on loans, for reasons
  unrelated to the model), then:

  - Making approval rates EQUAL (demographic parity)
    necessarily means the model treats otherwise-similar
    applicants from each group DIFFERENTLY based on group
    membership

  - Making error rates EQUAL (equalized odds) necessarily
    means approval RATES will differ across groups, since
    the underlying rates being predicted differ

  You literally CANNOT satisfy both when base rates differ —
  a mathematical fact, not a failure of effort or good intentions.
─────────────────────────────────────────

This is the single most important, most frequently misunderstood point in AI fairness discussions — choosing a fairness definition is a genuine value judgment about which trade-off is acceptable for a specific application, not a purely technical problem with one correct answer. A hiring model and a medical screening model might reasonably prioritize different fairness definitions given their different stakes and contexts.

7. Practical Mitigation Strategies

Interventions at Each Stage of the Pipeline
─────────────────────────────────────────
  Pre-processing:    fix the TRAINING DATA before modeling —
                        re-sampling underrepresented groups
                        (Data Science Notes' sampling techniques),
                        auditing for measurement bias in labels

  In-processing:        modify the LEARNING ALGORITHM itself to
                          explicitly optimize for a chosen fairness
                          definition alongside accuracy (covered
                          further in the Machine Learning Notes)

  Post-processing:         adjust the MODEL'S OUTPUTS after
                             training — e.g. different decision
                             thresholds per group to equalize a
                             chosen fairness metric
─────────────────────────────────────────

# A simplified post-processing example: different thresholds per group
# to achieve closer-to-equal true positive rates
 
def apply_group_specific_threshold(predicted_probability, group, thresholds):
    threshold = thresholds[group]
    return predicted_probability >= threshold
 
thresholds = {"group_a": 0.5, "group_b": 0.42}      # calibrated to equalize outcomes across groups

No mitigation technique is a complete, automatic fix — every intervention involves trade-offs (often between overall accuracy and a chosen fairness metric, and always requiring a deliberate choice about which fairness definition matters most for the specific application) that a responsible AI practitioner must make explicitly and document, not leave implicit.

8. Summary & Next Steps

Key Takeaways

• Bias enters AI systems through historical patterns in training data, underrepresentation of certain groups, or flawed proxy measurements — rarely through anyone's deliberate discriminatory intent.
• "The data doesn't lie" is a dangerously incomplete defense — historical data can be perfectly accurate while still reflecting a genuinely unfair past.
• Multiple, individually reasonable mathematical definitions of fairness exist (demographic parity, equalized odds, predictive parity) — and it's mathematically impossible to satisfy all of them simultaneously when groups have different base rates.
• Choosing which fairness definition to prioritize is a genuine value judgment specific to the application's context and stakes, not a purely technical decision with one universally correct answer.

Concept Check

1. Why can training data be perfectly accurate and still produce an unfair model?
2. What's the difference between representation bias and measurement bias?
3. Why is it mathematically impossible to satisfy demographic parity and equalized odds simultaneously when groups have different base rates?

Next Chapter

→ Chapter 2: Explainable AI (XAI)

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index