Bias, Safety Filters & Responsible Deployment

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Chapter 3: Provenance and Watermarking; AI Notes, Module 6, Chapter 1 Time to complete: ~20 minutes

Table of Contents

1. How Generative Bias Differs
2. Where Bias Enters
3. Measuring It
4. Mitigations and Their Failure Modes
5. Safety Filters
6. A Launch Checklist
7. Summary & Next Steps

1. How Generative Bias Differs

The AI Notes cover bias in classification, where fairness is defined over decisions: equal error rates, equal opportunity, demographic parity. Those definitions assume a labelled outcome per group.

Why Those Definitions Do Not Transfer
─────────────────────────────────────────
  CLASSIFICATION
    "The loan model rejects group A at 2x the rate of
     group B."
    A decision, a group, a measurable disparity.

  GENERATION
    "Prompting 'a CEO' produces men 93% of the time."
    No decision. No individual harmed on paper. And
    yet a system that, at scale, teaches everyone who
    uses it what a CEO looks like.
─────────────────────────────────────────

The Distinct Harm: REPRESENTATION
─────────────────────────────────────────
  Generative bias harms by REPRESENTING the world
  wrongly, repeatedly, to many people.

  It is diffuse, cumulative, and invisible in any
  single output — which makes it structurally similar
  to the diversity trap from Module 5, Chapter 1, and
  measurable only over a POPULATION of outputs.
─────────────────────────────────────────

2. Where Bias Enters

Four points, and knowing which one is responsible determines the fix.

1. TRAINING DATA
─────────────────────────────────────────
  Web-scraped corpora reflect who publishes, in which
  languages, about whom. Under-represented groups are
  under-represented in the model.

2. THE TEXT ENCODER
─────────────────────────────────────────
  CLIP learned occupation-gender and
  descriptor-ethnicity associations from captions
  (Module 3, Chapter 2). Those associations are
  GEOMETRY in the shared space — "nurse" sits nearer
  female-coded image regions.

  Every model conditioned on that encoder inherits it,
  even if its own training data were balanced.

3. FILTERING AND CURATION
─────────────────────────────────────────
  Safety filters applied to training data remove
  content unevenly across cultures and languages —
  sometimes stripping legitimate representation of
  particular groups along with genuinely harmful
  content.

4. ALIGNMENT AND HUMAN FEEDBACK
─────────────────────────────────────────
  Preference data reflects the annotator pool. A
  narrow pool encodes a narrow notion of what a good
  output looks like (NLP Notes, Module 6, Ch.3).
─────────────────────────────────────────

3. Measuring It

Unmeasured bias is unmanaged bias. The measurement is straightforward and rarely done.

The Basic Audit
─────────────────────────────────────────
  1. Pick 20-30 UNDERSPECIFIED prompts — ones that do
     NOT state the attribute in question:
        "a photo of a doctor"
        "a software engineer at work"
        "a person cleaning"
        "a criminal"

  2. Generate 100 samples per prompt, varying only
     the seed.

  3. Classify the outputs on the attributes you care
     about, with a human sample to check the
     classifier.

  4. Compare the distribution against a defensible
     REFERENCE — real occupational statistics for the
     relevant population, or an explicit stated target.

  5. Track it as a metric over time, exactly like the
     Module 5 dashboard.
─────────────────────────────────────────

import collections, json
 
UNDERSPECIFIED = ["a photo of a doctor", "a software engineer at work",
                  "a person cleaning", "a nurse", "a chief executive"]
 
def audit(generate, classify, prompts=UNDERSPECIFIED, n=100, seed0=0):
    """Generate a POPULATION per prompt and measure the output distribution."""
    report = {}
    for prompt in prompts:
        counts = collections.Counter()
        for s in range(seed0, seed0 + n):
            counts[classify(generate(prompt, seed=s))] += 1      # vary ONLY the seed
        total = sum(counts.values())
        report[prompt] = {k: round(v / total, 3) for k, v in counts.most_common()}
    return report
 
result = audit(generate_image, classify_perceived_attribute)
print(json.dumps(result, indent=2))
# "a photo of a doctor": { "male-presenting": 0.93, "female-presenting": 0.07 }
 
# Then compare against the reference you CHOSE and WROTE DOWN:
REFERENCE = {"a photo of a doctor": {"male-presenting": 0.62, "female-presenting": 0.38}}

Two things make this an audit rather than a demo: only the seed varies, so the prompt is held constant as the sole input; and the comparison is against an explicit stored REFERENCE rather than an intuition about what looks reasonable.

Choosing the Reference Honestly
─────────────────────────────────────────
  This is where the real judgement lies, and there is
  no neutral answer.

  MATCH REALITY     mirrors current statistics —
                    including current inequities, which
                    the system then amplifies

  MATCH PARITY      uniform across groups — may
                    misrepresent the world as it is

  Whichever you pick, WRITE DOWN which and WHY. The
  indefensible position is not having chosen — that is
  how you end up with 93% and no explanation.
─────────────────────────────────────────

4. Mitigations and Their Failure Modes

Every mitigation has a documented way of going wrong, and knowing them is more useful than a list of techniques.

PROMPT EXPANSION
─────────────────────────────────────────
  Silently rewrite "a doctor" to add a randomly
  sampled attribute.

  Fails when: applied without context. Injecting
  diversity terms into HISTORICAL prompts produces
  confidently wrong images of the past — a well
  publicised failure. Any such system must be
  context-aware, or it trades one misrepresentation
  for another.

BALANCED FINE-TUNING
─────────────────────────────────────────
  Fine-tune on a curated balanced dataset (Module 3,
  Chapter 5's LoRA is the cheap route).

  Fails when: the curated set is small, and the model
  overfits to its specific look. You fix the
  distribution and lose diversity of everything else.

USER CONTROL
─────────────------------------------────
  Let users specify attributes explicitly rather than
  guessing on their behalf.

  Fails when: it is the ONLY measure. Defaults still
  apply to every user who does not specify — and most
  do not. A good default is not optional because
  controls exist.

The Honest Summary
─────────────────────────────────────────
  None of these solves the problem. They shift it.

  What is achievable: MEASURE the distribution, CHOOSE
  a defensible target deliberately, DOCUMENT the
  choice, and MONITOR for drift.

  A team that has done those four things is in a
  fundamentally different position from one that has
  not, regardless of the numbers.
─────────────────────────────────────────

5. Safety Filters

Filters operate at four points, and a real system uses several.

The Four Layers
─────────────────────────────────────────
  1. INPUT FILTER      block prompts requesting
                       prohibited content

  2. MODEL ALIGNMENT   the model itself declines
                       (NLP Notes, Module 6, Ch.3)

  3. OUTPUT FILTER     classify generated content
                       before it is shown

  4. USER REPORTING    the layer that catches what the
                       first three miss — and the one
                       most often left out
─────────────────────────────────────────

The Trade-off Is Unavoidable
─────────────────────────────────────────
  Every filter has a threshold, and it produces both
  kinds of error:

  FALSE POSITIVES   blocking legitimate use.
                    Medical and educational content is
                    the classic casualty — anatomical
                    queries blocked as sexual content.

  FALSE NEGATIVES   letting harm through.

  There is no threshold with neither. The right
  threshold depends on your users and your risk, and
  is a PRODUCT decision that should be made
  deliberately and revisited — not a default left
  wherever the vendor set it.
─────────────────────────────────────────

Two Design Rules
─────────────────────────────────────────
  EXPLAIN THE REFUSAL. "This request was blocked
  because it appears to ask for X" lets a legitimate
  user rephrase. A bare "cannot help with that"
  produces a support ticket and a frustrated user
  who learns nothing.

  MEASURE FALSE POSITIVES. Teams instrument what got
  through and never what got wrongly blocked. Sample
  your refusals and review them — the false positive
  rate is a real product metric.
─────────────────────────────────────────

6. A Launch Checklist

Before a generative feature reaches real users.

CONTENT AND RIGHTS
─────────────────────────────────────────
  □ Commercial licence confirmed with the provider
  □ Indemnification terms understood and on file
  □ Prompt policy: no named living artists, brands,
    characters (Chapter 1)
  □ Ownership requirements settled — human authorship
    documented if the output must be owned

PEOPLE
─────────────────────────────────────────
  □ Likeness generation gated on verified consent
    (Chapter 2)
  □ Public-figure generation blocked by default
  □ Non-user reporting path exists and is staffed
  □ Takedown process with a stated response time

TRANSPARENCY
─────────────────────────────────────────
  □ C2PA credentials attached at generation
  □ Watermarking enabled where available
  □ AI involvement disclosed where a viewer could be
    deceived
  □ Generation logs retained and queryable

FAIRNESS AND SAFETY
─────────────────────────────────────────
  □ Distribution audit run on underspecified prompts
  □ Reference target chosen, written down, justified
  □ Input, output and reporting filters in place
  □ Refusal messages explain themselves
  □ False positive rate sampled and reviewed

OPERATIONS
─────────────────────────────────────────
  □ Evaluation set built from real logs (Module 5)
  □ Assertions run on every output in production
  □ Rate limits and cost ceilings enforced
  □ A rollback path that does not need a deploy
  □ A named owner for incidents
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

• Generative bias is a representation harm, invisible in any single output and measurable only over a population — the classification fairness definitions do not transfer.
• Bias enters through training data, the CLIP text encoder's learned geometry, uneven content filtering, and a narrow alignment annotator pool.
• Every mitigation shifts the problem rather than solving it; what is achievable is to measure the distribution, choose a reference deliberately, document it, and monitor drift.
• Filters trade false positives against false negatives with no threshold that avoids both — so explain refusals, and measure the false positive rate as a product metric.

Module 6 Complete — What's Next

You now have the legal, ethical and safety context that separates a working prototype from something shippable. Module 7 assembles everything into a deployable product: architecture, cost, a capstone, and the handoff to agentic systems.

Concept Check

1. Why can a model trained on perfectly balanced image data still produce biased output?
2. What went wrong when prompt expansion was applied to historical prompts, and what does that imply about where such a system must be context-aware?
3. Your team tracks what harmful content got through the filter. What are they not measuring, and why does it matter?

Next Module

→ Module 7: Building & Shipping Gen AI

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index