Why Generative Evaluation Is Hard

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Module 1, Chapter 1 Time to complete: ~15 minutes

Table of Contents

1. The Missing Denominator
2. Quality Is Several Things at Once
3. The Diversity Trap
4. What to Measure Instead
5. The Evaluation Ladder
6. Summary & Next Steps

1. The Missing Denominator

Every evaluation method from the Machine Learning Notes assumes a labelled test set: a known correct answer to compare against.

The Assumption That Breaks
─────────────────────────────────────────
  CLASSIFICATION
     prediction: "spam"      truth: "spam"     ✓
     Accuracy is COUNTABLE because "correct"
     is a single, known value.

  GENERATION
     output: "Introducing our lightest boot yet."
     truth:  ??????

     There are millions of acceptable product
     descriptions and no enumerable set of them.
─────────────────────────────────────────

The Consequence
─────────────────────────────────────────
  You cannot compute a score against ground truth.
  You must instead score against a DEFINITION of
  quality that YOU write down.

  Teams that skip writing that definition end up
  evaluating by vibes, and then cannot explain why
  they shipped one prompt over another.
─────────────────────────────────────────

2. Quality Is Several Things at Once

"Good output" is not one property. It decomposes into dimensions that trade against each other.

The Five Dimensions
─────────────────────────────────────────
  FIDELITY      Is it well-formed? Sharp image, fluent
                sentence, clean audio. The easiest to
                measure automatically.

  RELEVANCE     Does it match the request? A beautiful
                image of the wrong thing scores high on
                fidelity and zero on this.

  DIVERSITY     Across many samples, is there variety?
                Invisible in any single output — you
                can only see it in a population.

  CORRECTNESS   Are the factual claims true? Only
                applies to some tasks, and is where the
                real damage happens.

  SAFETY        Is it free of harmful, biased or
                infringing content? Non-negotiable and
                separately measured (Module 6).
─────────────────────────────────────────

Why They Must Be Measured Separately
─────────────────────────────────────────
  A single "quality score" hides the trade-offs.

  Raise guidance_scale (Module 3, Chapter 2):
     RELEVANCE goes UP
     DIVERSITY goes DOWN
     FIDELITY goes down past a threshold

  Averaged into one number, that change looks flat.
  Reported per dimension, it is an obvious trade you
  can make deliberately.
─────────────────────────────────────────

3. The Diversity Trap

The most common evaluation mistake, and the one that survives longest undetected.

How Mode Collapse Hides
─────────────────────────────────────────
  A model that produces ONE excellent output for every
  prompt scores PERFECTLY on any per-sample metric.

  Every image is sharp. Every sentence is fluent. Every
  reviewer rates each individual sample highly.

  And the product is useless, because a user asking for
  five options gets the same one five times.
─────────────────────────────────────────

The Rule
─────────────────────────────────────────
  Any metric computed on a SINGLE sample is blind to
  diversity.

  Diversity is a property of a POPULATION and must be
  measured over one — pairwise similarity across many
  samples, or a distribution-level metric like FID
  (Chapter 2).

  Always evaluate a BATCH, never one output.
─────────────────────────────────────────

import torch, itertools
 
@torch.no_grad()
def diversity(embeddings):
    """Mean pairwise DISTANCE across a batch. Low = mode collapse."""
    e = embeddings / embeddings.norm(dim=-1, keepdim=True)
    sims = [(e[i] @ e[j]).item() for i, j in itertools.combinations(range(len(e)), 2)]
    return 1 - sum(sims) / len(sims)          # 1 - mean cosine similarity
 
# Generate a BATCH from one prompt, varying only the seed:
batch = [generate(prompt, seed=s) for s in range(32)]
score = diversity(embed(batch))
 
if score < 0.15:                              # calibrate this against YOUR baseline
    print("WARNING: outputs are near-identical — likely mode collapse")

The key detail is the loop: one prompt, thirty-two seeds. Any function that takes a single output as its argument cannot compute this number, which is precisely why per-sample metrics are structurally blind to the problem.

This is not hypothetical. It is the standard failure mode of an over-trained LoRA (Module 3, Chapter 5), an over-guided diffusion sample, and a temperature-zero text pipeline.

4. What to Measure Instead

Since you cannot measure correctness against a reference, measure the things you can — and prefer measures tied to the actual job.

Four Substitutes, Best Last
─────────────────────────────────────────
  1. DISTRIBUTIONAL MATCH
     Do generated samples resemble real ones
     statistically? (FID, Chapter 2)
     Weakness: says nothing about any single output.

  2. CONDITION ADHERENCE
     Does the output match the prompt? (CLIPScore, or
     an LLM judge)
     Weakness: cannot see fidelity or diversity.

  3. HUMAN PREFERENCE
     Which of these two do people prefer?
     Weakness: slow, expensive, and needs a rubric to
     be repeatable.

  4. TASK OUTCOME
     Did it accomplish the actual job? Did the
     extracted invoice match the accounting system?
     Did the generated test catch the bug? Did the
     user keep the draft or discard it?

  ──► Prefer 4 wherever a task outcome exists. It is
      the only measure that cannot be gamed by
      improving something the user does not care about.
─────────────────────────────────────────

5. The Evaluation Ladder

Build these in order. Each rung is cheaper and faster than the one above it, and catches different failures.

Rung 1 — ASSERTIONS  (milliseconds, free)
─────────────────────────────────────────
  Deterministic checks that must never fail.
    - output parses against the schema (Module 2, Ch.3)
    - image is the requested dimensions
    - no banned terms present
    - response is within length bounds

  Run on EVERY output, in production. These are not
  really "evaluation" — they are a contract.

Rung 2 — AUTOMATIC METRICS  (seconds, cheap)
─────────────────────────────────────────
  CLIPScore, FID, WER, embedding similarity.
  Run on every commit, over a fixed sample set.
  Good for catching REGRESSION, poor at judging
  absolute quality.

Rung 3 — LLM-AS-JUDGE  (minutes, moderate)
─────────────────────────────────────────
  A model scores output against your written rubric.
  Run before every release, over 100-500 cases.
  Correlates reasonably with human judgement when the
  rubric is specific. Chapter 3.

Rung 4 — HUMAN REVIEW  (hours, expensive)
─────────────────────────────────────────
  The ground truth for your judge and your metrics.
  Run on a small set periodically, and always when
  the judge and the metrics disagree.

Rung 5 — PRODUCTION SIGNAL  (continuous, free)
─────────────────────────────────────────
  Regeneration rate, edit distance between the output
  and what the user shipped, thumbs, abandonment,
  A/B outcomes.

  The most honest measure you have, and the only one
  that reflects real users. Instrument for it from
  day one — Module 7, Chapter 1.
─────────────────────────────────────────

The Sequencing Advice
─────────────────────────────────────────
  Most teams start at rung 3 or 4 because it feels
  like "real" evaluation.

  Start at rung 1. Assertions catch the majority of
  production defects at zero cost, and every higher
  rung is wasted effort until the basics are enforced.
─────────────────────────────────────────

6. Summary & Next Steps

Key Takeaways

• Generative output has no enumerable ground truth, so evaluation scores against a definition of quality you write down rather than against labels.
• Quality decomposes into fidelity, relevance, diversity, correctness and safety; averaging them into one number hides the trade-offs you are actually making.
• Any single-sample metric is blind to mode collapse — always evaluate a batch, because diversity is a population property.
• Prefer task-outcome measures where they exist, and build the evaluation ladder from cheap assertions upward rather than starting with human review.

Concept Check

1. A model produces one excellent output per prompt, every time. Which metrics would miss the problem, and what would catch it?
2. Why does reporting a single averaged "quality score" make a guidance-scale change look like it did nothing?
3. Which rung of the ladder would you build first for a feature that returns JSON to a downstream system, and why?

Next Chapter

→ Chapter 2: Metrics by Modality

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index