Metrics by Modality

Jr Codex Generative AI Notes

Level: Intermediate–Advanced Prerequisites: Chapter 1: Why Generative Evaluation Is Hard Time to complete: ~20 minutes

Reading a Metric Correctly
Image Metrics — FID
Image Metrics — CLIPScore
Audio Metrics
Text Metrics, Briefly
A Metric Dashboard
Summary & Next Steps

1. Reading a Metric Correctly

Every metric in this chapter answers one narrow question. Most misuse comes from reading an answer to a question the metric was not asked.

Three Questions Before Trusting Any Number
─────────────────────────────────────────
  1. What EXACTLY does it compare?
  2. Is it computed per-sample, or over a population?
  3. Is the absolute value meaningful, or only the
     direction of change?

  For nearly every metric here, the honest answer to
  (3) is: only the direction, and only against your
  own baseline.
─────────────────────────────────────────

2. Image Metrics — FID

Fréchet Inception Distance measures how closely the distribution of generated images matches the distribution of real ones.

How It Works
─────────────────────────────────────────
  1. Pass real images and generated images through a
     pretrained Inception network (DL Notes, Mod.4),
     taking the 2048-dim feature vector for each.

  2. Model each set as a multivariate Gaussian —
     compute its mean and covariance.

  3. FID = the Fréchet distance between the two
     Gaussians.

  Lower is better. 0 means the two distributions'
  first two moments are identical.
─────────────────────────────────────────

Why It Is the Standard Image Metric
─────────────────────────────────────────
  It is the one common metric that sees DIVERSITY.

  A mode-collapsed model produces excellent individual
  images whose distribution is far too NARROW — small
  covariance — and FID punishes that heavily.

  This directly addresses Chapter 1's diversity trap.
─────────────────────────────────────────

from torchmetrics.image.fid import FrechetInceptionDistance
import torch
 
fid = FrechetInceptionDistance(feature=2048)
fid.update(real_images_uint8, real=True)          # tensors: (N, 3, H, W), uint8
fid.update(generated_images_uint8, real=False)
print(f"FID: {fid.compute():.2f}")

The Caveats That Matter
─────────────────────────────────────────
  SAMPLE SIZE   FID is biased at small N. Below ~2,000
                images the number is dominated by
                sampling noise. 10,000 is the
                convention. NEVER compare FIDs computed
                at different N.

  NOT COMPARABLE ACROSS PAPERS
                FID depends on the Inception weights,
                the resize method, and the reference
                set. Only compare numbers you computed
                identically yourself.

  BLIND TO PROMPTS
                FID never sees the prompt. A model that
                generates beautiful, diverse images of
                entirely the WRONG SUBJECT can score
                excellently.
─────────────────────────────────────────

That last caveat is why FID is never used alone.

3. Image Metrics — CLIPScore

CLIPScore covers exactly FID's blind spot: does the image match the prompt?

How It Works
─────────────────────────────────────────
  Encode the prompt with CLIP's text encoder.
  Encode the image with CLIP's image encoder.
  Take the cosine similarity between them.

  This is Module 3, Chapter 2's shared embedding space
  used as a MEASURING device instead of a steering one.

  Higher is better. Typical range for on-prompt
  images: 0.25 - 0.40.
─────────────────────────────────────────

import torch
from transformers import CLIPModel, CLIPProcessor
 
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
 
inputs = proc(text=[prompt], images=image, return_tensors="pt", padding=True)
with torch.no_grad():
    out = model(**inputs)
    img_e = out.image_embeds / out.image_embeds.norm(dim=-1, keepdim=True)
    txt_e = out.text_embeds / out.text_embeds.norm(dim=-1, keepdim=True)
    print(f"CLIPScore: {(img_e @ txt_e.T).item():.3f}")

The Circularity Warning
─────────────────────────────────────────
  Stable Diffusion is CONDITIONED on CLIP.
  CLIPScore MEASURES with CLIP.

  So CLIPScore rewards exactly what the model was
  optimised to produce, and shares every one of CLIP's
  blind spots — it cannot count either, and it will
  happily score "no cars" highly for an image full of
  cars (Module 3, Chapter 2).

  Use it to compare PROMPTS or SETTINGS on one model.
  Do not use it as an independent measure of truth.
─────────────────────────────────────────

FID and CLIPScore Together
─────────────────────────────────────────
  Raising guidance_scale moves them in OPPOSITE
  directions:

     CLIPScore ▲   (more on-prompt)
     FID       ▲   (worse — less diverse, more
                    saturated, further from the real
                    image distribution)

  Plotting both against guidance scale gives you a
  trade-off curve and a defensible choice of setting.
  This one plot is the most useful thing in this
  chapter.
─────────────────────────────────────────

4. Audio Metrics

Word Error Rate — for TRANSCRIPTION
─────────────────────────────────────────
  WER = (Substitutions + Insertions + Deletions)
        ────────────────────────────────────────
                 words in the reference

  Lower is better. This is a genuine, unambiguous
  accuracy metric — because transcription DOES have a
  ground truth, unlike most generation.

  <  5%   excellent, near-human
  5-10%   usable with review
  > 20%   unusable for most purposes

  Normalise before comparing: case, punctuation and
  numbers-vs-words will otherwise dominate the score.

For GENERATED audio (TTS, music)
─────────────────────────────────────────
  FAD (Fréchet Audio Distance)  the FID idea applied
                                to audio embeddings.
                                Same strengths, same
                                caveats.

  ROUND-TRIP WER                transcribe generated
                                speech with Whisper and
                                compare to the input
                                script. Cheap, and an
                                excellent automatic
                                proxy for
                                INTELLIGIBILITY.

  MOS (Mean Opinion Score)      humans rate naturalness
                                1-5. Still the only
                                trustworthy measure of
                                whether TTS sounds good.
─────────────────────────────────────────

Round-trip WER deserves emphasis: it is the rare automatic metric that is both cheap and genuinely meaningful, because a listener who cannot make out the words does not care how natural the voice is.

5. Text Metrics, Briefly

Covered in full in the NLP Notes, Module 9, Chapter 1. The summary for this curriculum:

Metric	Measures	Use it for	Not for
BLEU / ROUGE	N-gram overlap with a reference	Translation, extractive summarisation	Anything open-ended — it penalises correct paraphrase
Perplexity	How surprised the model is by text	Comparing base models on the same corpus	Comparing across tokenizers, or judging output quality
BERTScore	Embedding similarity to a reference	Semantic match where wording may vary	When there is no reference at all
LLM-as-judge	A rubric you wrote	Almost everything else	When you have not written the rubric (Chapter 3)

The Practical Reality
─────────────────────────────────────────
  For open-ended generation, the n-gram metrics are
  near-useless: a paraphrase that is BETTER than the
  reference scores WORSE.

  For most real text tasks the choice is between an
  assertion (rung 1) and an LLM judge (rung 3). The
  middle rung is thin for text.
─────────────────────────────────────────

6. A Metric Dashboard

What to actually track, per modality, and what each is for.

Images
─────────────────────────────────────────
  FID (n ≥ 2000)      diversity + realism, vs baseline
  CLIPScore           prompt adherence
  assertion pass %    dimensions, format, NSFW filter
  human pairwise      the tiebreaker (Chapter 3)

Text
─────────────────────────────────────────
  schema pass %       the contract (Module 2, Ch.3)
  judge score         per rubric dimension
  groundedness %      claims supported by sources
  p95 latency, cost   per request

Speech
─────────────────────────────────────────
  WER                 transcription accuracy
  round-trip WER      TTS intelligibility
  MOS                 naturalness, sampled

The Rule for All of Them
─────────────────────────────────────────
  Track the DELTA against a frozen baseline, on a
  FIXED prompt set, at a FIXED sample size.

  An absolute FID of 18 means nothing. "FID moved from
  18 to 24 after this change, on the same 5,000
  prompts" means everything.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

FID compares distributions and is the standard image metric precisely because it detects mode collapse; it needs thousands of samples and never sees the prompt.
CLIPScore measures prompt adherence using the same encoder that steered generation, so it is useful for comparing settings but is not an independent judge.
FID and CLIPScore move in opposite directions as guidance rises; plotting both gives a defensible trade-off curve.
WER is a true accuracy metric because transcription has ground truth; round-trip WER is the cheapest meaningful measure of generated speech.

Concept Check

A model produces gorgeous, varied images of the wrong subject. Which metric looks fine, which catches it, and why?
Why is comparing your FID of 22 against a published FID of 9 meaningless?
What makes round-trip WER a better first automatic metric for a TTS system than any measure of audio fidelity?

Next Chapter

→ Chapter 3: Human and LLM-as-Judge Evaluation

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index