Evaluating Generative Output
Human and LLM-as-Judge Evaluation
Before any judge, you need cases to judge. This is the step teams skip, and skipping it makes every later number meaningless.
JrCodex·8 min read
Jr Codex Generative AI Notes
Level: Advanced Prerequisites: Chapter 2: Metrics by Modality; NLP Notes, Module 9, Chapter 2 Time to complete: ~25 minutes
Table of Contents
- Building the Evaluation Set First
- Writing a Rubric That Works
- Pairwise Beats Absolute Scoring
- Implementing an LLM Judge
- Judge Biases and Their Fixes
- Calibrating the Judge Against Humans
- Summary & Next Steps
1. Building the Evaluation Set First
Before any judge, you need cases to judge. This is the step teams skip, and skipping it makes every later number meaningless.
What Goes In
─────────────────────────────────────────
TYPICAL 40% the ordinary requests that make
up most real traffic
EDGE 30% unusual but legitimate — very
long inputs, empty fields, mixed
languages, ambiguous requests
ADVERSARIAL 20% prompt injection attempts,
out-of-scope questions, requests
the system should REFUSE
REGRESSION 10% every bug you have ever fixed,
preserved forever so it cannot
come back
─────────────────────────────────────────
Size and Sourcing
─────────────────────────────────────────
START at 50 cases. That is enough to catch obvious
regressions and small enough to build in an
afternoon. Grow to 200-500 as the product matures.
SOURCE from real usage logs, not your imagination.
Your invented cases reflect what you expect users to
do. Logs reflect what they actually do, and the gap
between those is where the failures live.
The REGRESSION bucket is the highest-value one and
it grows for free — every production bug becomes a
permanent test case.
─────────────────────────────────────────
2. Writing a Rubric That Works
A rubric turns "is this good?" into questions with defensible answers. The difference between a useful and useless judge is almost entirely rubric quality.
Bad Rubric
─────────────────────────────────────────
"Rate the response quality from 1 to 10."
Two evaluators will not agree. The same evaluator
will not agree with themselves next week. A 7 means
nothing.
Good Rubric
─────────────────────────────────────────
Score each dimension independently.
GROUNDEDNESS (0-2)
0 — contains a claim absent from the sources
1 — all claims supported, but omits a key fact
the sources contain
2 — all claims supported and complete
RELEVANCE (0-2)
0 — does not address the question asked
1 — addresses it partially, or with padding
2 — directly and completely addresses it
TONE (0-1)
0 — hedging, apologetic, or over-familiar
1 — direct and professional
─────────────────────────────────────────
The Three Properties of a Usable Rubric
─────────────────────────────────────────
1. SEPARATE DIMENSIONS — never one blended score
(Chapter 1's point about hiding trade-offs)
2. FEW LEVELS — 0-2 or 0-3. Humans and judges cannot
reliably distinguish ten levels of anything
3. OBSERVABLE CRITERIA — each level describes
something you can POINT AT in the output, not a
feeling about it
─────────────────────────────────────────
3. Pairwise Beats Absolute Scoring
Given a choice, ask "which of these two is better?" rather than "how good is this?"
Why
─────────────────────────────────────────
ABSOLUTE requires an internalised, stable standard
of what a 7 is. Judges drift over a
session; different judges hold different
standards; scores cluster at 7-8 and stop
discriminating.
PAIRWISE requires only a COMPARISON, which humans
and models both do far more reliably.
Scores cannot drift, because there is no
scale to drift on.
─────────────────────────────────────────
Turning Comparisons into a Ranking
─────────────────────────────────────────
Run pairwise comparisons between variants and
aggregate with an Elo or Bradley-Terry rating —
the same method used for chess rankings and for
public LLM leaderboards.
Practical benefit: adding a new variant costs only
its matches against existing ones, not a re-scoring
of everything.
─────────────────────────────────────────
Use absolute scoring when you need a threshold ("groundedness must be 2 to ship"). Use pairwise when you need a decision between options.
4. Implementing an LLM Judge
from pydantic import BaseModel
from typing import Literal
class Judgement(BaseModel):
groundedness_reason: str # REASONING FIRST — see the note below
groundedness: Literal[0, 1, 2]
relevance_reason: str
relevance: Literal[0, 1, 2]
tone: Literal[0, 1]
JUDGE_PROMPT = """You are evaluating a support answer against its source documents.
SOURCES:
{sources}
QUESTION:
{question}
ANSWER:
{answer}
Score each dimension using ONLY these definitions:
GROUNDEDNESS
0 - contains a claim not present in the sources
1 - all claims supported, but omits a key fact from the sources
2 - all claims supported and complete
RELEVANCE
0 - does not address the question
1 - partially addresses it, or pads with irrelevant content
2 - directly and completely addresses it
TONE
0 - hedging, apologetic, or over-familiar
1 - direct and professional
For each dimension give your reason BEFORE the score, quoting the specific
text that determined it."""
def judge(client, sources, question, answer):
return client.chat.completions.parse(
model="gpt-4o", # use a STRONGER model to judge than to generate
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
sources=sources, question=question, answer=answer)}],
response_format=Judgement,
temperature=0,
).choices[0].message.parsedTwo Design Choices Doing Real Work
─────────────────────────────────────────
REASON BEFORE SCORE
The schema puts *_reason before its score, so the
model generates the justification FIRST. Because
generation is autoregressive (Module 1, Ch.2), the
score is then conditioned on the reasoning rather
than rationalised after it. This measurably
improves judge agreement with humans.
QUOTE THE EVIDENCE
Requiring a quote makes the judgement auditable.
When you disagree with a score you can see exactly
what the judge was looking at — which is how you
debug the RUBRIC rather than the model.
─────────────────────────────────────────
5. Judge Biases and Their Fixes
LLM judges have systematic, well-documented biases. Each has a mechanical fix, and applying them is what separates a judge you can trust from one you cannot.
POSITION BIAS
─────────────────────────────────────────
In pairwise comparison, judges favour whichever
answer is presented FIRST — measurably, and by a
wide margin.
Fix: run every comparison BOTH ways and average. If
the two orderings disagree, record it as a TIE.
Disagreement rate is itself a useful signal: a high
rate means your rubric is not discriminating.
LENGTH BIAS
─────────────────────────────────────────
Judges prefer longer answers, independent of
quality — the strongest and most persistent bias.
Fix: state in the rubric that length is not a
virtue, penalise padding explicitly (as the
RELEVANCE dimension above does), and monitor the
correlation between score and word count. If it is
strong, your judge is measuring length.
SELF-PREFERENCE BIAS
─────────────────────────────────────────
Models rate their own family's output higher.
Fix: judge with a model from a DIFFERENT provider
than the one generating, especially when comparing
candidate models.
LENIENCY
─────────────────────────────────────────
Absolute scores cluster high; judges rarely use the
bottom of a scale.
Fix: prefer pairwise (Section 3), and use short
scales where the top is genuinely hard to reach.
─────────────────────────────────────────
def unbiased_compare(client, question, answer_a, answer_b):
first = compare(client, question, answer_a, answer_b) # A then B
second = compare(client, question, answer_b, answer_a) # B then A
if first == "A" and second == "B": # both picked the same TEXT
return "A"
if first == "B" and second == "A":
return "B"
return "TIE" # order flipped the verdict6. Calibrating the Judge Against Humans
An LLM judge is a proxy. A proxy you have never checked is an assumption.
The Calibration Loop
─────────────────────────────────────────
1. Have humans score 50 cases with the SAME rubric.
2. Have the judge score the same 50.
3. Measure agreement (Cohen's kappa, or plain
percentage agreement per dimension).
4. Read every DISAGREEMENT individually.
5. Fix whichever is wrong — usually the RUBRIC.
6. Repeat until agreement is acceptable.
Then: re-run this on a fresh sample each quarter,
and whenever you change the judge model.
─────────────────────────────────────────
Reading the Result
─────────────────────────────────────────
Agreement > 80% the judge can replace humans for
routine runs
60-80% usable for detecting REGRESSIONS
(relative change), not for
absolute quality claims
< 60% the rubric is ambiguous. Fix the
rubric — do NOT switch judge
models and hope.
─────────────────────────────────────────
The Insight Worth Keeping
─────────────────────────────────────────
Most judge disagreement traces to the RUBRIC, not to
the model. If two careful humans also disagree on a
case, no judge will resolve it — the definition of
quality was never pinned down.
Calibration is therefore mostly a process for
discovering what you actually mean by "good".
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Build the evaluation set before the judge, sourced from real logs, with typical, edge, adversarial and regression cases — and let the regression bucket grow with every fixed bug.
- A usable rubric has separate dimensions, few levels, and observable criteria; a single 1–10 quality score is not reproducible.
- Pairwise comparison is more reliable than absolute scoring because it needs no stable internal standard; use absolute scores only for ship/no-ship thresholds.
- Judges have position, length and self-preference biases with mechanical fixes; calibrate against humans, and treat disagreement as evidence the rubric is unclear.
Module 5 Complete — What's Next
You can now measure generative output rather than guess at it. Module 6 turns to a set of questions no metric answers: whether you have the right to generate a thing at all, and what you owe the people affected by it.
Concept Check
- Why does putting the reasoning field before the score in the schema improve agreement with human judges?
- Your judge's scores correlate strongly with answer word count. What is happening, and what are two fixes?
- Human–judge agreement sits at 55%. Why is swapping to a stronger judge model the wrong first move?
Next Module
→ Module 6: Ethics, Law & Trust
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index