Evaluation Production And Capstone
Evaluating LLMs: Benchmarks, Human Eval, LLM-as-Judge
Chapter 1's metrics were designed for narrower tasks (translation, summarization) with fairly well-defined reference outputs. Modern LLMs handle open-ended conv
JrCodex·7 min read
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 1: Evaluating Classical NLP — BLEU, ROUGE, Perplexity Time to complete: ~25 minutes
Table of Contents
- Why LLM Evaluation Is Even Harder
- Standardized Benchmarks
- The Benchmark Contamination Problem
- Human Evaluation
- LLM-as-Judge
- Known Biases in LLM-as-Judge
- Evaluating RAG and Agent Systems Specifically
- Summary & Next Steps
1. Why LLM Evaluation Is Even Harder
Chapter 1's metrics were designed for narrower tasks (translation, summarization) with fairly well-defined reference outputs. Modern LLMs handle open-ended conversation, reasoning, coding, and creative writing — tasks where "correctness" is often multi-dimensional (helpfulness, accuracy, safety, tone) and genuinely hard to reduce to a single automated score.
2. Standardized Benchmarks
Benchmarks are large, standardized test sets designed to measure specific capabilities, allowing consistent comparison across different models.
Categories of Common Benchmarks
─────────────────────────────────────────
Knowledge/reasoning: broad, multiple-choice question
sets spanning many academic
subjects — testing factual
knowledge and reasoning together
Coding: programming problems, evaluated
by whether generated code
actually PASSES test cases
(an OBJECTIVE, automatically
checkable metric — genuinely
more reliable than Chapter 1's
overlap-based metrics)
Math: multi-step math problems,
testing chain-of-thought-
style reasoning (Module 7,
Ch.2) specifically
Instruction-following testing whether a model
and safety: correctly follows
instructions, avoids
harmful outputs, and
behaves as
INTENDED (Module 6,
Ch.3's alignment goals)
─────────────────────────────────────────
# A simplified illustration of running a model against a benchmark-style task
def evaluate_on_math_benchmark(model, tokenizer, problems):
correct = 0
for problem in problems:
prompt = f"{problem['question']}\nLet's think step by step." # Module 7, Ch.2's CoT
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(output[0])
predicted_answer = extract_final_number(response) # task-specific parsing
if predicted_answer == problem["answer"]:
correct += 1
accuracy = correct / len(problems)
return accuracy3. The Benchmark Contamination Problem
A genuine, well-documented issue: if benchmark questions (or very similar ones) appeared in a model's pretraining data (Module 3, Chapter 3's massive-scale web scraping), the model may simply be recalling an answer it memorized, rather than genuinely reasoning to it.
Why This Undermines Benchmark Validity
─────────────────────────────────────────
This directly echoes the ML NOTES' warning about DATA
LEAKAGE (Module 2's imbalanced data chapter and the general
train/test discipline, ML Notes Module 1, Ch.5) — if TEST
data effectively "leaked" into TRAINING data, performance on
that test no longer reliably measures GENUINE generalization
ability, just MEMORIZATION of that specific test set.
─────────────────────────────────────────
Mitigations
─────────────────────────────────────────
- Using FRESH, recently-created benchmark questions (created
AFTER a model's known training cutoff)
- Checking for VERBATIM or near-verbatim benchmark text
appearing in PUBLICLY available training data sources
- Preferring DYNAMIC or PROCEDURALLY GENERATED test problems
that can't have been memorized VERBATIM
─────────────────────────────────────────
4. Human Evaluation
For genuinely subjective qualities — helpfulness, tone, creativity — human evaluation remains the gold standard, directly connecting to Module 6, Chapter 3's RLHF preference data.
Common Human Evaluation Setups
─────────────────────────────────────────
Pairwise comparison: show TWO models' responses to the
SAME prompt, ask a human which
is BETTER (directly the SAME
format as Module 6, Chapter 3,
Section 5's reward model training
data)
Likert-scale ratings: rate a SINGLE response on a
numeric scale (e.g. 1-5) for
specific qualities (helpfulness,
accuracy, safety)
Red-teaming: humans DELIBERATELY try to
elicit harmful,
incorrect, or otherwise
problematic outputs —
testing WORST-CASE
behavior, not average
performance
─────────────────────────────────────────
The Practical Cost of Human Evaluation
─────────────────────────────────────────
Human evaluation is SLOW and EXPENSIVE relative to
automated metrics (Chapter 1, Section 5's LLM-as-judge) —
it doesn't scale to evaluating THOUSANDS of model outputs
during rapid iteration, but remains ESSENTIAL for final,
high-stakes validation before deployment (Chapter 3).
─────────────────────────────────────────
5. LLM-as-Judge
LLM-as-Judge uses a SEPARATE, typically more capable LLM to evaluate outputs automatically — a practical middle ground between Chapter 1's cheap-but-limited automated metrics and Section 4's expensive-but-thorough human evaluation.
def llm_judge_comparison(client, prompt, response_a, response_b):
judge_prompt = f"""You are an impartial judge evaluating two AI responses.
Prompt given to both models: {prompt}
Response A: {response_a}
Response B: {response_b}
Which response is more helpful, accurate, and well-written? Respond with
ONLY "A", "B", or "TIE", followed by a brief explanation."""
response = client.chat.completions.create(
model="gpt-4", messages=[{"role": "user", "content": judge_prompt}],
)
return response.choices[0].message.content
result = llm_judge_comparison(
client,
prompt="Explain photosynthesis simply.",
response_a="Plants use sunlight to make food from water and CO2.",
response_b="Photosynthesis is a biochemical process involving chlorophyll.",
)
print(result)Why This Scales Better Than Human Evaluation
─────────────────────────────────────────
An LLM judge can evaluate THOUSANDS of response pairs
quickly and CHEAPLY compared to human annotators (Section
4) — making it practical to run DURING rapid development
iteration (Module 6, Ch.4's fine-tuning recipe), reserving
EXPENSIVE human evaluation for FINAL validation before
deployment.
─────────────────────────────────────────
6. Known Biases in LLM-as-Judge
Documented Biases to Watch For
─────────────────────────────────────────
Position bias: judges tend to slightly FAVOR whichever
response appears FIRST in the prompt —
mitigate by evaluating BOTH orderings
and averaging
Length bias: judges tend to favor LONGER responses,
even when NOT genuinely better —
a documented, measurable effect
worth explicitly correcting for
Self-preference bias: a model used as JUDGE tends to
rate ITS OWN outputs (or
outputs from similar models)
more favorably than a
genuinely NEUTRAL evaluation
would
─────────────────────────────────────────
Mitigation: Combine With Chapter 1-4's Other Methods
─────────────────────────────────────────
Given these KNOWN biases, LLM-as-judge results should be
spot-checked against HUMAN evaluation (Section 4)
periodically, rather than trusted as a COMPLETE replacement
— directly echoing this curriculum's repeated theme (Module
4, Chapter 2's NSP lesson) that plausible-sounding automated
signals must be EMPIRICALLY validated, not simply assumed
reliable.
─────────────────────────────────────────
7. Evaluating RAG and Agent Systems Specifically
Module 8's RAG and agent systems need additional, system-specific evaluation dimensions beyond a raw LLM's output quality.
RAG-Specific Metrics
─────────────────────────────────────────
Retrieval precision/recall: of the chunks RETRIEVED
(Module 8, Ch.2), how many
were ACTUALLY relevant?
(directly the ML Notes'
precision/recall framework,
Module 4, Ch.6, applied to
retrieval)
Faithfulness/groundedness: does the GENERATED answer
ACTUALLY follow from the
retrieved context, or
does it hallucinate
beyond it (Module 8,
Ch.1, Sec.3)?
Answer relevance: does the final answer
actually ADDRESS the
user's original
question?
─────────────────────────────────────────
Agent-Specific Metrics
─────────────────────────────────────────
Task completion rate: did the agent (Module 8, Ch.4)
successfully complete the
REQUESTED task, end to end?
Tool selection accuracy: did the agent choose the
CORRECT tool at each step
(Module 8, Ch.4, Sec.4)?
Efficiency (steps taken): how many agent LOOP
iterations (Module 8,
Ch.4, Sec.2) were needed
— fewer, successful
steps generally indicate
a more RELIABLE agent
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Standardized benchmarks enable consistent model comparison, but suffer from a genuine contamination risk if test data leaked into training data — directly echoing the ML Notes' data leakage warnings.
- Human evaluation (pairwise comparison, Likert ratings, red-teaming) remains the gold standard for subjective qualities, but is slow and expensive relative to automated methods.
- LLM-as-judge offers a scalable middle ground between cheap automated metrics and expensive human evaluation, but carries documented biases (position, length, self-preference) requiring periodic validation against human judgment.
- RAG and agent systems need additional, system-specific metrics beyond raw output quality: retrieval precision/recall, faithfulness to retrieved context, and task completion rate.
Concept Check
- Why does benchmark contamination undermine the validity of a model's reported benchmark performance?
- Name the three documented biases in LLM-as-judge evaluation covered in this chapter.
- Why does evaluating a RAG system require metrics beyond just judging the final generated answer's quality?
Next Chapter
→ Chapter 3: LLMs in Production
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index