Evaluation Production And Capstone
Evaluating Classical NLP: BLEU, ROUGE, Perplexity
The ML Notes' classification metrics (precision, recall, F1) work perfectly for classification tasks (Module 1, Chapter 3; Module 4) — there's a single correct
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Module 8: RAG & Agents Time to complete: ~25 minutes
Table of Contents
- Why NLP Evaluation Needs Its Own Toolkit
- Perplexity — Measuring Language Model Quality
- BLEU — Evaluating Translation
- ROUGE — Evaluating Summarization
- The Shared Limitation of N-Gram Metrics
- Embedding-Based Metrics
- Summary & Next Steps
1. Why NLP Evaluation Needs Its Own Toolkit
The ML Notes' classification metrics (precision, recall, F1) work perfectly for classification tasks (Module 1, Chapter 3; Module 4) — there's a single correct label, and either the prediction matches it or it doesn't. Generation tasks (translation, summarization, open-ended text) have no single "correct" output — many different phrasings can all be equally valid — requiring fundamentally different evaluation approaches.
2. Perplexity — Measuring Language Model Quality
Perplexity measures how "surprised" a language model is by a piece of held-out text — directly derived from the Module 3, Chapter 3's CLM training loss.
The Formula
─────────────────────────────────────────
Perplexity = exp( average cross-entropy loss, DL Notes
Module 2 Ch.3, per token )
LOWER perplexity = the model assigned HIGHER probability to
the ACTUAL next tokens in the text — meaning it "predicted"
the real text well; HIGHER perplexity = the model was
frequently "surprised," assigning LOW probability to what
actually came next
─────────────────────────────────────────
import torch
import torch.nn.functional as F
def calculate_perplexity(model, tokenizer, text):
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"]
with torch.no_grad():
outputs = model(input_ids, labels=input_ids) # Module 5, Ch.1's CLM loss, computed directly
perplexity = torch.exp(outputs.loss)
return perplexity.item()
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
coherent_text = "The sun rose over the mountains, casting long shadows across the valley."
random_text = "Purple elephant quickly banana yesterday computer sadness."
print(f"Coherent text perplexity: {calculate_perplexity(model, tokenizer, coherent_text):.2f}") # LOWER
print(f"Random text perplexity: {calculate_perplexity(model, tokenizer, random_text):.2f}") # HIGHERWhat Perplexity Is (and Isn't) Good For
─────────────────────────────────────────
GOOD for: comparing DIFFERENT language models' RAW
predictive quality on the SAME text, or
tracking a SINGLE model's improvement during
pretraining (Module 3, Ch.3)
NOT good for: judging whether a SPECIFIC generated
response is HELPFUL, CORRECT, or
preferred by humans — a fluent,
LOW-perplexity response can still be
factually wrong or unhelpful (directly
echoing Module 8, Chapter 1's
hallucination discussion)
─────────────────────────────────────────
3. BLEU — Evaluating Translation
BLEU (Bilingual Evaluation Understudy) compares a machine-generated translation against one or more human "reference" translations, based on overlapping word sequences (n-grams).
from nltk.translate.bleu_score import sentence_bleu
reference = [["the", "cat", "is", "on", "the", "mat"]] # a human reference translation
candidate = ["the", "cat", "sat", "on", "the", "mat"] # the MODEL's generated translation
score = sentence_bleu(reference, candidate)
print(f"BLEU score: {score:.3f}")How BLEU Actually Works
─────────────────────────────────────────
BLEU counts how many n-grams (Module 1, Ch.3's related
concept, applied to sequences of 1-4 consecutive words) in
the CANDIDATE translation ALSO appear in the REFERENCE
translation(s), then combines these counts across different
n-gram sizes (unigrams through 4-grams, typically) into a
single score between 0 and 1 (or 0-100, depending on
convention).
─────────────────────────────────────────
4. ROUGE — Evaluating Summarization
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is BLEU's counterpart for summarization — measuring how much of a REFERENCE summary's content is captured by the generated summary.
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True) # Module 1, Ch.2's stemming
reference_summary = "The economy grew by 3% this quarter, driven by strong consumer spending."
generated_summary = "Consumer spending drove 3% economic growth this quarter."
scores = scorer.score(reference_summary, generated_summary)
for metric, score in scores.items():
print(f"{metric}: precision={score.precision:.3f}, recall={score.recall:.3f}, f1={score.fmeasure:.3f}")BLEU vs ROUGE — Precision vs Recall Emphasis
─────────────────────────────────────────
BLEU: emphasizes PRECISION — of the WORDS the
CANDIDATE generated, how many appear in the
reference? (penalizes ADDING extra, unsupported
content — appropriate for TRANSLATION, where
faithfulness to the source matters most)
ROUGE: emphasizes RECALL — of the WORDS in the
REFERENCE, how many did the candidate
CAPTURE? (penalizes MISSING important
content — appropriate for SUMMARIZATION,
where covering key points matters most)
─────────────────────────────────────────
5. The Shared Limitation of N-Gram Metrics
Both BLEU and ROUGE fundamentally compare surface word overlap — directly inheriting Module 1, Chapter 3's bag-of-words limitation: they cannot recognize that two DIFFERENTLY worded texts might convey the EXACT SAME meaning.
A Concrete Failure Case
─────────────────────────────────────────
Reference: "The company's profits increased significantly."
Candidate A: "The firm's earnings rose substantially."
(SAME meaning, DIFFERENT words — LOW
BLEU/ROUGE score, despite being an
EXCELLENT paraphrase)
Candidate B: "The company's profits increased
slightly."
(SIMILAR words, OPPOSITE meaning —
HIGH BLEU/ROUGE score, despite being
factually WRONG)
─────────────────────────────────────────
This is exactly the same limitation Module 2 addressed for individual words via dense embeddings — Section 6 applies that same fix to whole-text evaluation.
6. Embedding-Based Metrics
BERTScore addresses BLEU/ROUGE's surface-matching limitation directly, using Module 4's BERT embeddings to compare candidate and reference texts based on SEMANTIC similarity rather than exact word overlap.
from bert_score import score
candidates = ["The firm's earnings rose substantially."]
references = ["The company's profits increased significantly."]
precision, recall, f1 = score(candidates, references, lang="en")
print(f"BERTScore F1: {f1.item():.3f}") # correctly recognizes this as a GOOD match,
# unlike BLEU/ROUGE (Section 5)How BERTScore Fixes the Paraphrase Problem
─────────────────────────────────────────
Instead of comparing WORDS directly, BERTScore embeds EACH
token in BOTH texts (Module 4's contextual embeddings), then
matches tokens based on COSINE SIMILARITY in embedding space
— "firm's" and "company's," or "rose" and "increased," are
recognized as SEMANTICALLY similar, even though they're
DIFFERENT surface words — directly solving Section 5's
paraphrase-recognition failure.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Generation tasks have no single "correct" output, requiring different evaluation approaches than the ML Notes' classification metrics.
- Perplexity measures a language model's raw predictive quality on held-out text, but doesn't judge whether a specific response is helpful or correct.
- BLEU (precision-oriented, for translation) and ROUGE (recall-oriented, for summarization) both measure surface n-gram overlap with reference text.
- BERTScore addresses BLEU/ROUGE's inability to recognize valid paraphrases by comparing texts using contextual embeddings rather than exact word overlap.
Concept Check
- Why can't perplexity alone tell you whether a specific generated response is helpful or factually correct?
- What's the key difference in emphasis between BLEU and ROUGE, and why does that emphasis suit their respective tasks?
- How does BERTScore address the paraphrase-recognition failure that BLEU and ROUGE both share?
Next Chapter
→ Chapter 2: Evaluating LLMs — Benchmarks, Human Eval, LLM-as-Judge
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index