Fine Tuning And Aligning Llms
RLHF & Alignment
Chapter 1's instruction tuning teaches a model the format of following instructions — but doesn't necessarily teach it which of several valid-looking responses
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 2: Parameter-Efficient Fine-Tuning — LoRA & QLoRA Time to complete: ~25 minutes
Table of Contents
- Why Instruction Tuning Alone Isn't Enough
- What "Alignment" Means
- The RLHF Pipeline, Step by Step
- Step 1: Supervised Fine-Tuning (Recap)
- Step 2: Training a Reward Model
- Step 3: Reinforcement Learning With PPO
- DPO: a Simpler Alternative
- Known Limitations of RLHF
- Summary & Next Steps
1. Why Instruction Tuning Alone Isn't Enough
Chapter 1's instruction tuning teaches a model the format of following instructions — but doesn't necessarily teach it which of several valid-looking responses is genuinely more helpful, honest, or safe. Two instruction-following responses can both be grammatically fluent and on-topic while differing meaningfully in quality, helpfulness, or safety — a distinction instruction tuning's simple (instruction, response) format doesn't capture well.
2. What "Alignment" Means
Alignment refers to shaping a model's behavior to match human values and intentions — commonly summarized as being helpful, harmless, and honest.
Why This Requires Something BEYOND Instruction Tuning
─────────────────────────────────────────
"Helpful" and "harmless" often trade off against each other
in subtle ways that are hard to capture with a SINGLE
correct-response example per instruction (Chapter 1). Human
PREFERENCES between several possible responses — "this
answer is better than that one, because..." — carry richer
information than a single demonstrated "correct" response,
and this is exactly what RLHF is built to exploit.
─────────────────────────────────────────
3. The RLHF Pipeline, Step by Step
RLHF (Reinforcement Learning from Human Feedback) builds on AI Notes, Module 5's reinforcement learning foundations and ML Notes, Module 8's policy gradient/actor-critic methods, applying them to fine-tune a language model using human preference signals rather than a fixed reward function.
The Three-Stage Pipeline
─────────────────────────────────────────
Stage 1: Supervised Fine-Tuning (SFT) — Chapter 1's
instruction
tuning, recapped
in Section 4
Stage 2: Reward Model Training — training a
SEPARATE model
to PREDICT
human
preferences
(Section 5)
Stage 3: RL Fine-Tuning (PPO) — using the
reward model
to fine-tune
the LLM via
reinforcement
learning
(Section 6)
─────────────────────────────────────────
4. Step 1: Supervised Fine-Tuning (Recap)
This stage is exactly Chapter 1's instruction tuning — no new concept here, just the starting point for the stages that follow.
5. Step 2: Training a Reward Model
Human annotators are shown multiple responses to the same prompt (generated by the SFT model from Stage 1) and asked to rank them by quality/preference, rather than writing a single "correct" response.
# Human preference data format
preference_examples = [
{
"prompt": "Explain photosynthesis to a 10-year-old.",
"response_a": "Photosynthesis is the process by which plants use sunlight, water, and CO2 to make food.",
"response_b": "Photosynthesis is a biochemical process involving chlorophyll-mediated electron transport chains.",
"preferred": "a", # human annotators judged response A clearer/more appropriate for the audience
},
]A reward model — architecturally a Module 4-style encoder or Module 5-style decoder with a single scalar output head, instead of a classification head — is trained to predict a score reflecting how much a human would prefer a given response.
import torch
import torch.nn.functional as F
def reward_model_loss(reward_model, prompt, response_preferred, response_rejected):
"""The Bradley-Terry preference model — directly extends the ML Notes'
logistic regression (Module 4, Ch.1) to PAIRWISE preference data"""
reward_preferred = reward_model(prompt, response_preferred) # a SINGLE scalar score
reward_rejected = reward_model(prompt, response_rejected)
# Train the model so reward_preferred > reward_rejected, via a LOGISTIC loss
loss = -F.logsigmoid(reward_preferred - reward_rejected)
return lossWhy Train a SEPARATE Reward Model, Rather Than Using Preferences Directly
─────────────────────────────────────────
Human preference labeling is SLOW and EXPENSIVE — a
trained reward model acts as a SCALABLE, automated PROXY for
human judgment, providing a reward signal for as MANY
training examples as needed (Stage 3) without requiring a
human to evaluate every single one.
─────────────────────────────────────────
6. Step 3: Reinforcement Learning With PPO
The SFT model (Stage 1) is now fine-tuned using PPO (Proximal Policy Optimization) — directly the actor-critic-family method from ML Notes, Module 8, Chapter 3 — using the reward model (Stage 2) as the reward signal, treating the LLM itself as the RL "policy."
Mapping RLHF Onto the ML Notes' RL Framework
─────────────────────────────────────────
Agent (ML Notes, the LLM being fine-tuned
Module 8):
State: the prompt (and tokens generated
so far)
Action: choosing the NEXT token to
generate (DL Notes, Module 5,
Ch.4's autoregressive framing)
Reward: the SCALAR score from the
reward model (Section 5),
given to the FULL generated
response
Policy: the LLM's own probability
distribution over next
tokens — exactly what
Module 5, Chapter 1's
decoding process samples
from
─────────────────────────────────────────
import torch
def ppo_style_loss(policy_model, reference_model, reward_model, prompt, kl_coefficient=0.1):
"""A simplified conceptual sketch of the RLHF training objective"""
generated_response = policy_model.generate(prompt)
reward = reward_model(prompt, generated_response) # Section 5's learned reward
# A CRITICAL term: penalize the policy for straying too far from the
# ORIGINAL reference model (the Stage 1 SFT model, kept FROZEN)
policy_log_probs = policy_model.log_prob(generated_response, prompt)
reference_log_probs = reference_model.log_prob(generated_response, prompt)
kl_penalty = kl_coefficient * (policy_log_probs - reference_log_probs)
total_reward = reward - kl_penalty # reward the MODEL, but PENALIZE drifting too far from
# the original, coherent language model
return total_rewardWhy the KL Penalty Term Is Essential
─────────────────────────────────────────
Without it, the policy could learn to EXPLOIT weaknesses in
the reward model — generating text that scores HIGH on the
(imperfect) reward model, while becoming increasingly
INCOHERENT, repetitive, or nonsensical as actual LANGUAGE
(a failure mode called "REWARD HACKING," directly echoing
DL Notes, Module 8, Chapter 3's mode collapse concept applied
in a new context). The KL penalty keeps the fine-tuned
policy ANCHORED close to the original, fluent SFT model.
─────────────────────────────────────────
7. DPO: a Simpler Alternative
Direct Preference Optimization (DPO), a more recent technique, achieves a similar alignment effect without Stage 2's separate reward model or Stage 3's full RL training loop — it directly optimizes the language model on preference pairs using a single, simpler loss function derived mathematically to have the SAME optimal solution as the full RLHF pipeline.
Why DPO Has Become Popular
─────────────────────────────────────────
RLHF's THREE-stage pipeline (Sections 4-6) is complex,
requires training and maintaining an ADDITIONAL reward
model, and PPO's reinforcement learning training loop is
notoriously difficult to get right in practice (echoing DL
Notes, Module 8, Chapter 3's GAN training-stability
discussion — RL training shares SOME of that same
instability). DPO collapses this into a SINGLE, more stable
supervised-learning-STYLE optimization, directly on
preference data — simpler to implement, and empirically
competitive with full RLHF on many benchmarks.
─────────────────────────────────────────
8. Known Limitations of RLHF
Genuine, Documented Challenges
─────────────────────────────────────────
Reward hacking: the policy exploiting reward model
imperfections rather than genuinely
improving (Section 6)
Reward model bias: the reward model inherits WHATEVER
biases exist in its human
preference training data — it is
only ever a PROXY for human
judgment, not human judgment itself
Sycophancy: models trained via RLHF can learn
to tell users what they seem to
WANT to hear, rather than what's
most ACCURATE — since human
raters sometimes (even
unconsciously) prefer agreeable
responses over correct-but-
unwelcome ones
─────────────────────────────────────────
These are active, ongoing areas of alignment research — worth understanding as genuine, unsolved limitations, not merely implementation details to be perfected.
9. Summary & Next Steps
Key Takeaways
- RLHF's three stages are supervised fine-tuning (Chapter 1's instruction tuning), reward model training on human preference rankings, and RL fine-tuning of the policy using that reward model.
- The reward model acts as a scalable proxy for human judgment, avoiding the need for a human to evaluate every single training example directly.
- PPO fine-tunes the LLM as an RL policy, with a KL penalty term that's essential for preventing reward hacking — exploiting reward model weaknesses at the cost of incoherent output.
- DPO achieves similar alignment effects through a single, simpler supervised-learning-style objective, without a separate reward model or full RL training loop.
Module 6 Complete — What's Next
You've now covered the complete lifecycle from raw pretrained model to helpful, aligned assistant: instruction tuning, memory-efficient adaptation via LoRA/QLoRA, and alignment via RLHF/DPO. Module 7 shifts focus to using these already-trained, aligned models effectively — prompt engineering, without any further training at all.
Concept Check
- Why is a separate reward model trained, rather than directly using human preference labels during RL fine-tuning?
- What problem does the KL penalty term in PPO-based RLHF specifically address?
- What advantage does DPO offer over the full three-stage RLHF pipeline, and what's the tradeoff?
Next Chapter
→ Chapter 4: A Practical Fine-Tuning Recipe
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index