Fine Tuning And Aligning Llms
A Practical Fine-Tuning Recipe
This closing chapter assembles Chapters 1-3 into a single, practical, runnable recipe — directly extending DL Notes, Module 7, Chapter 4's fine-tuning recipes t
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 3: RLHF & Alignment Time to complete: ~25 minutes
Table of Contents
- Bringing This Module Together
- Step 1: Decide If Fine-Tuning Is Even Necessary
- Step 2: Prepare a Quality Dataset
- Step 3: Choose a Base Model
- Step 4: Fine-Tune With QLoRA
- Step 5: Evaluate Before and After
- Step 6: Watch for Catastrophic Forgetting
- A Complete Recipe, End to End
- Summary & Next Steps
1. Bringing This Module Together
This closing chapter assembles Chapters 1-3 into a single, practical, runnable recipe — directly extending DL Notes, Module 7, Chapter 4's fine-tuning recipes to the LLM-specific techniques covered in this module.
2. Step 1: Decide If Fine-Tuning Is Even Necessary
Before any of this module's techniques, consider whether Module 7's prompt engineering or Module 8's RAG could solve the problem WITHOUT any training at all.
A Practical Decision Order (Cheapest to Most Expensive)
─────────────────────────────────────────
1. Better PROMPTING (Module 7): free, instant to try,
no infrastructure
2. RAG (Module 8): needed when the
model lacks
specific KNOWLEDGE
(not behavior) —
still no training
3. Few-shot examples in the provide DEMONSTRATIONS
prompt (Module 5, Ch.3): directly, no
training required
4. Fine-tuning (THIS MODULE): needed when the
desired BEHAVIOR
(tone, format,
specialized
task performance)
can't be
achieved through
prompting alone
─────────────────────────────────────────
Fine-tuning is the most expensive, most complex option on this list — reach for it only after simpler approaches have genuinely been tried and found insufficient, directly echoing the ML Notes' "start simple" workflow principle.
3. Step 2: Prepare a Quality Dataset
Data Quality Guidance
─────────────────────────────────────────
Quality over QUANTITY: a few hundred CAREFULLY curated
examples often outperform
thousands of noisy, inconsistent
ones — echoing the ML Notes'
data quality principles, now
applied at instruction-tuning
scale
DIVERSITY of examples (Chapter 1, Section 5) — cover
(Chapter 1): the RANGE of inputs/phrasings
the model will actually
encounter in production
CONSISTENT format: use the SAME instruction/
response template
throughout (Chapter
1, Section 4)
Held-out VALIDATION set: NEVER skip this
(ML Notes, Module
1, Ch.5) — needed
for Step 5's
evaluation
─────────────────────────────────────────
4. Step 3: Choose a Base Model
Directly applying Module 5, Chapter 4's decision framework: fine-tuning generally requires an open-weight model, since most closed/proprietary models don't expose their weights for direct fine-tuning (though some providers offer managed fine-tuning APIs as an alternative).
Model Size vs Available Hardware
─────────────────────────────────────────
Small open models fine-tunable on MODEST hardware,
(a few billion even with QLoRA (Chapter 2) —
parameters): good for narrow, well-defined
tasks
Larger open models require MORE substantial GPU
(tens of billions resources, even with QLoRA —
of parameters): but generally provide
STRONGER baseline
capability BEFORE
fine-tuning even begins
─────────────────────────────────────────
5. Step 4: Fine-Tune With QLoRA
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training
from datasets import Dataset
import torch
model_name = "gpt2-large" # substitute an appropriately-sized open model in a real project
# QLoRA setup (Chapter 2, Section 5)
quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=quantization_config, device_map="auto")
model = prepare_model_for_kbit_training(model) # a peft utility handling quantization-specific setup
# LoRA setup (Chapter 2, Section 4)
lora_config = LoraConfig(task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["c_attn"])
peft_model = get_peft_model(model, lora_config)
peft_model.print_trainable_parameters()
# Instruction data (Chapter 1)
dataset = Dataset.from_list(instruction_examples).map(preprocess) # reusing Chapter 1's preprocessing
split_dataset = dataset.train_test_split(test_size=0.2) # ML Notes, Module 1, Ch.5's train/val split
training_args = TrainingArguments(
output_dir="./fine-tuned-assistant",
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=2e-4,
warmup_steps=50, # DL Notes, Module 3, Ch.5
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True, # DL Notes, Module 3, Ch.4's early-stopping-adjacent concept
)
trainer = Trainer(model=peft_model, args=training_args,
train_dataset=split_dataset["train"], eval_dataset=split_dataset["test"])
trainer.train()6. Step 5: Evaluate Before and After
Never trust that fine-tuning "worked" without directly comparing before/after behavior — directly echoing the ML Notes' rigorous model comparison discipline.
def compare_before_after(base_model, fine_tuned_model, test_prompts, tokenizer):
for prompt in test_prompts:
inputs = tokenizer(prompt, return_tensors="pt")
base_output = base_model.generate(**inputs, max_new_tokens=50)
fine_tuned_output = fine_tuned_model.generate(**inputs, max_new_tokens=50)
print(f"Prompt: {prompt}")
print(f"BEFORE: {tokenizer.decode(base_output[0])}")
print(f"AFTER: {tokenizer.decode(fine_tuned_output[0])}")
print("---")Module 9's evaluation chapters cover this in full rigor — both automated metrics and human evaluation.
7. Step 6: Watch for Catastrophic Forgetting
Directly the concern from DL Notes, Module 7, Chapter 4, Section 5 — verify the fine-tuned model hasn't lost GENERAL capabilities while gaining the NEW, specific behavior.
general_capability_prompts = [
"What is the capital of Japan?",
"Write a haiku about autumn.",
"Explain the water cycle briefly.",
]
# Run compare_before_after() on THESE prompts too — if the FINE-TUNED model performs
# noticeably WORSE on tasks UNRELATED to the fine-tuning goal, that's a signal to
# reduce the learning rate, reduce epochs, or reconsider LoRA's rank (Chapter 2)8. A Complete Recipe, End to End
The Full Practical Workflow
─────────────────────────────────────────
1. Confirm fine-tuning is NECESSARY (Section 2) — not
solvable by prompting/RAG alone
2. Curate a SMALL, HIGH-QUALITY, DIVERSE instruction
dataset (Section 3, Chapter 1)
3. Choose an appropriately-sized OPEN model (Section 4)
4. Fine-tune with QLoRA (Section 5) — memory-efficient,
fast to iterate
5. Evaluate RIGOROUSLY, before vs after, on BOTH the
target task AND general capabilities (Sections 6-7)
6. If alignment/preference behavior matters beyond simple
instruction-following, consider DPO (Chapter 3, Section
7) on top of the instruction-tuned result
─────────────────────────────────────────
9. Summary & Next Steps
Key Takeaways
- Fine-tuning should be the last resort after prompting and RAG have genuinely been tried, given its added cost and complexity relative to those alternatives.
- Data quality and diversity matter more than raw quantity for instruction-tuning datasets, and a held-out validation set is essential for honest evaluation.
- QLoRA is the practical default for fine-tuning modern LLMs on limited hardware, combining quantization and low-rank adaptation.
- Rigorous before/after evaluation must check both target-task improvement and general capability retention, guarding against catastrophic forgetting.
Module 6 Complete — What's Next
You've now covered the complete path from a raw pretrained model to a fine-tuned, aligned assistant: instruction tuning, parameter-efficient adaptation, RLHF/DPO alignment, and a practical end-to-end recipe. Module 7 covers the alternative, training-free path to getting more out of an LLM: prompt engineering.
Concept Check
- Why should prompting and RAG generally be tried before reaching for fine-tuning?
- What's the risk of curating a large but low-quality, inconsistent instruction dataset?
- Why is it important to test a fine-tuned model on prompts unrelated to the fine-tuning goal, not just on the target task?
Next Module
→ Module 7: Prompt Engineering
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index