NLP & LLMs

Fine Tuning And Aligning Llms

Instruction Tuning

Module 5, Chapter 2, Section 7 noted that a raw, CLM-pretrained model has no inherent alignment toward being helpful — it simply predicts statistically plausibl

JrCodex·6 min read

Jr Codex NLP & LLM Notes

Level: Advanced Prerequisites: Module 5: Decoder Models — GPT & Modern LLMs Time to complete: ~20 minutes

Table of Contents

  1. The Gap Between "Predicts Text" and "Follows Instructions"
  2. What Instruction Tuning Actually Does
  3. Instruction Datasets
  4. Instruction Tuning as Standard Fine-Tuning
  5. Instruction Tuning's Generalization Effect
  6. A Practical Instruction-Tuning Example
  7. Summary & Next Steps

1. The Gap Between "Predicts Text" and "Follows Instructions"

Module 5, Chapter 2, Section 7 noted that a raw, CLM-pretrained model has no inherent alignment toward being helpful — it simply predicts statistically plausible continuations. Ask a raw pretrained model "What's the capital of France?" and it might just as easily continue with more questions in the same style (since that's a plausible continuation of a list of trivia questions) rather than actually answering.

2. What Instruction Tuning Actually Does

Instruction tuning fine-tunes a pretrained model (DL Notes, Module 7) on a dataset of (instruction, response) pairs — directly teaching the model the behavior of following an instruction and producing a helpful response, rather than just continuing a pattern.

Before vs After Instruction Tuning
─────────────────────────────────────────
  RAW pretrained model,          "What's the capital of
  given "What's the capital         France? What's the capital
  of France?":                        of Germany? What's the..."
                                         (continuing the PATTERN
                                         of a trivia list)
  INSTRUCTION-TUNED model,          "The capital of France is
  given the SAME input:                Paris."
                                          (correctly interpreting
                                          this as a QUESTION
                                          requiring an ANSWER)
─────────────────────────────────────────

3. Instruction Datasets

# A small illustrative sample of instruction-tuning data format
instruction_examples = [
    {
        "instruction": "Summarize the following text in one sentence.",
        "input": "The Industrial Revolution was a period of major industrialization...",
        "output": "The Industrial Revolution transformed manufacturing and society through mechanization.",
    },
    {
        "instruction": "Translate the following sentence to French.",
        "input": "The weather is beautiful today.",
        "output": "Le temps est magnifique aujourd'hui.",
    },
    {
        "instruction": "Write a Python function to reverse a string.",
        "input": "",
        "output": "def reverse_string(s):\n    return s[::-1]",
    },
]
Where Instruction Data Comes From
─────────────────────────────────────────
  Human-written:      crowdworkers or domain experts write
                         (instruction, response) pairs directly
                         — HIGH quality, but expensive and
                         SLOW to scale
  Existing NLP                 repurposing datasets originally
  datasets, reformatted:          built for specific tasks
                                     (Module 4's classification/
                                     QA datasets) into an
                                     instruction-following format
  Model-generated                     using an ALREADY-capable
  ("self-instruct"):                     LLM to GENERATE new
                                            instruction/response
                                            pairs, then filtering
                                            for quality — a
                                            distillation-like
                                            approach (Module 4,
                                            Ch.4, Section 4)
                                            that scales far faster
                                            than pure human writing
─────────────────────────────────────────

4. Instruction Tuning as Standard Fine-Tuning

Mechanically, instruction tuning is exactly the CLM training objective from Module 3, Chapter 3 and Module 5, Chapter 1 — the only genuine change is what text the model is trained to predict.

def format_instruction_example(example):
    """Combining instruction + input + output into ONE training sequence"""
    if example["input"]:
        prompt = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n"
    else:
        prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n"
 
    full_text = prompt + example["output"]
    return prompt, full_text
 
prompt, full_text = format_instruction_example(instruction_examples[0])
print(full_text)
import torch
 
def instruction_tuning_loss(tokenizer, prompt, full_text, model):
    """Mask the LOSS for the prompt portion — only train the model to predict the RESPONSE"""
    full_ids = tokenizer(full_text, return_tensors="pt")["input_ids"]
    prompt_ids = tokenizer(prompt, return_tensors="pt")["input_ids"]
    prompt_length = prompt_ids.shape[1]
 
    labels = full_ids.clone()
    labels[:, :prompt_length] = -100      # DON'T train the model to "predict" the instruction itself,
                                              # only to predict the RESPONSE that follows it
 
    outputs = model(input_ids=full_ids, labels=labels)
    return outputs.loss
Why Mask the Prompt's Loss
─────────────────────────────────────────
  The GOAL is teaching the model to GENERATE good responses
  GIVEN an instruction — not to memorize or "predict"
  instructions themselves (which would be a strange, unhelpful
  training signal). Masking the prompt tokens (setting their
  labels to -100, exactly as MLM does for unmasked tokens in
  Module 4, Chapter 2) focuses the entire training signal on
  the RESPONSE portion.
─────────────────────────────────────────

5. Instruction Tuning's Generalization Effect

A genuinely important empirical finding: instruction tuning on a diverse set of tasks generalizes to instructions the model was never explicitly trained on.

Why Diversity Matters More Than Volume
─────────────────────────────────────────
  Instruction tuning on MANY DIFFERENT kinds of tasks
  (summarization, translation, coding, question answering)
  teaches the model the GENERAL BEHAVIOR of "read an
  instruction, understand its intent, produce an appropriate
  response" — a skill that TRANSFERS to genuinely NOVEL
  instructions at inference time, rather than the narrower
  skill of "produce this SPECIFIC output for this SPECIFIC
  instruction phrasing."

  This mirrors DL Notes, Module 7, Chapter 1, Section 3's
  finding about CNN layers — broad, general training produces
  broadly USEFUL, TRANSFERABLE capability, rather than narrow
  memorization.
─────────────────────────────────────────

6. A Practical Instruction-Tuning Example

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from datasets import Dataset
 
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token      # GPT-2 has no native pad token
model = AutoModelForCausalLM.from_pretrained(model_name)
 
def preprocess(example):
    prompt, full_text = format_instruction_example(example)
    tokenized = tokenizer(full_text, truncation=True, max_length=256, padding="max_length")
 
    prompt_length = len(tokenizer(prompt)["input_ids"])
    labels = tokenized["input_ids"].copy()
    labels[:prompt_length] = [-100] * prompt_length      # Section 4's masking
    tokenized["labels"] = labels
    return tokenized
 
dataset = Dataset.from_list(instruction_examples).map(preprocess)
 
training_args = TrainingArguments(
    output_dir="./instruction-tuned-gpt2",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    learning_rate=5e-5,      # DL Notes, Module 7, Ch.2's low fine-tuning rate
)
 
trainer = Trainer(model=model, args=training_args, train_dataset=dataset)
trainer.train()

7. Summary & Next Steps

Key Takeaways

  • A raw, CLM-pretrained model has no inherent instruction-following behavior; it simply continues text in statistically plausible ways.
  • Instruction tuning fine-tunes a pretrained model on (instruction, response) pairs, using the exact same CLM objective, but with the loss masked so only the response portion trains the model.
  • Instruction data can be human-written, adapted from existing NLP datasets, or generated by an already-capable model (self-instruct).
  • Diversity of instruction-tuning tasks matters more than raw volume — it teaches the general behavior of following novel instructions, generalizing well beyond the specific examples trained on.

Concept Check

  1. Why does a raw, pretrained-only model often fail to actually answer a question, even if it "knows" the answer?
  2. Why is the loss masked over the prompt/instruction portion during instruction tuning?
  3. Why does diversity of instruction-tuning tasks matter more than the sheer number of examples?

Next Chapter

→ Chapter 2: Parameter-Efficient Fine-Tuning — LoRA & QLoRA

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index