Transfer Learning And Practical Training
Practical Fine-Tuning Recipes
This closing chapter assembles Chapters 1-3 into complete, runnable recipes — one for vision, one for text — directly applying this module's concepts to real pr
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 3: Pretrained Models & Model Hubs Time to complete: ~30 minutes
Table of Contents
- A Complete Fine-Tuning Workflow
- Recipe: Fine-Tuning a Vision Model
- Recipe: Fine-Tuning a Transformer for Text Classification
- Parameter-Efficient Fine-Tuning (a Preview)
- Monitoring for Catastrophic Forgetting
- A Fine-Tuning Checklist
- Summary & Next Steps
1. A Complete Fine-Tuning Workflow
This closing chapter assembles Chapters 1-3 into complete, runnable recipes — one for vision, one for text — directly applying this module's concepts to real pretrained models.
The General Recipe
─────────────────────────────────────────
1. Choose a pretrained model APPROPRIATE to your task
(Chapter 3)
2. Replace the task-specific FINAL layer(s) (Chapter 2,
Section 2)
3. Choose a strategy: feature extraction, full fine-tuning,
or gradual unfreezing (Chapter 2)
4. Use the model's EXACT expected preprocessing (Chapter 3,
Section 5)
5. Train with a LOW learning rate if fine-tuning (Chapter 2,
Section 3), monitoring closely (Section 5, below)
─────────────────────────────────────────
2. Recipe: Fine-Tuning a Vision Model
import torch
import torch.nn as nn
from torchvision import models, transforms, datasets
from torch.utils.data import DataLoader
# Step 1-2: Load pretrained model, replace final layer
model = models.resnet50(weights="IMAGENET1K_V2")
num_classes = 5 # e.g. classifying 5 types of plant diseases
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Step 3: Feature extraction FIRST (Chapter 2, Section 2) — safest starting point
for name, param in model.named_parameters():
param.requires_grad = "fc" in name
# Step 4: EXACT preprocessing the pretrained model expects (Chapter 3, Section 5)
weights_meta = models.ResNet50_Weights.IMAGENET1K_V2
transform = weights_meta.transforms()
train_dataset = datasets.ImageFolder("plant_disease_data/train", transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
# Step 5: train the new layer only, first
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
optimizer = torch.optim.Adam(model.fc.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(5): # feature extraction phase — converges QUICKLY with few trainable params
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
optimizer.zero_grad()
loss = criterion(model(images), labels)
loss.backward()
optimizer.step()
print(f"Feature extraction epoch {epoch+1} complete")
# THEN, optionally, unfreeze the last block and fine-tune further at a LOW rate
for param in model.layer4.parameters():
param.requires_grad = True
fine_tune_optimizer = torch.optim.Adam([
{"params": model.layer4.parameters(), "lr": 1e-5}, # Chapter 2, Section 5
{"params": model.fc.parameters(), "lr": 1e-4},
])
model.train()
for epoch in range(3): # fine-tuning phase — FEWER epochs, smaller adjustments
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
fine_tune_optimizer.zero_grad()
loss = criterion(model(images), labels)
loss.backward()
fine_tune_optimizer.step()
print(f"Fine-tuning epoch {epoch+1} complete")3. Recipe: Fine-Tuning a Transformer for Text Classification
from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
from datasets import load_dataset
# Step 1-2: A pretrained ENCODER-ONLY model (Module 6, Ch.4, Sec.7), with a
# classification head automatically added
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2) # e.g. binary sentiment
dataset = load_dataset("imdb") # a standard sentiment classification dataset
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=256)
tokenized_dataset = dataset.map(tokenize_function, batched=True)
# Step 3-5: HuggingFace's Trainer handles the fine-tuning loop, using SENSIBLE
# fine-tuning defaults (LOW learning rate, few epochs — Chapter 2)
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3, # fine-tuning typically needs FEW epochs
per_device_train_batch_size=16,
learning_rate=2e-5, # a SMALL, typical Transformer fine-tuning rate (Chapter 2, Sec.3)
warmup_steps=500, # Module 3, Ch.5's warmup — near-mandatory for Transformers
weight_decay=0.01, # Module 3, Ch.4
evaluation_strategy="epoch",
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["test"],
)
trainer.train()This exact pattern — a pretrained encoder, a small added classification head, fine-tuned at a low learning rate with warmup — is the standard recipe covered in full depth in the NLP & LLM Notes' fine-tuning chapter.
4. Parameter-Efficient Fine-Tuning (a Preview)
For very large models (billions of parameters), even fine-tuning all layers (Chapter 2, Section 3) becomes prohibitively expensive in memory. Parameter-Efficient Fine-Tuning (PEFT) techniques — most notably LoRA (Low-Rank Adaptation) — freeze the entire pretrained model and instead train a small number of additional parameters injected into each layer.
The Core LoRA Idea (Briefly)
─────────────────────────────────────────
Instead of updating a large weight matrix W directly,
LoRA freezes W entirely and learns TWO much SMALLER
matrices (A and B) such that their product approximates
the NEEDED update to W — dramatically reducing the number
of trainable parameters (often by 100x or more), while
achieving fine-tuning quality close to full fine-tuning.
─────────────────────────────────────────
This technique is covered in full practical depth in the NLP & LLM Notes, where it matters most given the enormous scale of modern large language models — this preview exists simply to connect Chapter 2's general fine-tuning spectrum to the technique practitioners actually reach for at LLM scale.
5. Monitoring for Catastrophic Forgetting
Catastrophic forgetting is a specific risk during fine-tuning: if the learning rate is too high or training continues too long, the model can lose the general capabilities it gained from pretraining, in favor of overfitting narrowly to the new, smaller fine-tuning dataset.
Symptoms and Mitigations
─────────────────────────────────────────
Symptom: the fine-tuned model performs WORSE than the
original pretrained model on tasks OUTSIDE
the fine-tuning dataset's specific domain
Mitigation 1: use a LOWER learning rate (Chapter 2,
Section 3) and fewer epochs
Mitigation 2: use early stopping (Module 3, Ch.4)
based on a VALIDATION set that
reflects the model's INTENDED broader
use, not just the narrow fine-tuning
task
Mitigation 3: consider PEFT/LoRA (Section 4) —
since the ORIGINAL weights remain
completely frozen and unchanged,
catastrophic forgetting of the
base model's capabilities is
structurally impossible
─────────────────────────────────────────
6. A Fine-Tuning Checklist
Before Calling Fine-Tuning "Done"
─────────────────────────────────────────
☐ Preprocessing EXACTLY matches the pretrained model's
expected input (Chapter 3, Section 5)
☐ Learning rate is substantially LOWER than a from-scratch
training rate would use (Chapter 2, Section 3)
☐ Started with feature extraction, only progressing to
fine-tuning if needed (Chapter 2, Section 6's decision
guide)
☐ Checked for catastrophic forgetting on a held-out,
broader validation set (Section 5)
☐ Verified the pretrained model's LICENSE permits your
intended use (Chapter 3, Section 6)
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- A complete fine-tuning workflow combines choosing an appropriate pretrained model, replacing the final layer, selecting a strategy from Chapter 2's spectrum, matching preprocessing exactly, and training at a low learning rate.
- HuggingFace's
TrainerAPI implements standard fine-tuning defaults (low learning rate, warmup, few epochs) directly, reflecting established best practices for Transformer fine-tuning. - Parameter-Efficient Fine-Tuning (LoRA) trains a small number of additional parameters while freezing the entire pretrained model, dramatically reducing memory requirements for large models.
- Catastrophic forgetting — losing general pretrained capabilities during fine-tuning — is mitigated by low learning rates, early stopping on broad validation data, or PEFT techniques that never modify the original weights.
Module 7 Complete — What's Next
You now have a complete, practical transfer learning toolkit: knowing when to use it, how to choose between feature extraction and fine-tuning, where to find pretrained models responsibly, and concrete recipes for both vision and text. Module 8 shifts to an entirely different class of models: those that generate new data rather than just classifying or extracting features from existing data.
Concept Check
- Why does the fine-tuning recipe in Section 2 start with feature extraction before progressing to unfreezing layers?
- What problem does LoRA solve for very large models, and why does it structurally prevent catastrophic forgetting?
- What's one concrete way to detect catastrophic forgetting during fine-tuning?
Next Module
→ Module 8: Generative Deep Learning
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index