Deep Learning

Transfer Learning And Practical Training

Feature Extraction vs Fine-Tuning

Chapter 1, Section 5 introduced the transfer learning spectrum. This chapter covers its two most common practical points: feature extraction and fine-tuning.

JrCodex·6 min read

Jr Codex Deep Learning Notes

Level: Intermediate–Advanced Prerequisites: Chapter 1: Why Train From Scratch Rarely Makes Sense Time to complete: ~25 minutes

Table of Contents

  1. Two Concrete Strategies
  2. Feature Extraction
  3. Fine-Tuning
  4. Layer-Wise (Gradual) Unfreezing
  5. Choosing a Learning Rate for Fine-Tuning
  6. Deciding Which Strategy to Use
  7. Summary & Next Steps

1. Two Concrete Strategies

Chapter 1, Section 5 introduced the transfer learning spectrum. This chapter covers its two most common practical points: feature extraction and fine-tuning.

2. Feature Extraction

Feature extraction freezes every pretrained layer entirely — their weights never update during training — and adds a new, small, randomly-initialized final layer (or few layers) trained specifically for the new task.

import torch
import torch.nn as nn
from torchvision import models
 
# Load a model PRETRAINED on ImageNet
model = models.resnet50(weights="IMAGENET1K_V2")
 
# FREEZE every existing parameter — none of these will be updated during training
for param in model.parameters():
    param.requires_grad = False      # Module 2, Ch.4 — no gradients computed OR applied
 
# REPLACE the final classification layer with a NEW one, sized for OUR task
num_classes = 10      # e.g. our own 10-category classification task
model.fc = nn.Linear(model.fc.in_features, num_classes)      # ONLY this new layer has requires_grad=True by default
 
# Only the NEW layer's parameters get passed to the optimizer
optimizer = torch.optim.Adam(model.fc.parameters(), lr=0.001)
Why Freeze Everything
─────────────────────────────────────────
  The pretrained layers already encode GENERAL, useful visual
  features (Chapter 1, Section 3) — freezing them means:
    - MUCH faster training (backpropagation, Module 2 Ch.4,
        doesn't need to compute gradients for the frozen
        layers at all)
    - Works well even with a SMALL new dataset — there are
        very few new parameters (just the new final layer) to
        overfit
    - Cannot adapt the pretrained features AT ALL to the new
        task's specific quirks — a real limitation if the new
        task differs substantially from the original training
        data
─────────────────────────────────────────

3. Fine-Tuning

Fine-tuning goes further: after replacing the final layer (as in Section 2), it unfreezes some or all of the pretrained layers, and continues training them — typically at a much lower learning rate than would be used for training from scratch.

import torch
import torch.nn as nn
from torchvision import models
 
model = models.resnet50(weights="IMAGENET1K_V2")
model.fc = nn.Linear(model.fc.in_features, num_classes=10)
 
# UNFREEZE everything this time — allow the ENTIRE network to adapt
for param in model.parameters():
    param.requires_grad = True
 
# A MUCH lower learning rate than training from scratch would use (Module 3, Ch.5)
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)      # note: 100x smaller than Section 2's new-layer-only rate
Why Fine-Tuning Needs a MUCH Lower Learning Rate
─────────────────────────────────────────
  The pretrained weights already encode USEFUL, well-trained
  representations (Chapter 1) — a LARGE learning rate (Module
  3, Ch.5) risks DESTROYING this useful starting point with
  large, disruptive updates, effectively "forgetting" what was
  learned during pretraining before the new task's smaller
  dataset can teach it anything better. A small learning rate
  allows GENTLE adjustment, adapting pretrained features
  slightly toward the new task, without erasing their
  original value.
─────────────────────────────────────────

4. Layer-Wise (Gradual) Unfreezing

A refined middle-ground strategy: start with feature extraction (Section 2), then progressively unfreeze layers from the END of the network backward, retraining a bit more of the network at each stage.

def gradual_unfreezing_schedule(model, epoch):
    """A simplified illustration of progressively unfreezing layers, LATEST first"""
    if epoch == 0:
        # Only the new final layer is trainable
        for name, param in model.named_parameters():
            param.requires_grad = "fc" in name
    elif epoch == 5:
        # ALSO unfreeze the last residual block (Module 4, Ch.2)
        for name, param in model.named_parameters():
            if "fc" in name or "layer4" in name:
                param.requires_grad = True
    elif epoch == 10:
        # Unfreeze EVERYTHING
        for param in model.parameters():
            param.requires_grad = True
Why Unfreeze LATE Layers First
─────────────────────────────────────────
  Chapter 1, Section 3 established that LATE layers are the
  most TASK-SPECIFIC — they're the layers MOST likely to
  benefit from adapting to the new task, and least likely to
  contain broadly useful general features worth preserving
  UNCHANGED. Unfreezing them FIRST, before touching the
  general early layers, minimizes the risk of disrupting the
  valuable, general representations while still allowing
  meaningful task-specific adaptation.
─────────────────────────────────────────

5. Choosing a Learning Rate for Fine-Tuning

A Practical Layer-Wise Learning Rate Strategy
─────────────────────────────────────────
  EARLY layers (general features):     VERY low learning rate,
                                          or KEEP frozen entirely
                                          — minimal risk of
                                          disruption preferred
  MIDDLE layers:                            LOW-to-MODERATE
                                               learning rate
  LATE layers + the NEW final layer:            HIGHER learning
                                                   rate — these
                                                   need the MOST
                                                   adaptation to
                                                   the new task
─────────────────────────────────────────

import torch
 
# Different learning rates for different parts of the network — "discriminative fine-tuning"
optimizer = torch.optim.Adam([
    {"params": model.layer1.parameters(), "lr": 1e-6},      # EARLIEST layers — barely adjust
    {"params": model.layer2.parameters(), "lr": 1e-5},
    {"params": model.layer3.parameters(), "lr": 1e-5},
    {"params": model.layer4.parameters(), "lr": 1e-4},      # LATEST pretrained layers — adjust more
    {"params": model.fc.parameters(), "lr": 1e-3},              # NEW layer — largest learning rate
])

6. Deciding Which Strategy to Use

A Practical Decision Guide
─────────────────────────────────────────
  SMALL new dataset,      Feature extraction (Section 2) —
  task SIMILAR to            minimal risk of overfitting the
  original training:            few new parameters being trained
  SMALL new dataset,          Feature extraction still preferred
  task DIFFERENT from            — fine-tuning risks overfitting
  original training:                a small dataset with too
                                       many trainable parameters
  LARGE new dataset,              Fine-tuning (Section 3) — enough
  task SIMILAR to                    data to safely adapt more of
  original training:                    the network without
                                           overfitting
  LARGE new dataset,                  Fine-tuning, likely
  task DIFFERENT from                    unfreezing MORE layers,
  original training:                        or even considering
                                               training from
                                               scratch (Chapter 1,
                                               Section 4) if the
                                               domain gap is severe
─────────────────────────────────────────

This decision matrix directly parallels the ML Notes' bias-variance framing: fewer trainable parameters (feature extraction) reduces variance/overfitting risk on small datasets; more trainable parameters (fine-tuning) reduces bias, but requires enough data to safely support it.

7. Summary & Next Steps

Key Takeaways

  • Feature extraction freezes all pretrained layers and trains only a new final layer — fast, safe for small datasets, but cannot adapt pretrained features to the new task at all.
  • Fine-tuning unfreezes some or all pretrained layers, requiring a much lower learning rate to avoid destroying the useful pretrained representations with large, disruptive updates.
  • Gradual unfreezing progressively unfreezes layers from the end of the network backward, since late layers are most task-specific and safest to adapt first.
  • The choice between feature extraction and fine-tuning depends on dataset size and how similar the new task is to the original pretraining task, directly echoing the ML Notes' bias-variance tradeoff.

Concept Check

  1. Why does fine-tuning require a much lower learning rate than training a randomly-initialized network from scratch?
  2. Why does gradual unfreezing proceed from the last layers to the first, rather than the reverse?
  3. If you have a very small, task-different dataset, which strategy from this chapter would you choose, and why?

Next Chapter

→ Chapter 3: Pretrained Models & Model Hubs

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index