Transfer Learning And Practical Training
Why Train From Scratch Rarely Makes Sense
Every example so far in this curriculum (Module 2's from-scratch network, Module 4's CNN) initialized weights randomly (Module 3, Chapter 1) and trained on a sp
Jr Codex Deep Learning Notes
Level: Intermediate–Advanced Prerequisites: Module 6: Attention & Transformers Time to complete: ~15 minutes
Table of Contents
- What "Training From Scratch" Actually Requires
- The Core Insight Behind Transfer Learning
- Evidence: What Early Layers Actually Learn
- When Training From Scratch DOES Make Sense
- The Transfer Learning Spectrum
- Summary & Next Steps
1. What "Training From Scratch" Actually Requires
Every example so far in this curriculum (Module 2's from-scratch network, Module 4's CNN) initialized weights randomly (Module 3, Chapter 1) and trained on a specific dataset. For state-of-the-art architectures — ResNet-152, or a Transformer with billions of parameters (Module 6) — this requires:
The Real Cost of Training From Scratch
─────────────────────────────────────────
MASSIVE datasets: ImageNet (14 million images) for
vision; hundreds of billions of
tokens of text for large language
models
MASSIVE compute: training a large modern model can
cost from thousands to tens of
millions of dollars in GPU/TPU
compute time
Substantial EXPERTISE: getting Module 3's training
techniques right at this
scale (optimizers, learning
rate schedules, distributed
training across many GPUs)
requires deep, specialized
knowledge
─────────────────────────────────────────
For the overwhelming majority of practical projects, none of this is necessary.
2. The Core Insight Behind Transfer Learning
Transfer learning is based on a simple but powerful observation: a network trained on one large, general task learns representations — especially in its early layers — that are broadly useful for many related tasks, not just the one it was originally trained on.
The Insight, Concretely
─────────────────────────────────────────
A CNN (Module 4) trained on ImageNet's 1000 general object
categories learns early-layer filters that detect EDGES,
TEXTURES, and simple SHAPES (Module 4, Ch.1, Section 3) —
these are USEFUL for recognizing almost ANY visual object,
not just the specific 1000 ImageNet categories. A network
trained to classify medical X-rays would benefit from
STARTING with these same general visual features, rather
than re-discovering "edges are useful" completely from
scratch.
─────────────────────────────────────────
3. Evidence: What Early Layers Actually Learn
This isn't just a theoretical claim — visualizing learned filters (Module 4, Chapter 3, Section 6) across many different trained CNNs consistently shows the same pattern.
A Consistent Empirical Finding
─────────────────────────────────────────
EARLY layers: learn nearly IDENTICAL features across
VERY different final tasks — edge
detectors, color blob detectors,
simple Gabor-filter-like patterns —
regardless of whether the final task
is classifying dogs, cars, or X-rays
LATE layers: learn increasingly TASK-SPECIFIC
features — specific to the exact
categories the network was
trained to distinguish
─────────────────────────────────────────
This finding directly justifies transfer learning's core strategy (Chapter 2): reuse the general early layers, and only retrain (or lightly adjust) the task-specific late layers.
4. When Training From Scratch DOES Make Sense
Legitimate Reasons to Train From Scratch
─────────────────────────────────────────
Your data is FUNDAMENTALLY different: e.g. non-visual
sensor data with
NO resemblance to
natural images —
pretrained vision
models' learned
features may not
transfer usefully
at all
You have an ENORMOUS dataset AND if you have
compute budget of your own: millions of
task-specific
examples and
the compute to
match, training
from scratch
CAN outperform
transfer
learning,
though it
rarely
justifies the
added cost
You're doing FOUNDATIONAL research into new
on new architectures: architectures
themselves —
the whole
POINT is
studying
training
from scratch
─────────────────────────────────────────
For nearly every applied project — the vast majority of real-world deep learning work — transfer learning is the correct default starting point, echoing the ML Notes' "start simple" workflow principle applied at the level of "don't build your own foundation model when a good one already exists."
5. The Transfer Learning Spectrum
From "Reuse Everything" to "Retrain Everything"
─────────────────────────────────────────
Feature extraction FINE-TUNING Training from
(Chapter 2) (Chapter 2) scratch
│ │ │
Freeze ALL pretrained Unfreeze SOME or Random
layers; train ONLY a ALL pretrained initialization
new final layer layers; continue (Module 3,
training at a LOW Ch.1);
learning rate train
everything
LEAST compute, MODERATE compute, MOST compute,
LEAST data needed MODERATE data needed MOST data needed
─────────────────────────────────────────
This spectrum — covered in full practical detail in Chapter 2 — lets a practitioner choose exactly how much of a pretrained model to adapt, based on how much task-specific data is available and how similar the new task is to the original training task.
6. Summary & Next Steps
Key Takeaways
- Training modern architectures from scratch requires massive datasets, compute, and expertise that most practical projects don't have and don't need.
- Transfer learning exploits the finding that early network layers learn broadly general features (edges, textures), while only late layers become task-specific.
- Legitimate reasons to train from scratch include fundamentally different data domains, access to enormous task-specific datasets and compute, or foundational architecture research.
- Transfer learning exists on a spectrum from pure feature extraction (freeze everything, train only a new final layer) to full fine-tuning (retrain most or all layers at a low learning rate).
Concept Check
- What specific empirical finding about early vs late network layers justifies transfer learning as a strategy?
- Name two legitimate reasons a practitioner might choose to train a model from scratch instead of using transfer learning.
- What are the two ends of the transfer learning spectrum, and what determines where on that spectrum a project should land?
Next Chapter
→ Chapter 2: Feature Extraction vs Fine-Tuning
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index