Decoder Models Gpt And Modern Llms
Scaling Laws & the GPT Family's Evolution
DL Notes, Module 1, Chapter 3 and Module 6, Chapter 5 both mentioned that Transformers scale unusually well. The GPT family is the clearest empirical demonstrat
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 1: GPT — Architecture & Autoregressive Language Modeling Time to complete: ~20 minutes
Table of Contents
- The GPT Family's Trajectory
- GPT-1 (2018): Proof of Concept
- GPT-2 (2019): Scale Reveals Surprising Ability
- GPT-3 (2020): Scale Becomes the Story
- Scaling Laws, Precisely
- Why Scaling Laws Matter Practically
- The Limits of Pure Scaling
- Summary & Next Steps
1. The GPT Family's Trajectory
DL Notes, Module 1, Chapter 3 and Module 6, Chapter 5 both mentioned that Transformers scale unusually well. The GPT family is the clearest empirical demonstration of this — each generation is fundamentally the same architecture from Chapter 1, just larger, trained on more data, with more compute.
2. GPT-1 (2018): Proof of Concept
GPT-1's Contribution
─────────────────────────────────────────
Size: 117 million parameters
Key idea: demonstrated that a decoder-only Transformer,
pretrained via CLM (Module 3, Ch.3) on
unlabeled text, then fine-tuned (DL Notes,
Module 7) on specific tasks, could match or
beat task-specific architectures — the SAME
pretrain-then-fine-tune pattern BERT (Module
4) would apply on the encoder side, just
months later
─────────────────────────────────────────
3. GPT-2 (2019): Scale Reveals Surprising Ability
GPT-2's Contribution
─────────────────────────────────────────
Size: 1.5 billion parameters (~13x larger than GPT-1)
Key finding: at this scale, the model showed SURPRISING
"zero-shot" ability — performing REASONABLY
on tasks it was NEVER explicitly fine-tuned
for, purely from its CLM pretraining, simply
by framing the task as TEXT to continue
(e.g. "Translate to French: hello ->" and
letting the model CONTINUE the pattern)
─────────────────────────────────────────
This zero-shot finding was the first strong signal that scale itself — not just architecture — could unlock qualitatively new capabilities, foreshadowing GPT-3's far more dramatic demonstration of the same principle.
4. GPT-3 (2020): Scale Becomes the Story
GPT-3's Contribution
─────────────────────────────────────────
Size: 175 billion parameters (~117x larger than GPT-2)
Key finding: strong "FEW-SHOT" learning — given just a
HANDFUL of examples of a task WITHIN the
prompt itself (no fine-tuning, no gradient
updates at all), GPT-3 could perform the
task competently — this is called IN-CONTEXT
LEARNING, covered in full in Chapter 3
─────────────────────────────────────────
GPT-3's headline result was less about any single architectural innovation, and much more about demonstrating that scale itself was a genuinely powerful lever — directly validating the scaling-law predictions covered next.
5. Scaling Laws, Precisely
Research published alongside and following GPT-2/3 (notably by Kaplan et al., 2020) found remarkably predictable, smooth relationships between model performance and three factors.
The Three Scaling Dimensions
─────────────────────────────────────────
N — Model size: the number of parameters
D — Dataset size: the amount of training data (tokens)
C — Compute: the total computation used for
training (roughly, N × D)
─────────────────────────────────────────
The Key Empirical Finding
─────────────────────────────────────────
Loss (a direct measure of the model's next-token prediction
quality — Module 3, Ch.3) DECREASES smoothly and
PREDICTABLY as a POWER LAW with N, D, and C — plotted on a
log-log scale, these relationships appear as remarkably
straight lines, across MANY orders of magnitude.
This means: given a model's performance at a SMALL scale,
researchers can reasonably PREDICT how much better a
LARGER version will perform, BEFORE training it — a
genuinely unusual and valuable property, rarely seen this
cleanly in other areas of machine learning.
─────────────────────────────────────────
The Chinchilla Refinement (2022)
─────────────────────────────────────────
A later, influential paper (DeepMind's "Chinchilla") found
that many EARLIER large models (including the original
GPT-3) were actually UNDER-trained relative to their SIZE
— for a GIVEN compute budget, TRAINING A SMALLER MODEL ON
MORE DATA often outperforms training a LARGER model on LESS
data. This directly reshaped how compute budgets are
allocated when training modern LLMs.
─────────────────────────────────────────
6. Why Scaling Laws Matter Practically
Practical Consequences
─────────────────────────────────────────
Predictable ROI: an organization can estimate the
performance gain from a LARGER
training run BEFORE committing the
(often enormous) compute cost
Justifies MASSIVE the smooth, predictable curve is
investment: part of why organizations
committed to training
increasingly large models —
the returns, while requiring
exponentially MORE compute,
were NOT diminishing in the
way many other techniques
eventually plateau
Guides DATA vs SIZE the Chinchilla finding directly
tradeoffs: informs whether it's better
to scale UP a model or
gather MORE training data,
for a FIXED compute budget
─────────────────────────────────────────
7. The Limits of Pure Scaling
Scaling Alone Doesn't Solve Everything
─────────────────────────────────────────
A raw, CLM-pretrained model — however large — is STILL just
predicting plausible NEXT tokens (Module 3, Ch.3, Chapter 1,
Section 3), with NO inherent alignment toward being helpful,
truthful, or safe. Module 6's fine-tuning and RLHF ARE what
transform raw scale into the conversational, aligned
assistants users actually interact with — scale provides the
RAW CAPABILITY, but not, by itself, the DESIRED BEHAVIOR.
─────────────────────────────────────────
Additionally, high-quality training data is itself a finite resource — some analyses suggest the highest-quality publicly available text may be approaching practical limits, motivating research into synthetic data generation and more data-efficient training techniques as complements to pure scale.
8. Summary & Next Steps
Key Takeaways
- The GPT family (GPT-1 through GPT-3) is architecturally consistent — each generation is fundamentally the same decoder-only Transformer, scaled up in size, data, and compute.
- GPT-2 revealed surprising zero-shot task performance from scale alone; GPT-3 demonstrated strong few-shot, in-context learning without any fine-tuning.
- Scaling laws show model loss decreases smoothly and predictably as a power law with model size, data, and compute — an unusually clean empirical relationship.
- The Chinchilla finding showed many large models were under-trained relative to their size, reshaping how compute is allocated between model size and data quantity; scale alone doesn't produce alignment or helpfulness, which require the fine-tuning covered in Module 6.
Concept Check
- What is the key difference between GPT-2's zero-shot ability and GPT-3's few-shot, in-context learning?
- What did the Chinchilla paper find about how earlier large models allocated their compute budgets?
- Why doesn't scale alone guarantee a model is helpful, truthful, or safe?
Next Chapter
→ Chapter 3: In-Context Learning & Emergent Abilities
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index