Decoder Models Gpt And Modern Llms
The Modern LLM Landscape
This chapter deliberately focuses on categories and tradeoffs rather than specific model names and benchmark numbers, which change too quickly to remain accurat
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 3: In-Context Learning & Emergent Abilities Time to complete: ~20 minutes
Table of Contents
- A Rapidly Evolving Landscape
- Closed vs Open Models
- Major Model Families, at an Orientation Level
- Mixture of Experts — a Notable Architectural Variant
- Multimodal Models
- Choosing a Model for a Project
- Summary & Next Steps
1. A Rapidly Evolving Landscape
This chapter deliberately focuses on categories and tradeoffs rather than specific model names and benchmark numbers, which change too quickly to remain accurate for long. The underlying architectural concepts — from Chapter 1's decoder-only Transformer through Chapter 2's scaling laws — remain the stable foundation beneath whichever specific models are currently state-of-the-art.
2. Closed vs Open Models
Two Broad Categories
─────────────────────────────────────────
Closed (proprietary) models: accessed ONLY via an API —
weights are NOT released;
typically the LARGEST,
highest-performing models
at any given time; usage
incurs PER-TOKEN cost
(Module 9)
Open(-weight) models: model weights are
PUBLICLY downloadable
(though "open source"
in the FULL sense —
including training
data and code — is
rarer) — can be
self-hosted, fine-tuned
(Module 6), and run
WITHOUT per-token API
costs
─────────────────────────────────────────
The Practical Tradeoff
─────────────────────────────────────────
Closed models: generally STRONGER out-of-the-box
performance, zero infrastructure
burden, but ONGOING cost, less control,
and DEPENDENCY on a third party's
availability/pricing/policies
Open models: requires OWN infrastructure (DL Notes,
Module 9's GPU/serving concerns) or
a hosting provider, but full CONTROL,
one-time (rather than per-token)
cost at scale, and the ability to
fully fine-tune (Module 6) on
proprietary/sensitive data without
sending it to a third party
─────────────────────────────────────────
3. Major Model Families, at an Orientation Level
Categories Worth Knowing (Without Tying to Specific, Fast-Changing Names)
─────────────────────────────────────────
Frontier closed models: the LARGEST, most capable models
from major AI labs, typically
accessed via API — usually
leading published benchmarks
Open-weight "frontier- large, capable OPEN-weight
adjacent" models: models, released by various
labs — increasingly
competitive with closed
models on many benchmarks
Small, efficient open models designed to run on
models: MODEST hardware (even a
laptop or phone) —
trading some capability
for dramatically lower
resource requirements
— often fine-tuned
(Module 6) for a
NARROW, specific use
case
─────────────────────────────────────────
4. Mixture of Experts — a Notable Architectural Variant
Some modern large models use a Mixture of Experts (MoE) architecture — a genuine variation on Chapter 1's standard decoder, worth understanding at a conceptual level.
The MoE Idea
─────────────────────────────────────────
Instead of EVERY token passing through the SAME single
feed-forward sub-layer (DL Notes, Module 6, Ch.4, Sec.5),
an MoE layer contains MULTIPLE separate feed-forward
networks ("experts"), and a small ROUTER network decides
which FEW experts (e.g. 2 out of 8) should process each
specific token.
─────────────────────────────────────────
import torch
import torch.nn as nn
class SimplifiedMoELayer(nn.Module):
def __init__(self, d_model, d_ff, num_experts=8, top_k=2):
super().__init__()
self.experts = nn.ModuleList([
nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model))
for _ in range(num_experts)
])
self.router = nn.Linear(d_model, num_experts) # decides WHICH experts to use
self.top_k = top_k
def forward(self, x):
router_logits = self.router(x)
top_k_weights, top_k_indices = torch.topk(router_logits, self.top_k, dim=-1)
top_k_weights = torch.softmax(top_k_weights, dim=-1) # DL Notes, Module 2, Ch.2
output = torch.zeros_like(x)
for i in range(self.top_k):
expert_idx = top_k_indices[..., i]
weight = top_k_weights[..., i].unsqueeze(-1)
for e, expert in enumerate(self.experts):
mask = (expert_idx == e)
if mask.any():
output[mask] += weight[mask] * expert(x[mask])
return outputWhy MoE Matters: More Parameters, Similar Compute
─────────────────────────────────────────
A model with 8 experts has roughly 8x the TOTAL parameters
of a single-expert equivalent — but since only "top_k"
(e.g. 2) experts process any given token, the COMPUTE cost
per token stays much closer to a MUCH smaller dense model's
cost. This lets MoE models achieve very large total
parameter counts (and correspondingly large capacity) while
keeping INFERENCE compute more manageable — a genuinely
different scaling lever than Chapter 2's "just make the
dense model bigger" approach.
─────────────────────────────────────────
5. Multimodal Models
Many modern LLMs extend beyond pure text, accepting images (and sometimes audio/video) as input alongside text — directly connecting the DL Notes' Vision Transformer discussion to this curriculum's language-focused architecture.
The General Approach
─────────────────────────────────────────
An image is split into patches and embedded (DL Notes,
Module 6, Ch.5's ViT approach), producing a sequence of
"visual tokens" — these are projected into the SAME
embedding space as TEXT tokens (Chapter 1) and processed by
the SAME (or a closely related) Transformer decoder, letting
self-attention relate visual and textual information
directly, within a single unified sequence.
─────────────────────────────────────────
This is a direct, practical illustration of a point made repeatedly throughout the DL Notes: the Transformer is a general-purpose sequence architecture, not language-specific — multimodal models are simply feeding it a sequence that mixes token types.
6. Choosing a Model for a Project
A Practical Decision Framework
─────────────────────────────────────────
Need the ABSOLUTE best performance, Frontier closed
cost is secondary: model, via API
Need FULL control, or handling Open-weight
sensitive/proprietary data: model,
self-hosted
Need to run on LIMITED hardware Small,
(edge devices, modest servers): efficient
open model
Need a MODEL fine-tuned Open-
(Module 6) for a narrow, weight
specific task: model
(full
fine-
tuning
access
required)
Need MULTIMODAL (image + text) A
understanding: multimodal-
capable
model
(Section 5)
─────────────────────────────────────────
This decision directly connects to Module 9's production considerations — cost, latency, and data sensitivity are exactly the factors that should drive this choice in a real project.
7. Summary & Next Steps
Key Takeaways
- Closed models offer strong out-of-the-box performance with no infrastructure burden but ongoing per-token cost and less control; open-weight models offer full control and fine-tuning access at the cost of self-managed infrastructure.
- Mixture of Experts architectures route each token to only a few specialized sub-networks, achieving large total parameter counts while keeping per-token inference compute manageable.
- Multimodal models extend the same Transformer architecture to process images and text together, by projecting visual patches into the same embedding space as text tokens.
- Choosing a model for a project should be driven by performance needs, cost constraints, data sensitivity, and whether fine-tuning or multimodal capability is required.
Module 5 Complete — What's Next
You've now covered the complete decoder-only family: GPT's architecture and autoregressive generation, the scaling laws that drove its evolution, in-context learning's mechanism, and the modern landscape of open and closed models. Module 6 covers how raw pretrained models like these are actually adapted into helpful, aligned assistants — instruction tuning, parameter-efficient fine-tuning, and RLHF.
Concept Check
- What is the fundamental tradeoff between choosing a closed model versus an open-weight model for a project?
- How does a Mixture of Experts layer keep per-token compute manageable despite having a much larger total parameter count?
- How do multimodal models fit images into the same architecture used for pure text processing?
Next Module
→ Module 6: Fine-Tuning & Aligning LLMs
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index