Tokenization And Pretraining Foundations
Pre-training Objectives: MLM vs CLM
Every model in the ML Notes and most of the DL Notes required labeled data. Modern LLMs are pretrained on hundreds of billions to trillions of words of raw text
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 2: Applying the Transformer to Language Time to complete: ~25 minutes
Table of Contents
- The Core Problem: No Labels at This Scale
- Self-Supervised Learning, Revisited
- Masked Language Modeling (MLM)
- Causal Language Modeling (CLM)
- MLM vs CLM — Why the Choice Determines Everything Downstream
- Pre-training Scale, in Perspective
- From Pre-training to a Usable Model
- Summary & Next Steps
1. The Core Problem: No Labels at This Scale
Every model in the ML Notes and most of the DL Notes required labeled data. Modern LLMs are pretrained on hundreds of billions to trillions of words of raw text — labeling data at that scale by hand is completely infeasible. Pre-training objectives solve this by deriving labels automatically from the raw text itself.
2. Self-Supervised Learning, Revisited
Module 2, Chapter 2's Word2Vec and DL Notes, Module 8, Chapter 1's autoencoders both introduced this same idea: construct a prediction task where the "label" comes directly from the input data, requiring no human annotation at all.
The Pattern, Restated for Language
─────────────────────────────────────────
Take a piece of RAW, unlabeled text. HIDE or REMOVE some
part of it. Train the model to PREDICT the hidden/removed
part, using the REST of the text as its only clue. The
"label" is simply the ORIGINAL text that was hidden — free,
and available in effectively unlimited quantity from the
internet, books, and other text sources.
─────────────────────────────────────────
The two dominant ways of applying this pattern — Masked Language Modeling and Causal Language Modeling — directly determine whether a model becomes encoder-only (BERT-style) or decoder-only (GPT-style), per Chapter 2, Section 5.
3. Masked Language Modeling (MLM)
MLM, used to pretrain BERT (Module 4), randomly masks a percentage of tokens in the input, then trains the model to predict what those masked tokens originally were, using the FULL surrounding context (both before AND after each masked position).
import torch
def create_mlm_example(tokenizer, text, mask_probability=0.15):
encoded = tokenizer(text, return_tensors="pt")
input_ids = encoded["input_ids"].clone()
labels = input_ids.clone()
# Randomly select 15% of tokens to mask (BERT's original choice)
mask_positions = torch.rand(input_ids.shape) < mask_probability
mask_positions &= (input_ids != tokenizer.cls_token_id) & (input_ids != tokenizer.sep_token_id)
labels[~mask_positions] = -100 # -100 is IGNORED by the loss function — only masked
# positions contribute to training
input_ids[mask_positions] = tokenizer.mask_token_id # replace with [MASK]
return input_ids, labels
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
masked_input, labels = create_mlm_example(tokenizer, "The cat sat on the mat")
print(tokenizer.decode(masked_input[0]))
# e.g. "[CLS] the cat sat on the [MASK] [SEP]" — the model must predict "mat"
# using BOTH the words BEFORE ("the cat sat on the") and, in a longer
# example, potentially words AFTER the masked position tooWhy MLM Requires an ENCODER-ONLY (Unmasked) Architecture
─────────────────────────────────────────
To predict a masked word USING context from BOTH directions
(before AND after), the model's self-attention must be
ALLOWED to look at EVERY position, including ones AFTER the
masked token — this is EXACTLY the unmasked, bidirectional
self-attention of an encoder-only Transformer (DL Notes,
Module 6, Chapter 4, Section 7) — a decoder's CAUSAL masking
(Section 4) would make this task IMPOSSIBLE, since it
couldn't look ahead to see what follows the masked word.
─────────────────────────────────────────
4. Causal Language Modeling (CLM)
CLM, used to pretrain the GPT family (Module 5), trains the model to predict the next token, given only the tokens that came before it — directly the autoregressive generation pattern from DL Notes, Module 5, Chapter 4.
def create_clm_example(tokenizer, text):
encoded = tokenizer(text, return_tensors="pt")
input_ids = encoded["input_ids"]
# The "label" at each position is simply the NEXT token —
# shift the sequence by ONE position
labels = input_ids.clone()
labels[:, :-1] = input_ids[:, 1:]
labels[:, -1] = -100 # no "next token" exists for the LAST position
return input_ids, labels
from transformers import AutoTokenizer
gpt_tokenizer = AutoTokenizer.from_pretrained("gpt2")
input_ids, labels = create_clm_example(gpt_tokenizer, "The cat sat on the mat")
print(f"Input: {gpt_tokenizer.decode(input_ids[0])}")
print(f"Predicting: at position i, the label is the token at position i+1")
# The model learns: given "The", predict "cat"; given "The cat", predict "sat"; etc.
# — EVERY position in the sequence provides a training signal SIMULTANEOUSLYWhy CLM Requires a DECODER-ONLY (Masked) Architecture
─────────────────────────────────────────
If the model could see FUTURE tokens while predicting the
next one, it would trivially "cheat" by just copying the
answer directly from later in the sequence — MASKED
self-attention (DL Notes, Module 6, Chapter 2, Section 6)
strictly PREVENTS this, forcing genuine next-token
prediction using ONLY prior context — exactly matching how
the model must generate text at INFERENCE time (Chapter 2,
Section 5's decoder-only framing).
─────────────────────────────────────────
5. MLM vs CLM — Why the Choice Determines Everything Downstream
A Direct Comparison
─────────────────────────────────────────
MLM (BERT-style, Module 4): BIDIRECTIONAL context — sees
the WHOLE input at once —
excellent for UNDERSTANDING
tasks (classification,
extraction, Module 4) — but
CANNOT naturally generate
text left-to-right
CLM (GPT-style, Module 5): UNIDIRECTIONAL context —
sees only PRIOR tokens —
naturally suited to
GENERATION (Module 5) —
somewhat LESS efficient
at pure understanding
tasks, since it can't
use FUTURE context
directly
─────────────────────────────────────────
This is precisely why the field has largely converged on decoder-only, CLM-pretrained architectures for general-purpose LLMs (Module 5) — generation is the more flexible capability, and a sufficiently capable generative model can ALSO perform understanding tasks (by generating the appropriate label as text), while the reverse isn't naturally true for a pure encoder.
6. Pre-training Scale, in Perspective
The Scale Involved
─────────────────────────────────────────
BERT (2018): pretrained on ~3.3 billion words
(Wikipedia + BookCorpus)
GPT-3 (2020): pretrained on ~300 billion tokens
Modern frontier models pretrained on TRILLIONS of tokens
(2023+): — scraped from the web, books,
code repositories, and other
large text sources
This directly connects to DL Notes, Module 1, Chapter 3's
"data" enabler of deep learning's practical rise — LLM
pre-training is, in many ways, that same principle taken to
its logical extreme.
─────────────────────────────────────────
7. From Pre-training to a Usable Model
The Full Lifecycle (Module 6 Covers Steps 2-3 in Depth)
─────────────────────────────────────────
1. PRE-TRAINING (this chapter): MLM or CLM on massive,
raw, UNLABELED text —
produces a model with
general language
understanding/generation
ability, but NO specific
task focus or
conversational behavior
2. FINE-TUNING / adapting the pretrained
INSTRUCTION TUNING model to specific
(Module 6): tasks or to follow
instructions,
using SMALLER,
often labeled
datasets
3. ALIGNMENT (RLHF, further shaping
Module 6): the model's
behavior to
be helpful,
harmless, and
honest, using
human feedback
─────────────────────────────────────────
A raw, freshly pretrained model (step 1 alone) is often surprisingly not very useful conversationally — it simply continues text in statistically plausible ways, without any inherent notion of "answering a question helpfully." Steps 2-3, covered in Module 6, are what transform a raw pretrained model into something like ChatGPT.
8. Summary & Next Steps
Key Takeaways
- Pre-training objectives derive training labels automatically from raw text itself, avoiding the need for human annotation at the massive scale LLMs require.
- Masked Language Modeling (MLM) predicts randomly masked tokens using bidirectional context, requiring an encoder-only architecture — the basis for BERT.
- Causal Language Modeling (CLM) predicts the next token using only prior context, requiring a decoder-only (causally masked) architecture — the basis for GPT.
- A raw pretrained model is not inherently conversational or helpful; fine-tuning and alignment (Module 6) are what transform it into a usable assistant.
Module 3 Complete — What's Next
You now understand exactly how raw text becomes model input (tokenization), how the Transformer architecture applies to language (a direct extension of the DL Notes), and how models learn from raw text at massive scale (MLM/CLM). Module 4 covers BERT and the encoder-only family in full depth — architecture, pretraining specifics, and practical fine-tuning for downstream tasks.
Concept Check
- Why does MLM require bidirectional (unmasked) self-attention, while CLM requires causally masked self-attention?
- Why can't a freshly pretrained model (with no fine-tuning or alignment) reliably act as a helpful conversational assistant?
- Why has the field largely converged on decoder-only, CLM-pretrained architectures for general-purpose LLMs?
Next Module
→ Module 4: Encoder Models — BERT & Friends
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index