Tokenization And Pretraining Foundations
Subword Tokenization: BPE, WordPiece, SentencePiece
Module 1, Chapter 2's word-level tokenization and Module 2's word embeddings both assumed a fixed vocabulary — every word the model will ever see must have been
Jr Codex NLP & LLM Notes
Level: Intermediate Prerequisites: Module 2: Word Embeddings & Representations Time to complete: ~25 minutes
Table of Contents
- Why Whole-Word Tokenization Breaks Down
- The Subword Insight
- Byte-Pair Encoding (BPE)
- WordPiece
- SentencePiece
- Tokenization in Practice With HuggingFace
- Vocabulary Size — a Genuine Design Tradeoff
- Summary & Next Steps
1. Why Whole-Word Tokenization Breaks Down
Module 1, Chapter 2's word-level tokenization and Module 2's word embeddings both assumed a fixed vocabulary — every word the model will ever see must have been seen during training, with its own dedicated embedding row (Module 2, Chapter 1, Section 6).
The Out-of-Vocabulary (OOV) Problem
─────────────────────────────────────────
What happens when a model encounters "unfriendliness,"
a MADE-UP word like "Blorptastic," or a RARE technical term
never seen during training? A whole-word vocabulary has NO
entry for it — the model has NOTHING to look up (Module 2,
Chapter 1's embedding table simply doesn't have a row for
words outside its FIXED vocabulary).
─────────────────────────────────────────
2. The Subword Insight
Subword tokenization solves this by breaking words into smaller, more frequent pieces — so that even a completely novel word can usually be represented as a combination of familiar sub-parts.
Whole-Word vs Subword Tokenization
─────────────────────────────────────────
Whole-word: "unfriendliness" → EITHER a single token
(if seen during training) OR an
unrepresentable "unknown" token
Subword: "unfriendliness" → ["un", "friend",
"li", "ness"] — even if the FULL word
was never seen, its PIECES likely were,
since "un-", "friend", "-li-", "-ness"
are all common across MANY other words
─────────────────────────────────────────
This single idea is what allows every modern LLM (Modules 4-5) to handle typos, rare technical terms, made-up words, and even other languages, gracefully — without ever hitting a hard "unknown word" wall.
3. Byte-Pair Encoding (BPE)
BPE (adapted for NLP from a 1994 data compression algorithm) builds a subword vocabulary by iteratively merging the most frequently co-occurring pair of symbols.
The BPE Algorithm
─────────────────────────────────────────
1. Start with a vocabulary of INDIVIDUAL CHARACTERS
2. Count all ADJACENT symbol pairs across the training corpus
3. MERGE the single MOST FREQUENT pair into a new symbol
4. REPEAT steps 2-3 a fixed number of times (determining the
final vocabulary size)
─────────────────────────────────────────
from collections import defaultdict, Counter
def get_pair_frequencies(word_frequencies):
pairs = defaultdict(int)
for word, freq in word_frequencies.items():
symbols = word.split()
for i in range(len(symbols) - 1):
pairs[(symbols[i], symbols[i+1])] += freq
return pairs
def merge_pair(pair, word_frequencies):
new_word_frequencies = {}
bigram = " ".join(pair)
replacement = "".join(pair)
for word, freq in word_frequencies.items():
new_word = word.replace(bigram, replacement)
new_word_frequencies[new_word] = freq
return new_word_frequencies
# Starting vocabulary: characters, with a special end-of-word marker
word_frequencies = {
"l o w </w>": 5, "l o w e r </w>": 2, "n e w e s t </w>": 6, "w i d e s t </w>": 3,
}
num_merges = 10
for i in range(num_merges):
pairs = get_pair_frequencies(word_frequencies)
if not pairs:
break
best_pair = max(pairs, key=pairs.get) # the MOST FREQUENT pair
word_frequencies = merge_pair(best_pair, word_frequencies)
print(f"Merge {i+1}: {best_pair} -> {''.join(best_pair)}")Why Starting From Characters and MERGING Works
─────────────────────────────────────────
Common patterns (like "e s t" appearing in BOTH "newest" and
"widest") get MERGED early, since they're highly FREQUENT —
producing subword units like "est" that generalize across
many words. RARE words, meanwhile, simply remain SPLIT into
smaller, more common pieces, rather than needing their OWN
dedicated vocabulary entry.
─────────────────────────────────────────
BPE is the tokenization scheme used by GPT-2, GPT-3, and most of the GPT family (Module 5).
4. WordPiece
WordPiece, used by BERT (Module 4), is closely related to BPE but merges pairs based on a slightly different criterion: it picks the merge that maximizes the likelihood of the training data, not simply the raw frequency count.
BPE vs WordPiece — a Subtle but Real Difference
─────────────────────────────────────────
BPE (Section 3): merges the pair with the HIGHEST RAW
FREQUENCY count
WordPiece: merges the pair that most improves
the LANGUAGE MODEL's likelihood of
the training data — accounting for
how frequent each INDIVIDUAL symbol
already is, not just the PAIR's
raw count
─────────────────────────────────────────
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("unfriendliness")
print(tokens) # e.g. ['un', '##friend', '##li', '##ness']
# The "##" prefix marks a subword that CONTINUES the previous token,
# rather than starting a NEW word — a WordPiece-specific convention5. SentencePiece
SentencePiece solves a subtler problem: BPE and WordPiece both assume text is already split into words by whitespace — which fails for languages like Chinese or Japanese that don't use spaces between words at all.
SentencePiece's Key Difference
─────────────────────────────────────────
Instead of pre-splitting on whitespace FIRST (as BPE/WordPiece
implicitly assume), SentencePiece treats the input as a RAW,
UNSEGMENTED stream of characters (including spaces
themselves, represented as a special character, often "▁")
— making it LANGUAGE-AGNOSTIC and usable identically for
space-separated languages (English) and non-space-separated
languages (Chinese, Japanese, Thai) alike.
─────────────────────────────────────────
import sentencepiece as spm
# Training a SentencePiece model (requires a text file as input, illustrated conceptually)
# spm.SentencePieceTrainer.train(input="corpus.txt", model_prefix="tokenizer", vocab_size=8000)
# Using a PRETRAINED SentencePiece model (typically bundled with a HuggingFace tokenizer)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased") # uses SentencePiece internally
tokens = tokenizer.tokenize("Tokenization is fascinating")
print(tokens) # e.g. ['▁Token', 'ization', '▁is', '▁fascinat', 'ing']6. Tokenization in Practice With HuggingFace
from transformers import AutoTokenizer
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") # WordPiece
gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2") # BPE
text = "Tokenizers handle unfamiliar words gracefully."
print("BERT (WordPiece):", bert_tokenizer.tokenize(text))
print("GPT-2 (BPE):", gpt2_tokenizer.tokenize(text))
# Converting tokens to the NUMERICAL IDs a model actually consumes
input_ids = bert_tokenizer(text, return_tensors="pt")
print(input_ids["input_ids"])
print(bert_tokenizer.decode(input_ids["input_ids"][0])) # converting BACK to textSpecial Tokens Every Tokenizer Adds
─────────────────────────────────────────
[CLS] / <s>: marks the START of a sequence — often used
as an aggregate representation for
classification (Module 4)
[SEP] / </s>: marks the END of a sequence, or
separates TWO sequences (e.g. a
question and a passage)
[PAD]: pads shorter sequences to match a
BATCH's longest sequence (DL
Notes, Module 5, Ch.1's
variable-length problem, solved
practically here)
[UNK]: a fallback for any character/
symbol truly outside the
tokenizer's vocabulary — RARE
with subword tokenization,
but not IMPOSSIBLE
─────────────────────────────────────────
7. Vocabulary Size — a Genuine Design Tradeoff
Small Vocabulary vs Large Vocabulary
─────────────────────────────────────────
SMALL vocabulary words get split into MANY small
(e.g. 10,000 tokens): pieces — sequences become
LONGER (more tokens per
sentence), but the embedding
table (Module 2, Ch.1) is
SMALLER
LARGE vocabulary words stay MORE INTACT —
(e.g. 50,000+ tokens): sequences are SHORTER, but
the embedding table is
LARGER, and rare tokens
get LESS training signal
each
─────────────────────────────────────────
Modern LLMs typically use vocabularies in the 30,000-100,000+ token range, balancing these considerations — longer sequences directly increase compute cost, given self-attention's quadratic scaling with sequence length (DL Notes, Module 6, Chapter 5).
8. Summary & Next Steps
Key Takeaways
- Subword tokenization solves the out-of-vocabulary problem by breaking words into smaller, frequent pieces, so novel words can be represented as combinations of familiar sub-parts.
- BPE (used by GPT models) iteratively merges the most frequent adjacent symbol pairs; WordPiece (used by BERT) merges based on likelihood improvement rather than raw frequency.
- SentencePiece treats input as a raw character stream rather than pre-splitting on whitespace, making it usable identically across space-separated and non-space-separated languages.
- Vocabulary size is a genuine tradeoff between sequence length (affecting compute cost, given attention's quadratic scaling) and embedding table size/per-token training signal.
Concept Check
- What specific problem does subword tokenization solve that whole-word tokenization cannot?
- What's the key difference between how BPE and WordPiece decide which pair to merge next?
- Why is SentencePiece necessary for languages like Chinese or Japanese, where BPE/WordPiece's whitespace assumption breaks down?
Next Chapter
→ Chapter 2: Applying the Transformer to Language
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index