Tokenization And Pretraining Foundations
Applying the Transformer to Language
This chapter is intentionally short. Its entire purpose is connecting Chapter 1's tokenization output to DL Notes, Module 6's Transformer architecture, which is
Jr Codex NLP & LLM Notes
Level: Intermediate–Advanced Prerequisites: Chapter 1: Subword Tokenization Time to complete: ~20 minutes
Table of Contents
- A Deliberate Bridge Chapter
- From Tokens to Model Input
- The Complete Input Pipeline
- Recap: What the Transformer Does From Here
- Encoder-Only vs Decoder-Only, Revisited for Language
- Why This Curriculum Doesn't Re-Derive Attention
- Summary & Next Steps
1. A Deliberate Bridge Chapter
This chapter is intentionally short. Its entire purpose is connecting Chapter 1's tokenization output to DL Notes, Module 6's Transformer architecture, which is assumed as a complete prerequisite here — not re-derived.
2. From Tokens to Model Input
from transformers import AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "The cat sat on the mat"
encoded = tokenizer(text, return_tensors="pt")
print(encoded["input_ids"]) # token IDs — Chapter 1's subword vocabulary indices
print(encoded["attention_mask"]) # 1 for REAL tokens, 0 for PADDING (Chapter 1, Section 6)The Exact Same Pipeline Covered in DL Notes, Module 6
─────────────────────────────────────────
Token IDs (Chapter 1)
│
▼
Token Embedding lookup (Module 2, Chapter 1, Section 6 —
nn.Embedding — but now over a SUBWORD vocabulary, not
whole words)
│
▼
+ Positional Encoding (DL Notes, Module 6, Chapter 3) —
added element-wise, giving the model a sense of ORDER
│
▼
Transformer blocks (DL Notes, Module 6, Chapter 4) —
self-attention + feed-forward, repeated N times
─────────────────────────────────────────
Nothing about this pipeline is language-specific until the very first step (tokenization) — the embedding lookup, positional encoding, and Transformer blocks are the exact same mechanisms covered generically in the DL Notes, simply applied to a sequence of subword token IDs instead of, say, image patches (DL Notes, Module 6, Chapter 5's Vision Transformer discussion).
3. The Complete Input Pipeline
import torch
import torch.nn as nn
import math
class TransformerTextInput(nn.Module):
"""A minimal illustration connecting Chapter 1's tokenizer to DL Notes' Transformer components"""
def __init__(self, vocab_size, d_model, max_len=512):
super().__init__()
self.token_embedding = nn.Embedding(vocab_size, d_model) # Module 2, Ch.1
self.position_embedding = nn.Embedding(max_len, d_model) # DL Notes, Module 6, Ch.3
# (a LEARNED alternative to
# the sinusoidal encoding)
def forward(self, input_ids):
seq_len = input_ids.size(1)
positions = torch.arange(seq_len).unsqueeze(0)
token_embeds = self.token_embedding(input_ids)
position_embeds = self.position_embedding(positions)
return token_embeds + position_embeds # DL Notes, Module 6, Ch.3, Section 5's addition
vocab_size = tokenizer.vocab_size
model_input_layer = TransformerTextInput(vocab_size, d_model=768)
final_input = model_input_layer(encoded["input_ids"])
print(f"Final Transformer input shape: {final_input.shape}") # [1, seq_len, 768]From this point forward, final_input is fed directly into a stack of encoder or decoder blocks — identical in structure to DL Notes, Module 6, Chapter 4.
4. Recap: What the Transformer Does From Here
A Compressed Recap (Full Detail in DL Notes, Module 6)
─────────────────────────────────────────
Self-attention (Module 6, Ch.2): each token's representation
is updated based on ALL
other tokens — "bank"
gets INFORMED by "river"
or "deposited," directly
solving Module 2, Chapter
4's polysemy problem
Multi-head attention multiple attention
(Module 6, Ch.2): "views" simultaneously
— one head might
track syntax, another
might track semantic
relationships
Feed-forward + residual + processes each
LayerNorm (Module 6, Ch.4; position's
DL Notes Module 3): attention output
further, with
training
stability
techniques
from DL
Notes, Module 3
─────────────────────────────────────────
5. Encoder-Only vs Decoder-Only, Revisited for Language
DL Notes, Module 6, Chapter 4, Section 7 introduced this distinction generically. Applied specifically to language:
The Two Dominant Families
─────────────────────────────────────────
Encoder-only (BERT-style): EVERY token can attend to
EVERY other token, including
ones that come AFTER it —
ideal for UNDERSTANDING a
complete input at once
(classification, extraction)
— Module 4's full focus
Decoder-only (GPT-style): MASKED self-attention (DL
Notes, Module 6, Ch.2, Sec.6)
— a token can ONLY attend
to PRECEDING tokens — ideal
for GENERATING text one
token at a time — Module 5's
full focus
─────────────────────────────────────────
This single architectural choice — whether attention is masked or unmasked — is the fundamental fork in the road that separates every model covered in Modules 4-5.
6. Why This Curriculum Doesn't Re-Derive Attention
A Deliberate Scope Decision
─────────────────────────────────────────
Self-attention, multi-head attention, positional encoding,
and the full Transformer block structure are ARCHITECTURE
concepts, not LANGUAGE concepts — they apply identically
whether processing text, images (Vision Transformers, DL
Notes Module 6, Ch.5), or any other tokenizable sequence.
Re-deriving them here would DUPLICATE the DL Notes' Module 6
entirely, rather than building on it — this curriculum
instead focuses EXCLUSIVELY on what's GENUINELY NEW when
applying that architecture to language specifically:
tokenization (Chapter 1), pre-training objectives (Chapter
3), and everything from Module 4 onward.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Applying a Transformer to language requires only one new step beyond the DL Notes' Module 6 architecture: tokenization (Chapter 1) — everything after the embedding lookup is architecturally identical.
- Token embeddings and positional encodings combine exactly as covered in the DL Notes, just applied to subword token IDs instead of other sequence types.
- The encoder-only vs decoder-only distinction — whether self-attention is masked — is the fundamental fork separating BERT-style (Module 4) from GPT-style (Module 5) models.
- This curriculum deliberately does not re-derive attention or Transformer mechanics, building directly on the DL Notes instead.
Module 3 (Partial) — What's Next
With tokenization and the architecture bridge in place, Chapter 3 covers the final piece needed before diving into BERT and GPT specifically: the pre-training objectives — masked language modeling and next-token prediction — that actually teach these massive models to understand and generate language from raw, unlabeled text.
Concept Check
- What is the one genuinely new step required to apply a Transformer to text, beyond what's already covered in the DL Notes?
- What's the fundamental architectural difference between an encoder-only and a decoder-only Transformer, applied to language?
- Why does this curriculum avoid re-deriving self-attention and the Transformer block structure?
Next Chapter
→ Chapter 3: Pre-training Objectives — MLM vs CLM
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index