Encoder Models Bert And Friends
BERT: Architecture & Masked Language Modeling
BERT is, architecturally, simply a stack of Transformer encoder blocks — exactly the encoder covered in DL Notes, Module 6, Chapter 4, Section 2, with no decode
Jr Codex NLP & LLM Notes
Level: Intermediate–Advanced Prerequisites: Module 3: Tokenization & Pre-training Foundations Time to complete: ~25 minutes
Table of Contents
- BERT's Place in History
- BERT's Architecture
- The [CLS] Token
- Input Representation
- Loading and Inspecting BERT
- BERT's Bidirectional Advantage, Concretely
- Summary & Next Steps
1. BERT's Place in History
BERT (Bidirectional Encoder Representations from Transformers), released by Google in 2018, was the first model to demonstrate that pretraining a deep, bidirectional Transformer encoder on massive unlabeled text — then fine-tuning it for specific tasks — dramatically outperformed the previous generation of task-specific architectures (DL Notes, Module 5's RNNs, and shallow contextual embeddings like ELMo, Module 2, Chapter 4).
2. BERT's Architecture
BERT is, architecturally, simply a stack of Transformer encoder blocks — exactly the encoder covered in DL Notes, Module 6, Chapter 4, Section 2, with no decoder at all.
BERT's Two Standard Sizes
─────────────────────────────────────────
BERT-base: 12 encoder blocks, 768-dimensional hidden
size, 12 attention heads — ~110 million
parameters
BERT-large: 24 encoder blocks, 1024-dimensional
hidden size, 16 attention heads —
~340 million parameters
─────────────────────────────────────────
from transformers import BertModel, BertConfig
config = BertConfig(
vocab_size=30522, # WordPiece vocabulary (Module 3, Ch.1, Section 4)
hidden_size=768, # d_model, in DL Notes' Module 6 terminology
num_hidden_layers=12, # the number of stacked encoder blocks
num_attention_heads=12, # DL Notes, Module 6, Chapter 2's multi-head attention
intermediate_size=3072, # the feed-forward sub-layer's hidden dimension
# (DL Notes, Module 6, Ch.4, Sec.5)
)
model = BertModel(config) # an UNTRAINED BERT — Section 5 loads a PRETRAINED one instead
print(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}")Every architectural piece here — encoder blocks, multi-head attention, feed-forward sub-layers — was already covered in full in the DL Notes' Module 6. BERT's genuine novelty is entirely in how it's trained (Module 3, Chapter 3's MLM) and what data it's trained on, not in any new architectural component.
3. The [CLS] Token
BERT prepends a special [CLS] ("classification") token to the start of every input sequence.
Why [CLS] Exists
─────────────────────────────────────────
After passing through all 12 (or 24) encoder layers, [CLS]'s
final hidden state — having ATTENDED to every other token in
the sequence via self-attention (DL Notes, Module 6, Ch.2) —
serves as an AGGREGATE representation of the ENTIRE input,
suitable for feeding into a classification head (Chapter 3)
WITHOUT needing to separately pool or combine every
individual token's representation.
─────────────────────────────────────────
from transformers import AutoTokenizer, AutoModel
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
text = "BERT understands context bidirectionally."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
cls_representation = outputs.last_hidden_state[:, 0, :] # position 0 is ALWAYS [CLS]
print(f"[CLS] representation shape: {cls_representation.shape}") # [1, 768]
pooled_output = outputs.pooler_output # a further-processed version of [CLS], BERT provides directly
print(f"Pooled output shape: {pooled_output.shape}") # [1, 768]4. Input Representation
BERT combines three separate embeddings for each token — directly extending Module 3, Chapter 2's token + positional embedding sum.
BERT's Three Embedding Types, Summed
─────────────────────────────────────────
Token embeddings: the WordPiece vocabulary lookup
(Module 3, Ch.1, Section 4)
Position embeddings: LEARNED positional information
(Module 3, Ch.2, Section 3's
learned alternative to sinusoidal
encoding)
Segment embeddings: identifies whether a token belongs
to "sentence A" or "sentence B" —
needed for tasks INVOLVING TWO
sentences at once (e.g. question
+ passage), a BERT-specific
addition beyond the generic
Transformer input from Module 3
─────────────────────────────────────────
[CLS] this [SEP] is sentence A [SEP] this is sentence B [SEP]
↑ ↑ ↑
Segment A Segment A Segment B
5. Loading and Inspecting BERT
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased") # loads PRETRAINED weights directly (DL Notes, Module 7)
# Inspect the encoder stack directly
print(f"Number of encoder layers: {model.config.num_hidden_layers}") # 12
print(f"Hidden size: {model.config.hidden_size}") # 768
print(f"Number of attention heads: {model.config.num_attention_heads}") # 12
# Access a SPECIFIC layer's self-attention weights, for inspection
first_layer_attention = model.encoder.layer[0].attention.self
print(first_layer_attention)6. BERT's Bidirectional Advantage, Concretely
from transformers import AutoTokenizer, AutoModel
import torch
def get_contextual_representation(sentence, target_word):
inputs = tokenizer(sentence, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
idx = tokens.index(target_word)
return outputs.last_hidden_state[0, idx]
# Directly reproducing Module 2, Chapter 4's polysemy example
bank_financial = get_contextual_representation("I deposited money in the bank", "bank")
bank_river = get_contextual_representation("We sat by the river bank", "bank")
similarity = torch.cosine_similarity(bank_financial.unsqueeze(0), bank_river.unsqueeze(0))
print(f"Similarity: {similarity.item():.3f}") # noticeably LOWER than 1.0 — confirming BERT
# distinguishes the two MEANINGS, unlike Module
# 2's static Word2Vec/GloVe embeddingsThis directly confirms Module 2, Chapter 4's prediction: because BERT's self-attention lets "bank" attend to ITS SPECIFIC surrounding words ("river" vs "deposited") in BOTH directions, its representation of "bank" genuinely differs based on context — something no static embedding technique could achieve.
7. Summary & Next Steps
Key Takeaways
- BERT is architecturally just a stack of Transformer encoder blocks, identical to the DL Notes' Module 6 encoder — its novelty is entirely in training methodology, not new architectural components.
- The [CLS] token's final hidden state, having attended to the entire input via self-attention, serves as an aggregate representation suitable for classification tasks.
- BERT combines three embedding types (token, position, segment), with segment embeddings enabling tasks involving two sentences at once.
- BERT's bidirectional self-attention directly produces genuinely contextual representations that distinguish word meanings by context, confirming the motivation established in Module 2, Chapter 4.
Concept Check
- What genuinely new architectural component does BERT introduce beyond the DL Notes' Module 6 Transformer encoder?
- What is the [CLS] token's purpose, and why is it well-suited to classification tasks specifically?
- Why do BERT's segment embeddings exist, and what kind of task requires them?
Next Chapter
→ Chapter 2: BERT's Pretraining — MLM + NSP, and What Came After
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index