Encoder Models Bert And Friends
The BERT Family: RoBERTa, ALBERT, DistilBERT, ELECTRA
Chapter 2, Section 6 previewed four models that each address a different limitation of the original BERT. This chapter covers each in enough depth to make an in
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 3: Fine-Tuning BERT for Downstream Tasks Time to complete: ~20 minutes
Table of Contents
- Four Directions of Improvement
- RoBERTa: Better Training, Same Architecture
- ALBERT: Parameter Efficiency
- DistilBERT: Knowledge Distillation
- ELECTRA: a Better Pretraining Objective
- Choosing Among the BERT Family
- Using Any of These in Practice
- Summary & Next Steps
1. Four Directions of Improvement
Chapter 2, Section 6 previewed four models that each address a different limitation of the original BERT. This chapter covers each in enough depth to make an informed choice among them.
2. RoBERTa: Better Training, Same Architecture
RoBERTa (Robustly Optimized BERT Approach), from Facebook AI in 2019, made zero architectural changes to BERT — every improvement came purely from how it was trained.
RoBERTa's Training Changes
─────────────────────────────────────────
Removed NSP entirely: (Chapter 2, Section 5) — training
on LONGER, more coherent
sequences of text instead
Trained on 10x MORE data: 160GB of text vs BERT's ~16GB
Trained with LARGER batches and for MORE steps overall
and LONGER:
Dynamic masking: generates a NEW random
mask pattern EVERY time
a sequence is seen
(across multiple training
epochs), rather than
BERT's original approach
of masking ONCE during
preprocessing and reusing
that SAME mask repeatedly
─────────────────────────────────────────
The Broader Lesson
─────────────────────────────────────────
RoBERTa is direct empirical evidence that TRAINING
methodology (data quantity, training duration, masking
strategy) can matter AS MUCH as architectural innovation —
echoing DL Notes, Module 1, Chapter 3's "data + compute +
algorithms" framing of what actually drives progress.
─────────────────────────────────────────
3. ALBERT: Parameter Efficiency
ALBERT (A Lite BERT), also 2019, reduces BERT's parameter count dramatically through two specific architectural changes, while retaining comparable performance.
ALBERT's Two Key Techniques
─────────────────────────────────────────
Factorized embedding the TOKEN embedding
parameterization: dimension is decoupled
from the HIDDEN layer
dimension — using a
SMALLER embedding
dimension, then
projecting UP to the
hidden size, rather
than tying them together
(as BERT does)
Cross-layer parameter ALL encoder blocks
sharing: SHARE the exact same
weights, rather than
each of the 12 (or
24) layers having
its OWN independent
parameters
─────────────────────────────────────────
Why Parameter Sharing Works Reasonably Well
─────────────────────────────────────────
Reusing the SAME transformation repeatedly, layer after
layer, is conceptually similar to an RNN's weight-sharing
ACROSS time steps (DL Notes, Module 5, Chapter 2, Section
2) — rather than ACROSS TIME, ALBERT shares weights ACROSS
DEPTH. This dramatically reduces total parameters (ALBERT-
large has ~18 million parameters vs BERT-large's ~340
million) at a modest performance cost.
─────────────────────────────────────────
4. DistilBERT: Knowledge Distillation
DistilBERT (2019) uses knowledge distillation: training a smaller "student" model to mimic a larger, already-trained "teacher" model's (BERT's) behavior, rather than pretraining from scratch on raw text.
import torch
import torch.nn.functional as F
def distillation_loss(student_logits, teacher_logits, true_labels, temperature=2.0, alpha=0.5):
# The student learns from TWO signals simultaneously:
soft_targets = F.softmax(teacher_logits / temperature, dim=-1) # the TEACHER's full
# probability distribution
# (softened by temperature) —
# richer signal than a single label
soft_student = F.log_softmax(student_logits / temperature, dim=-1)
distillation_term = F.kl_div(soft_student, soft_targets, reduction="batchmean") * (temperature ** 2)
hard_target_term = F.cross_entropy(student_logits, true_labels) # the ACTUAL correct label
# (DL Notes, Module 2, Ch.3)
return alpha * distillation_term + (1 - alpha) * hard_target_termWhy the Teacher's FULL Distribution Is More Informative Than a Single Label
─────────────────────────────────────────
A true label says ONLY "the correct class is X." The
TEACHER's full softmax output (e.g. 70% cat, 25% dog, 5%
everything else) reveals RICHER information — that "dog" is
a much more plausible SECOND guess than "car" — implicitly
teaching the student about CLASS RELATIONSHIPS, not just
the single correct answer.
─────────────────────────────────────────
DistilBERT retains 97% of BERT's performance on standard benchmarks, while being 40% smaller and 60% faster — directly relevant to DL Notes, Module 9's production latency concerns.
5. ELECTRA: a Better Pretraining Objective
ELECTRA (2020) replaces MLM entirely with a more sample-efficient objective inspired by GANs (DL Notes, Module 8, Chapter 3): a small "generator" replaces some tokens with plausible alternatives, and a "discriminator" (the model actually being pretrained) must identify which tokens were replaced.
ELECTRA vs MLM — Why This Is More Sample-Efficient
─────────────────────────────────────────
MLM (Chapter 2, Section 2): the loss is computed on ONLY
the 15% of masked positions
— the other 85% of tokens
provide NO training signal
at that step
ELECTRA: the discriminator makes a
prediction for EVERY
SINGLE token position
(replaced or not) — 100%
of tokens contribute to
the training signal, at
EVERY step
─────────────────────────────────────────
The GAN-Like Structure, Directly Echoing DL Notes Module 8
─────────────────────────────────────────
Generator: a SMALL MLM-style model, generating
plausible REPLACEMENT tokens (analogous to
DL Notes, Module 8, Chapter 3's Generator
network)
Discriminator: the MAIN model being pretrained — learns
to distinguish ORIGINAL tokens from
GENERATOR-replaced ones (analogous to
the Discriminator network) — this
discriminator is what gets KEPT and
fine-tuned for downstream tasks
(Chapter 3)
─────────────────────────────────────────
6. Choosing Among the BERT Family
A Practical Decision Guide
─────────────────────────────────────────
Best RAW performance, RoBERTa or ELECTRA — both
compute NOT a concern: meaningfully outperform
original BERT
Tight MEMORY budget, ALBERT — dramatically fewer
performance flexible: parameters via layer sharing
Tight LATENCY/inference DistilBERT — 60% faster,
speed budget: 97% of BERT's performance
Best sample efficiency ELECTRA — every token
during PRETRAINING position contributes
(if training your OWN training signal
model from scratch):
─────────────────────────────────────────
7. Using Any of These in Practice
Thanks to HuggingFace's Auto classes (DL Notes, Module 7, Chapter 3, Section 3), switching between any of these models requires changing only the model name string — every fine-tuning recipe from Chapter 3 applies unchanged.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
for model_name in ["bert-base-uncased", "roberta-base", "albert-base-v2", "distilbert-base-uncased"]:
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
print(f"{model_name}: {sum(p.numel() for p in model.parameters()):,} parameters")8. Summary & Next Steps
Key Takeaways
- RoBERTa improves on BERT purely through training methodology (more data, longer training, dynamic masking, no NSP), without any architectural changes.
- ALBERT reduces parameters via factorized embeddings and cross-layer parameter sharing, similar in spirit to an RNN's weight sharing across time steps.
- DistilBERT uses knowledge distillation — training a smaller student to mimic a larger teacher's full output distribution — retaining most performance at a fraction of the size.
- ELECTRA replaces MLM with a GAN-inspired replaced-token-detection objective, making every token position contribute training signal rather than just the 15% that MLM masks.
Module 4 Complete — What's Next
You've now covered the complete encoder-only family: BERT's architecture, its pretraining objectives, practical fine-tuning across four major task types, and the variants that improved on it in different directions. Module 5 shifts to the other major Transformer family: decoder-only models, built for generation rather than understanding — GPT and the modern LLM landscape.
Concept Check
- What specific insight does RoBERTa provide about the relative importance of architecture versus training methodology?
- Why is a teacher model's full probability distribution more informative for a student model than just the correct label?
- Why does ELECTRA's replaced-token-detection objective use training signal more efficiently than BERT's original MLM?
Next Module
→ Module 5: Decoder Models — GPT & Modern LLMs
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index