Encoder Models Bert And Friends
BERT's Pretraining: MLM + NSP, and What Came After
Module 3, Chapter 3 introduced Masked Language Modeling generically. BERT's original 2018 paper actually combined two pretraining tasks simultaneously: MLM (pre
JrCodex·6 min read
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 1: BERT — Architecture & Masked Language Modeling Time to complete: ~20 minutes
Table of Contents
- BERT's Two Original Pretraining Tasks
- Masked Language Modeling, in Full Detail
- Next Sentence Prediction (NSP)
- The Pretraining Data and Objective, Combined
- NSP's Discovered Weakness
- What Came After BERT: a Brief Orientation
- Summary & Next Steps
1. BERT's Two Original Pretraining Tasks
Module 3, Chapter 3 introduced Masked Language Modeling generically. BERT's original 2018 paper actually combined two pretraining tasks simultaneously: MLM (predicting masked tokens) and Next Sentence Prediction (NSP) (predicting whether two sentences genuinely follow each other).
2. Masked Language Modeling, in Full Detail
BERT's Specific Masking Strategy (Not Simply "Always Mask")
─────────────────────────────────────────
Of the 15% of tokens selected for masking (Module 3, Ch.3,
Section 3):
80% of the time: replaced with the actual [MASK] token
10% of the time: replaced with a RANDOM other token
10% of the time: left UNCHANGED (but still
included in the loss)
─────────────────────────────────────────
Why This Specific 80/10/10 Split, Not Simply 100% [MASK]
─────────────────────────────────────────
If EVERY masked position were replaced with the literal
[MASK] token, the model would only ever need to produce
useful representations for [MASK] tokens specifically — but
at FINE-TUNING time (Chapter 3) and INFERENCE time, [MASK]
NEVER appears in real input. This mismatch between
pretraining and downstream use is called PRETRAIN-FINETUNE
DISCREPANCY. Occasionally substituting a RANDOM token or
leaving the ORIGINAL token unchanged forces the model to
build genuinely robust representations for EVERY token
position, not just ones marked with a special symbol.
─────────────────────────────────────────
import torch
import random
def bert_masking_strategy(tokenizer, input_ids, mask_probability=0.15):
labels = input_ids.clone()
special_tokens_mask = torch.tensor([
tokenizer.get_special_tokens_mask(ids, already_has_special_tokens=True)
for ids in input_ids.tolist()
]).bool()
probability_matrix = torch.full(input_ids.shape, mask_probability)
probability_matrix.masked_fill_(special_tokens_mask, value=0.0)
masked_indices = torch.bernoulli(probability_matrix).bool()
labels[~masked_indices] = -100 # only masked positions contribute to loss
# 80% -> [MASK]
indices_replaced = torch.bernoulli(torch.full(input_ids.shape, 0.8)).bool() & masked_indices
input_ids[indices_replaced] = tokenizer.mask_token_id
# 10% -> random token
indices_random = torch.bernoulli(torch.full(input_ids.shape, 0.5)).bool() & masked_indices & ~indices_replaced
random_tokens = torch.randint(len(tokenizer), input_ids.shape, dtype=torch.long)
input_ids[indices_random] = random_tokens[indices_random]
# remaining 10% -> UNCHANGED (no further action needed)
return input_ids, labels3. Next Sentence Prediction (NSP)
NSP trains BERT to predict whether sentence B genuinely follows sentence A in the original text, or is a randomly sampled, unrelated sentence — using the [CLS] token's representation (Chapter 1, Section 3) directly for this binary classification.
# Positive example (50% of training pairs) — B ACTUALLY follows A in the source text
sentence_a = "The chef prepared a delicious meal."
sentence_b = "Everyone at the table complimented the food."
label = 1 # "IsNext"
# Negative example (50% of training pairs) — B is a RANDOM sentence from elsewhere in the corpus
sentence_a_neg = "The chef prepared a delicious meal."
sentence_b_neg = "The stock market fell sharply today."
label_neg = 0 # "NotNext"
inputs = tokenizer(sentence_a, sentence_b, return_tensors="pt") # Chapter 1, Section 4's segment embeddings
# mark WHICH sentence each token belongs toThe Motivation Behind NSP
─────────────────────────────────────────
MLM alone (Section 2) only teaches WORD-level and
SENTENCE-INTERNAL understanding. BERT's authors hypothesized
that many important downstream tasks — question answering,
natural language inference — require understanding
RELATIONSHIPS BETWEEN sentences, which MLM alone doesn't
directly train for. NSP was added specifically to address
this gap.
─────────────────────────────────────────
4. The Pretraining Data and Objective, Combined
BERT's Original Training Setup
─────────────────────────────────────────
Data: BooksCorpus (800 million words) + English
Wikipedia (2,500 million words)
Combined loss: MLM loss (Section 2) + NSP loss (Section 3),
summed and optimized TOGETHER in every
training step
Training compute: 4 days on 16 TPU chips (BERT-large) —
substantial by 2018 standards, though
modest compared to modern frontier
models (Module 3, Ch.3, Section 6)
─────────────────────────────────────────
5. NSP's Discovered Weakness
Subsequent research (notably the RoBERTa paper, Section 6) found that NSP's contribution to downstream performance was much smaller than originally believed — and in some analyses, removing it entirely and training longer on MLM alone actually improved results.
Why NSP Turned Out to Be a Weaker Signal Than Expected
─────────────────────────────────────────
The NEGATIVE examples (Section 3) — a random, UNRELATED
sentence from elsewhere in the corpus — turned out to be
TOO EASY to distinguish from genuine continuations, often
based on SUPERFICIAL topic differences alone, rather than
requiring the deep, coherent reasoning about sentence
relationships the task was intended to teach.
─────────────────────────────────────────
This is a valuable, concrete lesson in ML/DL research methodology (echoing the ML Notes' rigorous evaluation discipline): a plausible-sounding training objective doesn't automatically improve results, and must be validated empirically, not just assumed.
6. What Came After BERT: a Brief Orientation
The BERT Family's Evolution (Full Detail in Chapter 4)
─────────────────────────────────────────
RoBERTa (2019): REMOVED NSP entirely (Section 5),
trained on MORE data for LONGER, with
larger batches — outperformed the
original BERT substantially, from
TRAINING changes alone, no architecture
changes
ALBERT (2019): reduced parameter count via
parameter-SHARING across layers —
similar performance, far fewer
parameters
DistilBERT (2019): a SMALLER, faster model trained to
mimic BERT's behavior (DL Notes,
Module 7's transfer learning
concept, applied as
"distillation")
ELECTRA (2020): replaced MLM with a MORE
sample-efficient objective
(detecting REPLACED tokens,
rather than predicting
MASKED ones)
─────────────────────────────────────────
Chapter 4 covers this family in depth.
7. Summary & Next Steps
Key Takeaways
- BERT's original pretraining combined two tasks: MLM (predicting masked tokens) and NSP (predicting whether two sentences genuinely follow each other).
- BERT's 80/10/10 masking strategy (mask, random token, or unchanged) specifically addresses a mismatch between pretraining (where [MASK] appears) and downstream use (where it doesn't).
- Subsequent research found NSP contributed far less than originally believed, and RoBERTa's removal of it (with more training data and time) improved results — a lesson in empirically validating training objectives.
- Later BERT-family models (RoBERTa, ALBERT, DistilBERT, ELECTRA) each address a specific limitation of the original: training methodology, parameter efficiency, or pretraining objective design.
Concept Check
- Why does BERT's masking strategy sometimes leave a token unchanged or replace it with a random token, instead of always using [MASK]?
- What was NSP intended to teach the model that MLM alone could not?
- What did the RoBERTa paper's finding about NSP demonstrate about validating training objectives empirically?
Next Chapter
→ Chapter 3: Fine-Tuning BERT for Downstream Tasks
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index