Word Embeddings And Representations
Word2Vec: CBOW & Skip-Gram
Word2Vec is trained to predict SURROUNDING words from a
JrCodex·7 min read
Jr Codex NLP & LLM Notes
Level: Intermediate Prerequisites: Chapter 1: From One-Hot to Dense Embeddings Time to complete: ~30 minutes
Table of Contents
- Word2Vec's Core Trick
- Two Architectures: CBOW and Skip-Gram
- CBOW in Detail
- Skip-Gram in Detail
- Negative Sampling — Making Training Tractable
- Training Word2Vec in Practice
- What the Embeddings Actually Learn
- Summary & Next Steps
1. Word2Vec's Core Trick
Word2Vec (2013) doesn't care about its own prediction accuracy — its actual goal is a clever byproduct: training a simple neural network on an easy self-supervised task (predicting context from a word, or vice versa) forces its hidden layer to learn genuinely useful word embeddings, purely to solve that task well.
The Key Insight
─────────────────────────────────────────
Word2Vec is trained to predict SURROUNDING words from a
target word (or vice versa) — a task with NO labels needed
(directly echoing DL Notes, Module 8, Chapter 1's
self-supervised autoencoder framing). The network is
FORCED, in order to do this prediction task well, to learn
a hidden representation that captures MEANING — words used
in similar contexts must end up with similar hidden-layer
representations to make similar predictions.
─────────────────────────────────────────
Once trained, the prediction task itself is thrown away — what's kept is the hidden layer's weight matrix, which becomes the embedding lookup table from Chapter 1, Section 6.
2. Two Architectures: CBOW and Skip-Gram
Two Ways to Frame the Same Self-Supervised Task
─────────────────────────────────────────
CBOW (Continuous Bag-of-Words): given the SURROUNDING
context words, predict
the MIDDLE (target)
word
Skip-Gram: given the MIDDLE
(target) word,
predict each of the
SURROUNDING context
words
─────────────────────────────────────────
A Concrete Example Window
─────────────────────────────────────────
Sentence: "the quick brown fox jumps over"
Target word: "fox" Context window (size 2): "quick",
"brown", "jumps", "over"
CBOW: input = ["quick", "brown", "jumps", "over"]
→ predict → "fox"
Skip-Gram: input = "fox"
→ predict → "quick", "brown", "jumps", "over"
(one at a time, or all at once)
─────────────────────────────────────────
3. CBOW in Detail
import numpy as np
def cbow_forward(context_word_indices, W_in, W_out):
"""
context_word_indices: indices of the SURROUNDING words
W_in: (vocab_size, embed_dim) — the EMBEDDING matrix being learned
W_out: (embed_dim, vocab_size) — used only during TRAINING
"""
context_embeddings = W_in[context_word_indices] # look up each context word's embedding
averaged_context = np.mean(context_embeddings, axis=0) # AVERAGE the context (CBOW's defining step)
output_scores = averaged_context @ W_out # predict the TARGET word
return output_scores # fed through softmax (DL Notes, Module 2, Ch.2) to get a probability distributionWhy CBOW Averages the Context
─────────────────────────────────────────
Averaging MULTIPLE context words into ONE vector before
predicting the target makes CBOW naturally SMOOTH over
several examples at once — this tends to work well for
FREQUENT words, where averaging many similar contexts
produces a stable, well-trained signal.
─────────────────────────────────────────
4. Skip-Gram in Detail
def skip_gram_forward(target_word_index, W_in, W_out):
"""
target_word_index: index of the MIDDLE (target) word
"""
target_embedding = W_in[target_word_index] # look up the TARGET word's embedding
output_scores = target_embedding @ W_out # predict EACH context word (one training example per context word)
return output_scoresWhy Skip-Gram Tends to Work Better for RARE Words
─────────────────────────────────────────
Unlike CBOW's averaging (Section 3), Skip-Gram creates a
SEPARATE training example for EACH (target, context) word
pair — a rare word gets to directly influence training
MULTIPLE times (once per context word it appears next to),
rather than being diluted through averaging. This is why
Skip-Gram is generally the MORE POPULAR of the two Word2Vec
variants in practice, despite being computationally more
expensive per epoch.
─────────────────────────────────────────
5. Negative Sampling — Making Training Tractable
A naive implementation of either architecture requires a full softmax (DL Notes, Module 2, Chapter 2) over the ENTIRE vocabulary — computationally prohibitive for vocabularies of hundreds of thousands of words, at every single training step.
The Negative Sampling Trick
─────────────────────────────────────────
Instead of predicting the FULL probability distribution over
every possible word, reframe the task as SIMPLE BINARY
classification: "is THIS specific (target, context) pair a
REAL pair seen in the text, or a RANDOMLY sampled FAKE pair?"
For each REAL (target, context) pair, sample a SMALL number
(e.g. 5-20) of RANDOM "negative" words as fake pairs — the
model just needs to distinguish real from fake, a MUCH
cheaper computation than a full softmax over the entire
vocabulary.
─────────────────────────────────────────
import numpy as np
def negative_sampling_loss(target_embedding, context_embedding, negative_embeddings):
"""A simplified sketch of the negative sampling objective"""
def sigmoid(x):
return 1 / (1 + np.exp(-x))
positive_score = sigmoid(np.dot(target_embedding, context_embedding)) # want this CLOSE TO 1
positive_loss = -np.log(positive_score + 1e-10)
negative_loss = 0
for neg_embedding in negative_embeddings: # want THESE close to 0
negative_score = sigmoid(-np.dot(target_embedding, neg_embedding))
negative_loss += -np.log(negative_score + 1e-10)
return positive_loss + negative_lossThis directly echoes DL Notes, Module 2, Chapter 3's binary cross-entropy — negative sampling reframes an expensive multi-class problem as a cheap series of binary classification problems.
6. Training Word2Vec in Practice
from gensim.models import Word2Vec
sentences = [
["the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"],
["the", "dog", "barks", "at", "the", "cat"],
["the", "cat", "chases", "the", "mouse"],
# a REAL application would use THOUSANDS to MILLIONS of sentences
]
model = Word2Vec(
sentences,
vector_size=100, # embedding dimension (Chapter 1, Section 5)
window=5, # context window size (Section 2's example used window=2)
min_count=1, # ignore words appearing fewer than this many times
sg=1, # sg=1 for Skip-Gram (Section 4), sg=0 for CBOW (Section 3)
negative=5, # negative sampling (Section 5) — 5 negative samples per positive
)
print(model.wv["dog"]) # the learned embedding vector for "dog"
print(model.wv.most_similar("dog", topn=3)) # words with the MOST SIMILAR embeddings7. What the Embeddings Actually Learn
Emergent Properties (Not Explicitly Programmed)
─────────────────────────────────────────
Semantic similarity: "cat" and "dog" end up close in
embedding space, purely from
appearing in similar contexts
(Chapter 1, Section 4's
distributional hypothesis)
Analogical structure: vector arithmetic like
king - man + woman ≈ queen
(Chapter 1, Section 5) emerges
from the training process,
without ever being an
explicit training objective
Clustering by category: related concepts (colors,
countries, professions)
naturally cluster together
in the embedding space
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Word2Vec trains a simple network on a self-supervised prediction task purely to force its hidden layer to learn useful embeddings — the prediction task itself is discarded after training.
- CBOW predicts a target word from its averaged context; Skip-Gram predicts each context word from the target, generally performing better on rare words since each context word provides a separate training signal.
- Negative sampling reframes the expensive full-vocabulary softmax as cheap binary classification between real and randomly-sampled fake word pairs.
- Semantic similarity and analogical structure (like king - man + woman ≈ queen) emerge automatically from training, without being explicit objectives.
Concept Check
- Why is the actual Word2Vec prediction task discarded after training, while the hidden layer's weights are kept?
- Why does Skip-Gram tend to perform better than CBOW on rare words?
- What problem does negative sampling solve, and how does it reframe the training objective?
Next Chapter
→ Chapter 3: GloVe & Evaluating Embeddings
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index