Word Embeddings And Representations
GloVe & Evaluating Embeddings
def build_cooccurrence_matrix(sentences, window_size=2):
Jr Codex NLP & LLM Notes
Level: Intermediate Prerequisites: Chapter 2: Word2Vec — CBOW & Skip-Gram Time to complete: ~20 minutes
Table of Contents
- A Different Approach: Global Statistics
- Building a Co-Occurrence Matrix
- GloVe's Training Objective
- GloVe vs Word2Vec
- Using Pretrained Embeddings
- Evaluating Embedding Quality
- Summary & Next Steps
1. A Different Approach: Global Statistics
GloVe (Global Vectors), also from 2014, pursues the same goal as Word2Vec — dense, meaningful word embeddings — through a fundamentally different mechanism: instead of predicting context word-by-word from a sliding window (Chapter 2), GloVe directly factorizes global word co-occurrence statistics computed across the entire corpus at once.
2. Building a Co-Occurrence Matrix
import numpy as np
from collections import defaultdict
def build_cooccurrence_matrix(sentences, window_size=2):
vocab = sorted(set(word for sentence in sentences for word in sentence))
word_to_idx = {word: i for i, word in enumerate(vocab)}
cooccurrence = defaultdict(float)
for sentence in sentences:
for i, word in enumerate(sentence):
word_idx = word_to_idx[word]
start = max(0, i - window_size)
end = min(len(sentence), i + window_size + 1)
for j in range(start, end):
if i != j:
context_idx = word_to_idx[sentence[j]]
distance = abs(i - j)
cooccurrence[(word_idx, context_idx)] += 1.0 / distance # CLOSER words weighted MORE
return cooccurrence, word_to_idx
sentences = [
["the", "cat", "sat", "on", "the", "mat"],
["the", "dog", "sat", "on", "the", "log"],
]
cooc_matrix, vocab_index = build_cooccurrence_matrix(sentences)
print(f"Vocabulary size: {len(vocab_index)}")
print(f"Sample co-occurrence: {list(cooc_matrix.items())[:3]}")Unlike Word2Vec's sliding-window training process (Chapter 2), this co-occurrence matrix is computed ONCE, upfront, capturing GLOBAL statistics across the ENTIRE corpus — how often every word pair appears near each other, anywhere in the training text, rather than learning incrementally from a stream of local context windows.
3. GloVe's Training Objective
GloVe then learns embeddings such that the dot product between two word vectors approximates the logarithm of their co-occurrence count.
The Core Relationship GloVe Optimizes For
─────────────────────────────────────────
vector(word_i) · vector(word_j) ≈ log( co-occurrence(word_i, word_j) )
Words that co-occur FREQUENTLY should have a LARGE dot
product (vectors pointing in similar directions); words that
RARELY co-occur should have a SMALL dot product.
─────────────────────────────────────────
import numpy as np
def glove_loss(word_vectors, context_vectors, biases_w, biases_c, cooccurrence_matrix, x_max=100, alpha=0.75):
total_loss = 0
for (i, j), count in cooccurrence_matrix.items():
# A WEIGHTING function — down-weights extremely frequent pairs (e.g. "the", "a")
# so they don't dominate training, echoing Module 1, Chapter 3's TF-IDF intuition
weight = (count / x_max) ** alpha if count < x_max else 1.0
prediction = np.dot(word_vectors[i], context_vectors[j]) + biases_w[i] + biases_c[j]
target = np.log(count)
total_loss += weight * (prediction - target) ** 2 # a WEIGHTED squared error (DL Notes, Module 2, Ch.3's MSE)
return total_loss4. GloVe vs Word2Vec
Two Different Roads to a Similar Destination
─────────────────────────────────────────
Word2Vec (Ch.2): LOCAL context windows, processed
incrementally, one window at a time —
a PREDICTIVE model (predict context
from word, or vice versa)
GloVe: GLOBAL co-occurrence STATISTICS,
computed upfront, then factorized
directly — a COUNT-BASED model
(fit vectors to MATCH observed
co-occurrence counts)
In PRACTICE, both approaches tend to produce embeddings of
broadly SIMILAR quality — the choice historically came down
to available implementations, corpus size, and specific task
performance, rather than one being categorically superior.
─────────────────────────────────────────
5. Using Pretrained Embeddings
Training either Word2Vec or GloVe from scratch requires a very large corpus — in practice, most projects use pretrained embeddings, trained once on massive corpora (Wikipedia, Common Crawl) and freely distributed.
import gensim.downloader as api
# Downloads (and caches) pretrained GloVe embeddings, trained on 6 billion words
glove_vectors = api.load("glove-wiki-gigaword-100") # 100-dimensional vectors
print(glove_vectors["computer"][:10]) # the first 10 dimensions of "computer"'s embedding
print(glove_vectors.most_similar("computer", topn=5))
print(glove_vectors.most_similar(positive=["king", "woman"], negative=["man"], topn=3))
# Reproduces the "king - man + woman ≈ queen" arithmetic from Chapter 1, Section 5This directly parallels DL Notes, Module 7's transfer learning concept — reusing embeddings trained by someone else on a massive corpus, rather than training your own from a smaller, task-specific dataset.
6. Evaluating Embedding Quality
Two Standard Evaluation Approaches
─────────────────────────────────────────
Intrinsic evaluation: test embeddings DIRECTLY, without a
downstream task — e.g. word
similarity benchmarks (do
human-rated "similar" word pairs
ALSO have high cosine similarity
in the embedding space?), or
analogy tasks (Section 5's
king/queen example, tested
systematically across many
analogies)
Extrinsic evaluation: test embeddings INDIRECTLY, by
using them as INPUT FEATURES
for a real downstream task
(e.g. Module 1, Chapter 3's
text classification) and
measuring the ML Notes'
standard task performance
metrics
─────────────────────────────────────────
from scipy.stats import spearmanr
# A SIMPLIFIED intrinsic evaluation — comparing embedding similarity
# to KNOWN human similarity judgments
human_judged_pairs = [("car", "automobile", 0.98), ("cat", "dog", 0.7), ("cat", "computer", 0.1)]
model_similarities = [glove_vectors.similarity(w1, w2) for w1, w2, _ in human_judged_pairs]
human_similarities = [score for _, _, score in human_judged_pairs]
correlation, _ = spearmanr(model_similarities, human_similarities)
print(f"Correlation with human judgment: {correlation:.3f}") # HIGHER = better-aligned embeddings7. Summary & Next Steps
Key Takeaways
- GloVe factorizes global word co-occurrence statistics computed across the entire corpus at once, in contrast to Word2Vec's incremental, sliding-window training.
- Both approaches produce embeddings of broadly similar practical quality, despite their different underlying mechanisms.
- Pretrained embeddings (trained on massive corpora like Wikipedia) are the practical default for most projects, directly echoing the DL Notes' transfer learning principle.
- Embedding quality is evaluated intrinsically (similarity/analogy benchmarks against human judgment) or extrinsically (performance on a real downstream task).
Module 2 (Partial) — What's Next
You now understand two foundational, "static" embedding techniques — every word gets exactly ONE fixed vector, regardless of context. Chapter 4 confronts this limitation directly: the same word can mean different things in different sentences, something neither Word2Vec nor GloVe can represent.
Concept Check
- What is the fundamental difference in HOW GloVe and Word2Vec each learn their embeddings?
- Why does GloVe's loss function down-weight extremely frequent co-occurring pairs?
- What's the difference between intrinsic and extrinsic embedding evaluation, with an example of each?
Next Chapter
→ Chapter 4: Contextual Embeddings — Why Static Embeddings Fall Short
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index