Word Embeddings And Representations
From One-Hot to Dense Embeddings
Module 1, Chapter 3 represented documents as word counts — but never addressed a deeper question: how should an individual word be represented in a way a model
Jr Codex NLP & LLM Notes
Level: Beginner–Intermediate Prerequisites: Module 1: Foundations of NLP Time to complete: ~20 minutes
Table of Contents
- The Problem With Bag-of-Words Persists
- One-Hot Encoding
- Why One-Hot Vectors Fail to Capture Meaning
- The Distributional Hypothesis
- What a Dense Embedding Actually Is
- Embeddings in PyTorch
- Summary & Next Steps
1. The Problem With Bag-of-Words Persists
Module 1, Chapter 3 represented documents as word counts — but never addressed a deeper question: how should an individual word be represented in a way a model can use mathematically? This module answers that question.
2. One-Hot Encoding
The simplest possible word representation: a vector as long as the entire vocabulary, with a single 1 at the position corresponding to that word, and 0 everywhere else — directly the ML Notes' categorical encoding technique, applied to words.
vocabulary = ["cat", "dog", "king", "queen", "apple"]
def one_hot_encode(word, vocab):
vector = [0] * len(vocab)
vector[vocab.index(word)] = 1
return vector
print(one_hot_encode("cat", vocabulary)) # [1, 0, 0, 0, 0]
print(one_hot_encode("dog", vocabulary)) # [0, 1, 0, 0, 0]
print(one_hot_encode("king", vocabulary)) # [0, 0, 1, 0, 0]3. Why One-Hot Vectors Fail to Capture Meaning
Two Fundamental Problems
─────────────────────────────────────────
1. DIMENSIONALITY EXPLOSION: a real vocabulary has
tens of thousands to
hundreds of thousands of
words — each one-hot
vector is THAT long, with
exactly ONE non-zero
entry — extremely
wasteful
2. NO NOTION OF SIMILARITY: "cat" and "dog" (similar,
both animals) are
EQUALLY as different,
mathematically, as "cat"
and "apple" (unrelated)
— the DOT PRODUCT (or
cosine similarity)
between ANY two distinct
one-hot vectors is
ALWAYS exactly zero
─────────────────────────────────────────
import numpy as np
cat = np.array(one_hot_encode("cat", vocabulary))
dog = np.array(one_hot_encode("dog", vocabulary))
apple = np.array(one_hot_encode("apple", vocabulary))
print(f"cat . dog similarity: {np.dot(cat, dog)}") # 0 — no similarity captured
print(f"cat . apple similarity: {np.dot(cat, apple)}") # 0 — IDENTICAL score, despite being unrelatedThis is the exact problem embeddings exist to solve — a representation where semantically similar words end up mathematically close to each other, and unrelated words end up far apart.
4. The Distributional Hypothesis
The key theoretical insight underlying every embedding technique in this module: "You shall know a word by the company it keeps" (J.R. Firth, 1957) — words that appear in similar contexts tend to have similar meanings.
The Distributional Hypothesis, Illustrated
─────────────────────────────────────────
"The CAT sat on the mat and purred."
"The DOG sat on the mat and barked."
"The CAT chased the mouse."
"The DOG chased the mailman."
"cat" and "dog" appear in VERY similar surrounding contexts
(sitting on mats, chasing things) — the distributional
hypothesis predicts they should have SIMILAR representations,
purely from observing these patterns across a large amount
of text, with NO explicit definition of "animal" ever
provided.
─────────────────────────────────────────
Every embedding technique in this module — Word2Vec (Chapter 2), GloVe (Chapter 3) — is, at its core, a specific mathematical way of exploiting this hypothesis.
5. What a Dense Embedding Actually Is
A dense embedding represents each word as a vector of real numbers (typically 50-300 dimensions for classical embeddings, hundreds to thousands for modern Transformer models), where every dimension can be non-zero, and geometric closeness reflects semantic similarity.
One-Hot vs Dense Embeddings
─────────────────────────────────────────
One-hot (Section 2): length = VOCABULARY SIZE (e.g.
50,000) — SPARSE (one 1, rest 0s)
— no similarity structure
Dense embedding: length = a CHOSEN dimension
(e.g. 300) — DENSE (every
value can be non-zero) —
similar words are
GEOMETRICALLY close
─────────────────────────────────────────
Famous Embedding Arithmetic (an Illustrative, Well-Known Result)
─────────────────────────────────────────
vector("king") - vector("man") + vector("woman") ≈ vector("queen")
This isn't magic — it's a direct, empirical CONSEQUENCE of
the distributional hypothesis (Section 4): "king" and "queen"
tend to appear in similar ROYALTY-related contexts, while
differing specifically along a GENDER-related direction that
ALSO separates "man" and "woman" — and the embedding
training process (Chapter 2) discovers this structure purely
from co-occurrence patterns in text.
─────────────────────────────────────────
6. Embeddings in PyTorch
import torch
import torch.nn as nn
vocab_size = 10000
embedding_dim = 300
embedding_layer = nn.Embedding(vocab_size, embedding_dim)
# A "lookup table" — given a word's INDEX, returns its embedding VECTOR
word_index = torch.tensor([42]) # e.g. the index for "cat" in some vocabulary
word_vector = embedding_layer(word_index)
print(f"Embedding shape: {word_vector.shape}") # [1, 300]
# In an UNTRAINED embedding layer, vectors are RANDOM — meaning emerges
# only through training (Chapter 2), or by loading PRETRAINED weights
# (Chapter 3, and DL Notes Module 7's transfer learning concept)Why nn.Embedding Is Just a Lookup Table
─────────────────────────────────────────
Internally, nn.Embedding stores a (vocab_size × embedding_dim)
MATRIX — one row per word. Looking up a word's embedding is
simply SELECTING that word's row — mathematically equivalent
to multiplying a one-hot vector (Section 2) by this matrix,
but implemented far more efficiently as a direct lookup
rather than an actual matrix multiplication.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- One-hot encoding represents words as sparse vectors with a single 1, but treats every pair of distinct words as equally dissimilar, capturing no semantic relationships.
- The distributional hypothesis — words appearing in similar contexts have similar meanings — is the theoretical foundation for every embedding technique that follows.
- Dense embeddings represent words as vectors where geometric closeness reflects semantic similarity, discovered purely from co-occurrence patterns in text.
- PyTorch's
nn.Embeddingis simply a lookup table mapping word indices to their (learned or pretrained) vector representations.
Concept Check
- Why does the dot product between any two distinct one-hot vectors always equal zero, and why is that a problem?
- State the distributional hypothesis in your own words, with an example.
- What does an
nn.Embeddinglayer actually store internally, and how does looking up a word's embedding work?
Next Chapter
→ Chapter 2: Word2Vec — CBOW & Skip-Gram
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index