Contextual Embeddings: Why Static Embeddings Fall Short

Jr Codex NLP & LLM Notes

Level: Intermediate Prerequisites: Chapter 3: GloVe & Evaluating Embeddings Time to complete: ~20 minutes

Table of Contents

1. The Polysemy Problem
2. Why Word2Vec and GloVe Can't Solve This
3. An Early Attempt: ELMo
4. The Real Solution: Attention
5. Static vs Contextual, Side by Side
6. A Preview: Contextual Embeddings in Practice
7. Summary & Next Steps

1. The Polysemy Problem

Polysemy — a single word having multiple, distinct meanings — is a direct instance of Module 1, Chapter 1's lexical ambiguity.

The Same Word, Different Meanings
─────────────────────────────────────────
  "I deposited money in the BANK."           (a financial institution)
  "We sat by the river BANK."                    (the edge of a river)
  "The pilot had to BANK the plane sharply."        (to tilt/turn)

  Three COMPLETELY different meanings for the identical word
  "bank," disambiguated ENTIRELY by surrounding context.
─────────────────────────────────────────

2. Why Word2Vec and GloVe Can't Solve This

Both Chapter 2's Word2Vec and Chapter 3's GloVe produce static embeddings: exactly one fixed vector per word, looked up identically regardless of the surrounding sentence.

# Regardless of context, a STATIC embedding gives the IDENTICAL vector:
sentence_1 = "I deposited money in the bank"
sentence_2 = "We sat by the river bank"
 
# glove_vectors["bank"] returns the SAME vector in BOTH cases —
# forced to be some kind of "average" blend of ALL of "bank"'s meanings,
# since the training process (Ch.2-3) never distinguished between them

Why This Is a Genuine, Practical Problem
─────────────────────────────────────────
  A static embedding for "bank" must somehow represent BOTH
  "financial institution" and "river edge" simultaneously, in
  ONE fixed vector — inevitably a COMPROMISE that captures
  NEITHER meaning particularly well, and provides NO way to
  tell a downstream model (Module 1, Chapter 3's classifier
  or ANY task built on top) which meaning is intended in a
  GIVEN sentence.
─────────────────────────────────────────

3. An Early Attempt: ELMo

ELMo (Embeddings from Language Models), introduced in 2018, was an early, partial solution: instead of a fixed lookup table, ELMo used a bidirectional LSTM (DL Notes, Module 5) to compute a word's embedding dynamically, based on the entire sentence it appeared in.

ELMo's Core Idea
─────────────────────────────────────────
  Run the ENTIRE sentence through a trained LSTM — the
  representation for "bank" that comes OUT is now influenced
  by every OTHER word in the sentence, via the LSTM's hidden
  state (DL Notes, Module 5, Ch.2) — producing a DIFFERENT
  vector for "bank" in "river bank" versus "bank account."
─────────────────────────────────────────

ELMo represented real, meaningful progress — but inherited RNNs' known limitations (DL Notes, Module 5, Chapter 5): sequential processing that couldn't parallelize, and imperfect handling of very long-range dependencies.

4. The Real Solution: Attention

DL Notes, Module 6 covered self-attention in full depth — the mechanism that ultimately solved this problem far more effectively than ELMo's LSTM approach.

Why Self-Attention Is a NATURAL Fit for Contextual Embeddings
─────────────────────────────────────────
  Self-attention (DL Notes, Module 6, Ch.2) computes a NEW
  representation for EVERY token, as a WEIGHTED COMBINATION of
  every OTHER token in the sequence — this is PRECISELY what a
  contextual embedding needs: "bank"'s representation should be
  a function of ITS SPECIFIC surrounding words ("river" vs
  "deposited"), computed DIRECTLY and IN PARALLEL (unlike
  ELMo's sequential LSTM), regardless of how far away the
  disambiguating word appears.
─────────────────────────────────────────

This is exactly why BERT (Module 4) and every subsequent Transformer-based language model produce genuinely contextual embeddings as a natural consequence of their architecture — not as a bolted-on afterthought, but as a direct result of self-attention's core mechanism.

5. Static vs Contextual, Side by Side

A Direct Comparison
─────────────────────────────────────────
  Static (Word2Vec/GloVe,      ONE vector per WORD, looked up
  Ch.2-3):                        from a fixed table — FAST,
                                     simple, but cannot
                                     disambiguate polysemy
                                     (Section 1)
  Contextual                         a DIFFERENT vector per
  (ELMo/BERT/GPT/                       WORD OCCURRENCE — computed
  modern LLMs):                           dynamically from the
                                             SPECIFIC surrounding
                                             sentence — correctly
                                             disambiguates polysemy,
                                             at higher computational
                                             cost
─────────────────────────────────────────

Where Static Embeddings STILL Have a Place
─────────────────────────────────────────
  - Extremely LOW-latency or resource-constrained applications,
      where a full Transformer forward pass is too expensive
  - As a FAST baseline or FEATURE for a classical ML pipeline
      (Module 1, Chapter 3's TF-IDF-style workflow, but with
      averaged embeddings instead of raw word counts)
  - Educational/interpretability purposes — static embeddings'
      simpler structure makes concepts like Chapter 1's vector
      arithmetic easier to directly inspect and demonstrate
─────────────────────────────────────────

6. A Preview: Contextual Embeddings in Practice

from transformers import AutoTokenizer, AutoModel
import torch
 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
 
def get_contextual_embedding(sentence, target_word):
    inputs = tokenizer(sentence, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
 
    token_embeddings = outputs.last_hidden_state[0]      # a DIFFERENT vector for EVERY token position
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
 
    target_idx = tokens.index(target_word)
    return token_embeddings[target_idx]
 
embedding_financial = get_contextual_embedding("I deposited money in the bank", "bank")
embedding_river = get_contextual_embedding("We sat by the river bank", "bank")
 
similarity = torch.cosine_similarity(embedding_financial.unsqueeze(0), embedding_river.unsqueeze(0))
print(f"Similarity between the two 'bank' meanings: {similarity.item():.3f}")
# Expect a RELATIVELY LOW similarity — confirming BERT represents
# these as MEANINGFULLY DIFFERENT, despite being the same surface word

This is precisely where Module 4's BERT picks up in full depth.

7. Summary & Next Steps

Key Takeaways

• Polysemy — one word, multiple meanings — cannot be represented by static embeddings, which assign exactly one fixed vector per word regardless of context.
• ELMo was an early, partial solution using bidirectional LSTMs to compute context-dependent embeddings, but inherited RNNs' sequential and long-range limitations.
• Self-attention (DL Notes, Module 6) is a natural, more effective fit for contextual embeddings — it computes each token's representation as a function of its specific surrounding context, in parallel and regardless of distance.
• Static embeddings retain a place in low-latency applications, classical ML pipelines, and educational contexts, but modern LLMs universally use contextual, Transformer-based representations.

Module 2 Complete — What's Next

You've now traced the full evolution of word representation: from one-hot vectors, through static Word2Vec/GloVe embeddings, to the contextual representations that modern Transformers produce naturally. Module 3 covers exactly how raw text becomes a Transformer's input — tokenization — and how these models are actually pretrained on massive amounts of text.

Concept Check

1. Why can't Word2Vec or GloVe distinguish between "river bank" and "bank account"?
2. What limitation did ELMo inherit from its LSTM-based architecture, despite solving the polysemy problem partially?
3. Why is self-attention a more natural fit for producing contextual embeddings than a sequential LSTM?

Next Module

→ Module 3: Tokenization & Pre-training Foundations

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index