Classical NLP: Bag-of-Words, TF-IDF & Naive Bayes

Jr Codex NLP & LLM Notes

Level: Beginner Prerequisites: Chapter 2: Text Preprocessing Fundamentals Time to complete: ~25 minutes

Table of Contents

1. Representing Text as Numbers
2. Bag-of-Words
3. The Problem With Raw Counts
4. TF-IDF
5. Naive Bayes for Text Classification
6. A Complete Classical Pipeline
7. Why Classical Methods Still Matter
8. Summary & Next Steps

1. Representing Text as Numbers

Every ML algorithm in the ML Notes requires numerical input — text must be converted into numbers before any classifier can use it. This chapter covers the classical (pre-embedding, pre-Transformer) approach to this problem.

2. Bag-of-Words

The bag-of-words (BoW) model represents a document as a vector of word counts, completely ignoring word order — hence "bag," not "sequence."

from sklearn.feature_extraction.text import CountVectorizer
 
documents = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are great pets",
]
 
vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(documents)
 
print(vectorizer.get_feature_names_out())
# ['and' 'are' 'cat' 'cats' 'dog' 'dogs' 'great' 'log' 'mat' 'on' 'pets' 'sat' 'the']
 
print(bow_matrix.toarray())
# [[0 0 1 0 0 0 0 0 1 1 0 1 2]     <- "the cat sat on the mat"
#  [0 0 0 0 1 0 0 1 0 1 0 1 2]     <- "the dog sat on the log"
#  [1 1 0 1 0 1 1 0 0 0 1 0 0]]    <- "cats and dogs are great pets"

What Gets LOST in a Bag-of-Words Representation
─────────────────────────────────────────
  "The dog bit the man" and "The man bit the dog" produce the
  EXACT SAME bag-of-words vector — directly echoing DL Notes,
  Module 5, Chapter 1's point about why order matters. This is
  bag-of-words' most fundamental limitation, one that word
  embeddings (Module 2) and Transformers (Module 3+) each
  progressively address better.
─────────────────────────────────────────

3. The Problem With Raw Counts

Raw word counts have an obvious flaw: common words ("the," "a") get very high counts in almost every document, regardless of what the document is actually about — drowning out genuinely distinguishing words.

4. TF-IDF

TF-IDF (Term Frequency-Inverse Document Frequency) rebalances raw counts, boosting words that are frequent within a document but rare across the whole collection — capturing the intuition that a document's most distinguishing words are its best content signal.

The Formula
─────────────────────────────────────────
  TF(word, doc)   = (count of word in doc) / (total words in doc)
  IDF(word)          = log( total documents / documents containing word )
  TF-IDF(word, doc)     = TF(word, doc) × IDF(word)

  A word that's COMMON within a document (high TF) but RARE
  across the collection (high IDF) gets a HIGH score — exactly
  the "distinguishing word" signal being sought.
─────────────────────────────────────────

from sklearn.feature_extraction.text import TfidfVectorizer
 
tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(documents)
 
import pandas as pd
df = pd.DataFrame(tfidf_matrix.toarray(), columns=tfidf_vectorizer.get_feature_names_out())
print(df.round(3))
 
# Note how "the" — appearing in ALL three documents — gets a LOW IDF weight,
# while distinguishing words like "cats," "great," "pets" get HIGHER weight
# in the document where they actually appear

5. Naive Bayes for Text Classification

Multinomial Naive Bayes — a specific variant of the ML Notes' Naive Bayes classifier — was, for decades, the dominant algorithm for text classification tasks like spam detection.

from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
 
texts = ["free money now", "meeting scheduled for tomorrow", "win a free prize",
         "please review the attached document", "claim your free gift today",
         "let's discuss the quarterly report"]
labels = [1, 0, 1, 0, 1, 0]      # 1 = spam, 0 = not spam
 
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(texts)
 
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.33, random_state=42)
 
model = MultinomialNB()
model.fit(X_train, y_train)
 
# Reusing the ML Notes' full classification evaluation toolkit directly
from sklearn.metrics import accuracy_score
predictions = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, predictions):.2f}")

Why Naive Bayes Worked Well for Text Specifically
─────────────────────────────────────────
  The "naive" independence assumption — that each word's
  presence is independent of every other word, GIVEN the
  class — is CLEARLY false for real language (echoing Section
  2's word-order loss), but works SURPRISINGLY well in
  practice for classification: what matters most for spam
  detection is often just WHICH distinctive words appear
  ("free," "prize," "claim"), not their precise grammatical
  relationship to each other.
─────────────────────────────────────────

6. A Complete Classical Pipeline

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
 
# Directly reusing the ML Notes' Pipeline pattern (Module 9, Chapter 1)
classical_nlp_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(stop_words="english", max_features=5000)),      # Ch.2's stopword removal
    ("classifier", MultinomialNB()),
])
 
classical_nlp_pipeline.fit(texts, labels)
new_prediction = classical_nlp_pipeline.predict(["free trial offer expires today"])
print(f"Prediction: {'spam' if new_prediction[0] == 1 else 'not spam'}")

7. Why Classical Methods Still Matter

When Bag-of-Words/TF-IDF + Naive Bayes Is STILL a Reasonable Choice
─────────────────────────────────────────
  - VERY small datasets, where a Transformer (Modules 3+)
      would badly overfit (DL Notes, Module 1's "classical ML
      often wins on small data" point)
  - Extremely TIGHT latency/compute budgets — a TF-IDF +
      Naive Bayes pipeline runs in microseconds, versus a
      Transformer's much higher inference cost
  - As a FAST baseline (ML Notes' "start simple" workflow
      principle) before reaching for a more complex,
      expensive Transformer-based approach
  - Full INTERPRETABILITY requirements — which exact words
      drove a prediction is directly inspectable, unlike a
      Transformer's distributed representations
─────────────────────────────────────────

For the vast majority of modern NLP applications, though, the rest of this curriculum's embedding-based and Transformer-based approaches substantially outperform this chapter's classical toolkit — precisely because they address bag-of-words' core limitation (Section 2): capturing word order and context, not just word presence.

8. Summary & Next Steps

Key Takeaways

• Bag-of-words represents documents as word count vectors, discarding word order entirely — the same limitation that motivates every technique in the rest of this curriculum.
• TF-IDF rebalances raw counts to emphasize words that are distinctive to a specific document, rather than common across the whole collection.
• Naive Bayes' independence assumption is technically false for language but works surprisingly well for text classification, since distinctive word presence often matters more than precise grammatical structure.
• Classical methods remain a reasonable choice for very small datasets, tight compute budgets, fast baselines, or full interpretability requirements — but are generally outperformed by embedding and Transformer-based approaches.

Module 1 Complete — What's Next

You now understand what NLP is, why language is hard, how to preprocess raw text, and the classical bag-of-words/TF-IDF/Naive Bayes toolkit. Module 2 addresses bag-of-words' central limitation directly: representing words as dense vectors that capture meaning and relationships, not just presence or absence.

Concept Check

1. Why does TF-IDF weight a word higher when it's frequent in one document but rare across the whole collection?
2. Why does Naive Bayes' false independence assumption still work reasonably well for text classification?
3. Name two situations where a classical bag-of-words pipeline might still be the right choice today.

Next Module

→ Module 2: Word Embeddings & Representations

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index