Foundations Of NLP
Text Preprocessing Fundamentals
Raw text — full of capitalization, punctuation, typos, and inconsistent formatting — is rarely fed directly into a model. This chapter covers the classical prep
JrCodex·6 min read
Jr Codex NLP & LLM Notes
Level: Beginner Prerequisites: Chapter 1: What Is NLP & Why Language Is Hard Time to complete: ~20 minutes
Table of Contents
- Why Raw Text Needs Preprocessing
- Tokenization — the First Step
- Normalization: Lowercasing and Cleaning
- Stopword Removal
- Stemming vs Lemmatization
- A Full Preprocessing Pipeline
- What Modern LLMs Do Differently
- Summary & Next Steps
1. Why Raw Text Needs Preprocessing
Raw text — full of capitalization, punctuation, typos, and inconsistent formatting — is rarely fed directly into a model. This chapter covers the classical preprocessing toolkit, which remains directly relevant for classical NLP methods (Chapter 3) and still echoes (in modified form) in how modern LLMs prepare text (Section 7, and Module 3).
2. Tokenization — the First Step
Tokenization splits raw text into smaller units — typically words or punctuation marks — that become the basic building blocks for everything downstream.
text = "Dr. Smith isn't going to the U.S. tomorrow."
# Naive whitespace splitting — clearly insufficient
naive_tokens = text.split()
print(naive_tokens)
# ['Dr.', 'Smith', "isn't", 'going', 'to', 'the', 'U.S.', 'tomorrow.']
# Problems: "Dr." keeps its period, "isn't" isn't split into "is" + "n't",
# "U.S." is ambiguous with sentence-ending periods
import nltk
nltk.download("punkt")
from nltk.tokenize import word_tokenize
proper_tokens = word_tokenize(text)
print(proper_tokens)
# ['Dr.', 'Smith', 'is', "n't", 'going', 'to', 'the', 'U.S.', 'tomorrow', '.']
# A proper tokenizer handles contractions and abbreviations using
# learned rules and exception listsWhy This Is Harder Than It Looks
─────────────────────────────────────────
Should "isn't" become ["isn't"] or ["is", "n't"]? Should
"U.S." keep its periods? Should a hyphenated word like
"state-of-the-art" be ONE token or FOUR? There's no single
universally "correct" answer — different tokenizers make
different choices, and the choice affects everything
downstream.
─────────────────────────────────────────
3. Normalization: Lowercasing and Cleaning
import re
def normalize_text(text):
text = text.lower() # "Apple" and "apple" treated as the SAME token
text = re.sub(r"[^\w\s]", "", text) # remove punctuation
text = re.sub(r"\d+", "<NUM>", text) # replace numbers with a placeholder
text = re.sub(r"\s+", " ", text).strip() # collapse multiple spaces
return text
print(normalize_text("Hello!!! I have 3 cats, and I LOVE them."))
# "hello i have <num> cats and i love them"The Tradeoff Lowercasing Introduces
─────────────────────────────────────────
Lowercasing helps a MODEL treat "Apple" (the sentence-
initial word) and "apple" (the fruit) as the same token —
usually desirable for reducing VOCABULARY size and data
sparsity. But it ALSO destroys a genuinely useful signal:
"Apple" (the COMPANY) vs "apple" (the fruit) becomes
indistinguishable by CASE alone — a real information loss
that matters more for some tasks (like NER) than others
(like topic classification).
─────────────────────────────────────────
4. Stopword Removal
Stopwords are extremely common words ("the," "is," "and," "a") that carry little distinguishing meaning for many tasks.
from nltk.corpus import stopwords
nltk.download("stopwords")
stop_words = set(stopwords.words("english"))
tokens = ["the", "cat", "sat", "on", "the", "mat"]
filtered = [word for word in tokens if word not in stop_words]
print(filtered) # ['cat', 'sat', 'mat']When Stopword Removal Helps vs Hurts
─────────────────────────────────────────
HELPS: classical bag-of-words models (Chapter 3) — removing
stopwords reduces noise and vocabulary size,
letting the model focus on CONTENT-bearing words
HURTS: tasks where FUNCTION words carry meaning — "to be
or NOT to be" loses its entire meaning if "not"
(often a stopword) is removed; sentiment
analysis and modern Transformer-based models
(Module 3+) generally do NOT remove stopwords,
since attention (DL Notes, Module 6) can learn
to weight them appropriately on its own
─────────────────────────────────────────
5. Stemming vs Lemmatization
Both reduce words to a common base form, but through very different means.
from nltk.stem import PorterStemmer, WordNetLemmatizer
nltk.download("wordnet")
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
words = ["running", "runs", "ran", "better", "studies"]
for word in words:
print(f"{word}: stem={stemmer.stem(word)}, lemma={lemmatizer.lemmatize(word, pos='v')}")
# running: stem=run, lemma=run
# runs: stem=run, lemma=run
# ran: stem=ran, lemma=run <- lemmatization correctly handles the IRREGULAR verb
# better: stem=better, lemma=good <- (with pos='a' for adjective) lemmatization understands
# "better" is the comparative of "good"
# studies: stem=studi, lemma=study <- stemming produces a NON-WORD ("studi")Stemming vs Lemmatization
─────────────────────────────────────────
Stemming: a CRUDE, rule-based chopping of word endings —
fast, but can produce non-words ("studi") and
handles irregular forms POORLY ("ran" stays
"ran," not "run")
Lemmatization: uses a DICTIONARY and (often) part-of-speech
information to find the correct DICTIONARY
form — slower, more accurate, correctly
handles irregular forms
─────────────────────────────────────────
6. A Full Preprocessing Pipeline
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
def preprocess_pipeline(text):
text = text.lower()
text = re.sub(r"[^\w\s]", "", text)
tokens = word_tokenize(text)
stop_words = set(stopwords.words("english"))
tokens = [t for t in tokens if t not in stop_words]
lemmatizer = WordNetLemmatizer()
tokens = [lemmatizer.lemmatize(t) for t in tokens]
return tokens
result = preprocess_pipeline("The cats were running quickly through the gardens!")
print(result) # ['cat', 'running' -> 'running' (default pos='n'), 'quickly', 'garden']7. What Modern LLMs Do Differently
A critical scope note for the rest of this curriculum: modern Transformer-based models (Modules 3-9) generally skip most of this chapter's steps entirely.
Why Modern LLMs Don't Lowercase, Remove Stopwords, or Stem
─────────────────────────────────────────
Case and exact word form CARRY information: "Apple" vs
"apple"
(Section 3),
"running"
vs "run"
(Section 5)
are
DIFFERENT
TOKENS
with
potentially
different
meaning or
grammatical
role —
throwing
this away
loses
information
a large
model could
otherwise
use
Subword tokenization (Module 3) handles rare/unseen words
far more gracefully than
this chapter's whole-word
approach — no need to
normalize away
"irregularity" the model
can learn to handle
directly
Attention (DL Notes, Module 6) lets the model learn
WHICH words matter in
context — manually
removing stopwords
PRE-EMPTS a decision
the model can make
better, contextually,
on its own
─────────────────────────────────────────
This chapter's techniques remain genuinely useful for classical methods (Chapter 3) and some lightweight production pipelines — but understand that Module 3 onward uses a fundamentally different preprocessing approach (subword tokenization, no lowercasing/stemming) suited to Transformer architectures.
8. Summary & Next Steps
Key Takeaways
- Tokenization splits raw text into words/punctuation, but even this "simple" first step involves genuinely ambiguous choices (contractions, abbreviations, hyphenation).
- Lowercasing and stopword removal reduce vocabulary size and noise for classical methods, but discard information (case, function words) that can matter for some tasks.
- Lemmatization uses dictionary knowledge to find correct base word forms, handling irregular words far better than crude, rule-based stemming.
- Modern Transformer-based LLMs (Module 3 onward) generally skip lowercasing, stopword removal, and stemming entirely — case and exact word form carry information that attention-based models can use directly.
Concept Check
- Why is tokenization harder than simply splitting text on whitespace?
- Give an example where lowercasing text loses meaningful information.
- Why do modern LLMs generally skip stopword removal, unlike classical NLP pipelines?
Next Chapter
→ Chapter 3: Classical NLP — Bag-of-Words, TF-IDF & Naive Bayes
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index