Encoding Categorical Variables

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 1 Time to complete: ~25 minutes

Table of Contents

1. Why Encoding Is Necessary
2. One-Hot Encoding
3. Ordinal Encoding
4. The Dummy Variable Trap
5. High-Cardinality Categoricals
6. Target Encoding
7. Encoding the Target Variable Itself
8. Choosing the Right Encoding
9. Summary & Next Steps

1. Why Encoding Is Necessary

The Data Science Notes' feature engineering chapter introduced encoding briefly; this chapter goes deeper, since choosing the wrong encoding for a given algorithm (Modules 3-6) can silently hurt model performance in ways that are easy to miss.

import pandas as pd
 
df = pd.DataFrame({"color": ["red", "blue", "green", "red", "blue"]})
 
# Feeding raw text directly into most ML algorithms simply doesn't work —
# they need NUMBERS. But the CHOICE of how to convert matters enormously.

2. One-Hot Encoding

Creates one binary (0/1) column per category — the standard choice when categories have no inherent order.

import pandas as pd
 
df = pd.DataFrame({"color": ["red", "blue", "green", "red"]})
 
one_hot = pd.get_dummies(df["color"], prefix="color")
print(one_hot)
#    color_blue  color_green  color_red
# 0       False        False       True
# 1        True        False      False
# 2       False         True      False
# 3       False        False       True

from sklearn.preprocessing import OneHotEncoder
 
encoder = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
encoded = encoder.fit_transform(df[["color"]])
print(encoder.get_feature_names_out())      # ['color_blue', 'color_green', 'color_red']
print(encoded)

Why "No False Ordering" Matters
─────────────────────────────────────────
  If "red"=0, "blue"=1, "green"=2 were used directly as a
  SINGLE numeric column instead, many algorithms (especially
  Module 3's linear/logistic regression) would incorrectly
  interpret this as "green is somehow MORE than blue is more
  than red" — a meaningless, damaging assumption for a
  genuinely unordered category.
─────────────────────────────────────────

handle_unknown="ignore" is essential for production use — it prevents a crash if the deployed model later encounters a category (e.g. a new city) that never appeared in training data (a preview of Module 9's production concerns).

3. Ordinal Encoding

For categories with a genuine, meaningful order, ordinal encoding preserves that relationship as a single numeric column — directly appropriate where one-hot would throw away real information.

from sklearn.preprocessing import OrdinalEncoder
 
df = pd.DataFrame({"size": ["medium", "small", "large", "medium"]})
 
# Specify the TRUE order explicitly — never let the encoder guess alphabetically!
encoder = OrdinalEncoder(categories=[["small", "medium", "large"]])
df["size_encoded"] = encoder.fit_transform(df[["size"]])
print(df)
#      size  size_encoded
# 0  medium           1.0
# 1   small           0.0
# 2   large           2.0
# 3  medium           1.0

One-Hot vs Ordinal — the Rule from the Data Science Notes, Reapplied
─────────────────────────────────────────
  One-hot:   NO natural order (color, city, product category)
  Ordinal:      a REAL, meaningful order exists (size, education
                  level, satisfaction rating)

  Getting this wrong in EITHER direction hurts model quality:
  ordinal-encoding an unordered category implies a FALSE ranking;
  one-hot-encoding an ordered category THROWS AWAY the order
  information a model could have used directly.
─────────────────────────────────────────

4. The Dummy Variable Trap

A subtle issue specific to one-hot encoding, particularly relevant for Module 3's linear/logistic regression: including all one-hot columns creates perfect multicollinearity — one column is always fully predictable from the others.

import pandas as pd
 
df = pd.DataFrame({"color": ["red", "blue", "green"]})
one_hot_full = pd.get_dummies(df["color"])
print(one_hot_full)
#    blue  green  red
# 0 False False   True
# 1  True False  False
# 2 False  True  False
 
# If you know blue=0 AND green=0, you ALREADY know red=1 — the
# "red" column is REDUNDANT, fully determined by the other two
 
one_hot_dropped = pd.get_dummies(df["color"], drop_first=True)      # drops ONE column
print(one_hot_dropped)
#    blue  green
# 0 False False        ← this row IS "red" — implied by BOTH being False
# 1  True False
# 2 False  True

Why This Matters Specifically for Regression (Module 3)
─────────────────────────────────────────
  Linear/logistic regression's underlying math (Module 3, Ch.1)
  can become UNSTABLE or ill-defined when features are perfectly
  redundant with each other (multicollinearity) — dropping one
  dummy column removes this redundancy WITHOUT losing any
  actual information (it's still fully recoverable from the
  remaining columns).

  Tree-based models (Module 4's decision trees, Module 5's
  ensembles) are generally UNAFFECTED by this issue and don't
  strictly require dropping a column — but doing so anyway is
  a harmless, common convention.
─────────────────────────────────────────

5. High-Cardinality Categoricals

A categorical column with hundreds or thousands of unique values (e.g. "zip code," "product SKU") creates a serious problem for one-hot encoding: it explodes into an enormous number of columns, most containing very few examples each.

import pandas as pd
import numpy as np
 
# A column with 5,000 unique zip codes would create 5,000 one-hot columns!
n_zip_codes = 5000
print(f"One-hot encoding would create {n_zip_codes} new columns")

Problems With Naive One-Hot Encoding at High Cardinality
─────────────────────────────────────────
  - Massively increases the FEATURE COUNT — can worsen the
    curse of dimensionality (a concept Module 6's dimensionality
    reduction chapters address more fully)
  - Most resulting columns are EXTREMELY sparse (mostly zeros),
    with too few examples per category to learn anything reliable
  - Dramatically increases memory usage and training time
─────────────────────────────────────────

Common fixes: group rare categories into a single "Other" bucket before encoding, use ordinal/hash encoding as a more compact alternative, or — the most statistically principled option — target encoding (Section 6).

6. Target Encoding

Replaces each category with a number derived from the target variable's average value for that category — dramatically more compact than one-hot for high-cardinality features, at the cost of requiring careful handling to avoid leakage.

import pandas as pd
 
df = pd.DataFrame({
    "city": ["Boston", "NYC", "Boston", "LA", "NYC", "Boston"],
    "purchased": [1, 0, 1, 1, 1, 0],      # the TARGET variable (Module 4's classification)
})
 
target_means = df.groupby("city")["purchased"].mean()
print(target_means)
# city
# Boston    0.667
# LA        1.000
# NYC       0.500
 
df["city_target_encoded"] = df["city"].map(target_means)
print(df)

The Serious Leakage Risk Here
─────────────────────────────────────────
  Computing target_means using the FULL dataset (including
  what will become test data) leaks the TARGET's information
  directly into a FEATURE — a severe violation of Module 1,
  Chapter 5's anti-leakage principle, since the encoding for
  each row implicitly "knows" its own (or nearby rows') true label.

  CORRECT practice: compute target means using ONLY training
  data (within each cross-validation fold, ideally), then apply
  those FIXED values to validation/test data — never
  recomputed using data outside the current training fold.
─────────────────────────────────────────

from sklearn.model_selection import train_test_split
 
# Correct approach — compute encoding ONLY from training data
X_train, X_test, y_train, y_test = train_test_split(
    df[["city"]], df["purchased"], test_size=0.33, random_state=42
)
 
train_means = y_train.groupby(X_train["city"]).mean()      # ONLY from training data
X_train_encoded = X_train["city"].map(train_means)
X_test_encoded = X_test["city"].map(train_means)      # REUSE training-derived values, never recompute

7. Encoding the Target Variable Itself

For classification (Module 4), the label itself often needs encoding too — text categories like "spam"/"ham" become 0/1 (or 0/1/2/... for multi-class problems).

from sklearn.preprocessing import LabelEncoder
 
y = ["spam", "ham", "spam", "ham", "ham"]
label_encoder = LabelEncoder()
y_encoded = label_encoder.fit_transform(y)
print(y_encoded)                              # [1, 0, 1, 0, 0]
print(label_encoder.classes_)                   # ['ham', 'spam'] — alphabetical by default
 
# Converting back to original labels after prediction
predictions_encoded = [1, 0, 1]
print(label_encoder.inverse_transform(predictions_encoded))      # ['spam', 'ham', 'spam']

LabelEncoder is specifically for target/label encoding, not for input features — for input features with no order, OneHotEncoder (Section 2) remains the correct choice, since LabelEncoder's single-column numeric output would introduce the same false-ordering problem Section 2 warned against.

8. Choosing the Right Encoding

Decision Guide
─────────────────────────────────────────
  Category has NO natural order, LOW cardinality (< ~15 values)
      → One-hot encoding (Section 2)

  Category HAS a genuine, meaningful order
      → Ordinal encoding, with the order specified EXPLICITLY (Section 3)

  Category has HIGH cardinality (hundreds/thousands of values)
      → Target encoding (Section 6, with careful leakage
        prevention) or group rare categories into "Other" first

  Encoding the TARGET/label itself (classification)
      → LabelEncoder (Section 7)
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

• One-hot encoding creates a binary column per category — the correct default for unordered, low-cardinality categorical features.
• Ordinal encoding preserves a genuine order as a single numeric column — always specify the true order explicitly, never let an encoder guess alphabetically.
• The dummy variable trap (perfect multicollinearity from including all one-hot columns) matters specifically for regression models; drop one column as a harmless convention.
• High-cardinality categoricals overwhelm one-hot encoding — target encoding is more compact but requires strict leakage prevention, computing encodings only from training data.
• LabelEncoder is for encoding the target/label in classification, not for input features, where it would introduce a false ordering.

Concept Check

1. Why is one-hot encoding preferred over ordinal encoding for a "city" column, but not for a "shirt size" column?
2. Why does target encoding carry a serious data leakage risk, and how is it correctly prevented?
3. Why shouldn't LabelEncoder be used to encode an unordered input feature like "color"?

Next Chapter

→ Chapter 3: Feature Scaling & Normalization

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index