Feature Engineering Basics

Jr Codex Data Science Notes

Level: Intermediate Prerequisites: Chapter 4 Time to complete: ~25 minutes

What is a Feature, and Why Engineer One?
Creating Derived Features
Binning — Turning Continuous Values into Categories
Encoding Categorical Variables
Scaling Numeric Features
Extracting Features from Dates
Extracting Features from Text
Summary & Next Steps

1. What is a Feature, and Why Engineer One?

A feature is a single measurable column of input data used for analysis or modeling. Feature engineering is creating new, more useful features from the raw ones you started with — often the single highest-leverage activity in a data science project, more impactful than switching algorithms in the Machine Learning notes.

Raw Data                              Engineered Features
─────────────────────────────         ─────────────────────────────
  signup_date: "2024-01-15"             days_since_signup: 543
  purchase_amount: 45.00                 is_weekend_purchase: True
                                           purchase_amount_bracket: "medium"
─────────────────────────────         ─────────────────────────────

This chapter previews the topic; the Machine Learning notes' Data Preprocessing chapter goes further into feature engineering specifically for model training.

2. Creating Derived Features

The simplest form: combine or transform existing columns into something more directly useful.

import pandas as pd
 
df = pd.DataFrame({
    "first_name": ["Alice", "Bob"],
    "last_name": ["Smith", "Jones"],
    "price": [100, 250],
    "quantity": [3, 1],
})
 
df["full_name"] = df["first_name"] + " " + df["last_name"]
df["total_cost"] = df["price"] * df["quantity"]
df["price_per_unit"] = df["total_cost"] / df["quantity"]
 
# Ratio features are often more informative than either raw column alone
df["is_bulk_order"] = df["quantity"] >= 3

The best derived features usually come from domain knowledge — knowing that "price per unit" or "days since last purchase" matters requires understanding the actual business problem, not just the columns available.

3. Binning — Turning Continuous Values into Categories

Binning groups a continuous numeric variable into discrete buckets — useful for simplifying a variable, handling non-linear effects, or making a value more interpretable in a report.

import pandas as pd
 
df = pd.DataFrame({"age": [22, 35, 45, 19, 62, 51, 28]})
 
# Fixed-width bins
df["age_group"] = pd.cut(
    df["age"],
    bins=[0, 25, 40, 60, 100],
    labels=["18-25", "26-40", "41-60", "60+"],
)
print(df)
 
# Equal-FREQUENCY bins (quantiles) — each bin gets roughly the same NUMBER of records
df["age_quartile"] = pd.qcut(df["age"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])

cut() vs qcut()
─────────────────────────────────────────
  pd.cut()   → bins of EQUAL WIDTH (e.g. every 20 years) — some bins may end up
                with far more records than others
  pd.qcut()   → bins with roughly EQUAL COUNT of records — bin WIDTHS may vary
─────────────────────────────────────────

4. Encoding Categorical Variables

Most statistical and ML tools require numeric input — encoding converts categories (strings) into numbers, in a way that preserves their meaning as much as possible.

import pandas as pd
 
df = pd.DataFrame({
    "size": ["small", "medium", "large", "medium", "small"],
    "color": ["red", "blue", "green", "red", "blue"],
})
 
# One-hot encoding — one binary column PER category, no false ordering implied
one_hot = pd.get_dummies(df["color"], prefix="color")
print(one_hot)
#    color_blue  color_green  color_red
# 0       False        False       True
# 1        True        False      False
# ...
 
# Ordinal encoding — for categories that DO have a natural order
size_order = {"small": 0, "medium": 1, "large": 2}
df["size_encoded"] = df["size"].map(size_order)
print(df[["size", "size_encoded"]])

One-Hot vs Ordinal Encoding — When to Use Which
─────────────────────────────────────────
  One-hot:   NO natural order (color, city, product category)
              → creates multiple binary columns, avoids implying a false ranking

  Ordinal:    a REAL, meaningful order exists (small < medium < large)
               → a single numeric column, preserving the order relationship
─────────────────────────────────────────

Common mistake: ordinal-encoding a variable with no real order (e.g. {"red": 0, "blue": 1, "green": 2}) silently implies green > blue > red, which is meaningless and can mislead any statistical method or model that assumes numeric relationships matter.

5. Scaling Numeric Features

Different numeric columns often live on wildly different scales — age (0-100) versus income ($0-$500,000). Scaling puts them on a comparable footing.

import pandas as pd
 
df = pd.DataFrame({
    "age": [25, 45, 35, 50, 23],
    "income": [45000, 95000, 65000, 120000, 38000],
})
 
# Min-Max scaling — rescales to a fixed [0, 1] range
df["age_scaled"] = (df["age"] - df["age"].min()) / (df["age"].max() - df["age"].min())
 
# Standardization (Z-score) — rescales to mean=0, std_dev=1 (directly using Module 1's stats)
df["income_standardized"] = (df["income"] - df["income"].mean()) / df["income"].std()
 
print(df)

Min-Max Scaling                       Standardization (Z-score)
─────────────────────────────         ─────────────────────────────
  Result range: [0, 1]                  Result: mean=0, std dev=1
  Formula: (x - min) / (max - min)       Formula: (x - mean) / std_dev
  Sensitive to outliers (min/max          Less sensitive to outliers than
   can be extreme values)                  min-max, but not immune
─────────────────────────────         ─────────────────────────────

Why this matters: many statistical/ML methods (distance-based methods, gradient-based optimization) implicitly treat larger-scale numbers as "more important" unless everything is scaled comparably first — this is a recurring theme picked back up in the Machine Learning notes' preprocessing chapter.

6. Extracting Features from Dates

Building directly on Chapter 3's pd.to_datetime() — a single date column often hides several separately useful features.

import pandas as pd
 
df = pd.DataFrame({"order_date": pd.to_datetime(["2024-01-15", "2024-06-22", "2024-12-25"])})
 
df["year"] = df["order_date"].dt.year
df["month"] = df["order_date"].dt.month
df["day_of_week"] = df["order_date"].dt.day_name()
df["is_weekend"] = df["order_date"].dt.dayofweek >= 5      # Saturday=5, Sunday=6
df["quarter"] = df["order_date"].dt.quarter
df["days_since_order"] = (pd.Timestamp.now() - df["order_date"]).dt.days
 
print(df)

These derived features (day of week, weekend flag, quarter) often reveal patterns a raw timestamp never would — e.g. "orders spike on weekends" is invisible until is_weekend exists as its own column to group/filter by.

7. Extracting Features from Text

A brief preview — full text feature extraction (TF-IDF, embeddings) is covered in depth in the NLP/LLM notes, but a few simple, high-value features are worth knowing here.

import pandas as pd
 
df = pd.DataFrame({
    "review": [
        "This product is absolutely amazing!",
        "Terrible. Do not buy.",
        "It's okay, nothing special.",
    ]
})
 
df["review_length"] = df["review"].str.len()
df["word_count"] = df["review"].str.split().str.len()
df["has_exclamation"] = df["review"].str.contains("!")
df["contains_negative_word"] = df["review"].str.lower().str.contains("terrible|bad|awful")
 
print(df)

Even these simple counts (length, word count, punctuation presence) are legitimate, useful features on their own — full semantic understanding of text is a much deeper topic, picked up later in this curriculum's NLP/LLM notes.

8. Summary & Next Steps

Key Takeaways

Feature engineering — creating new, more useful columns from raw data — is often the highest-leverage activity in a data science project.
Binning (pd.cut/pd.qcut) turns continuous variables into interpretable categories; choose equal-width vs equal-frequency bins based on the goal.
One-hot encode categories with no natural order; ordinal-encode only when a real order exists — mixing these up silently implies false rankings.
Scaling (min-max or standardization) puts differently-scaled numeric columns on comparable footing — important for many downstream statistical and ML methods.
Dates and text both hide multiple simple, high-value derived features (day of week, weekend flag, word count, punctuation) beyond their raw form.

Module 2 Complete — Next Module

You can now acquire data from real sources, query it with SQL, clean it, handle missing/duplicate records deliberately, and engineer useful features. Module 3 puts all of this together into a structured process for actually exploring and understanding a dataset — Exploratory Data Analysis.

Concept Check

Why is ordinal-encoding a variable like color (with no real order) a common and misleading mistake?
What's the difference between pd.cut() and pd.qcut()?
Why might "day of week" extracted from a date column reveal a pattern the raw timestamp never would?

Next Chapter

→ Module 3: Exploratory Data Analysis

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index