Feature Engineering & Selection for ML

Jr Codex ML Notes

Level: Intermediate Prerequisites: Chapter 3 Time to complete: ~25 minutes

Table of Contents

1. The Highest-Leverage Skill in Applied ML
2. Creating Interaction Features
3. Polynomial Features
4. Domain-Specific Feature Engineering
5. Why More Features Isn't Always Better
6. Filter Methods for Feature Selection
7. Wrapper Methods for Feature Selection
8. Embedded Methods for Feature Selection
9. Summary & Next Steps

1. The Highest-Leverage Skill in Applied ML

The Data Science Notes' feature engineering chapter covered the general techniques (binning, date extraction, text features). This chapter goes deeper into feature engineering and selection specifically for ML model performance — often the single highest-leverage activity in a modeling project, frequently mattering more than which specific algorithm (Modules 3-5) you choose.

Why This Matters More Than Algorithm Choice, Often
─────────────────────────────────────────
  A well-engineered feature set fed into a SIMPLE algorithm
  (Module 3's linear regression) often outperforms a poorly-
  engineered feature set fed into a SOPHISTICATED algorithm
  (Module 5's gradient boosting) — the algorithm can only find
  patterns that are actually PRESENT and ACCESSIBLE in the
  features it's given.
─────────────────────────────────────────

2. Creating Interaction Features

Sometimes the combination of two features carries information neither carries alone.

import pandas as pd
 
df = pd.DataFrame({
    "bedrooms": [2, 3, 4, 2],
    "bathrooms": [1, 2, 3, 2],
})
 
# An interaction feature — the RATIO can matter more than either raw number
df["bed_bath_ratio"] = df["bedrooms"] / df["bathrooms"]
 
# A multiplicative interaction — captures COMBINED effect
df["bed_times_bath"] = df["bedrooms"] * df["bathrooms"]
 
print(df)

Why Interactions Matter
─────────────────────────────────────────
  A house with 4 bedrooms and 1 bathroom is a VERY different
  property than one with 4 bedrooms and 4 bathrooms — but a
  MODEL given only the two raw numbers separately (especially
  Module 3's LINEAR regression, which assumes each feature
  contributes INDEPENDENTLY) might not automatically discover
  this relationship on its own.

  Explicitly engineering the ratio/product gives a linear model
  direct access to a pattern it couldn't otherwise represent —
  though NON-linear models (Module 4's decision trees, Module 5's
  ensembles) can sometimes discover simple interactions
  automatically, from raw features alone.
─────────────────────────────────────────

3. Polynomial Features

Automatically generates interaction and power terms, rather than hand-crafting them one at a time — directly connects to Chapter 3's earlier discussion of non-linear regression (previewed here, covered fully in Module 3, Chapter 3).

from sklearn.preprocessing import PolynomialFeatures
import numpy as np
 
X = np.array([[2, 3], [4, 5]])      # 2 features
 
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
 
print(poly.get_feature_names_out())      # ['x0', 'x1', 'x0^2', 'x0 x1', 'x1^2']
print(X_poly)
# [[2, 3, 4, 6, 9], [4, 5, 16, 20, 25]]

What Just Got Created
─────────────────────────────────────────
  Original:     x0, x1
  degree=2:        x0, x1, x0², x0·x1, x1²

  This gives a LINEAR model (Module 3) access to CURVED,
  interacting relationships, without switching to a fundamentally
  different algorithm — the model is still linear in these NEW
  features, even though the relationship to the ORIGINAL features
  is now non-linear.
─────────────────────────────────────────

Caution: polynomial features multiply quickly — with many original features and a high degree, the feature count explodes combinatorially, risking exactly the overfitting Module 1, Chapter 4 warned about (more features than the data can reliably support).

4. Domain-Specific Feature Engineering

The single most valuable form of feature engineering usually comes from genuine understanding of the problem domain, not generic transformations.

import pandas as pd
 
df = pd.DataFrame({
    "signup_date": pd.to_datetime(["2024-01-15", "2023-06-20"]),
    "last_purchase_date": pd.to_datetime(["2024-06-01", "2024-06-01"]),
})
 
# Domain knowledge: "customer tenure" and "recency" are often FAR more
# predictive than the raw dates themselves (directly echoing the
# Data Science Notes' date-feature-extraction chapter)
df["tenure_days"] = (df["last_purchase_date"] - df["signup_date"]).dt.days
df["days_since_last_purchase"] = (pd.Timestamp.now() - df["last_purchase_date"]).dt.days
 
print(df)

Domain Knowledge Beats Generic Transformation
─────────────────────────────────────────
  A data scientist who understands e-commerce KNOWS that
  "recency, frequency, monetary value" (RFM) are classically
  strong predictors of customer behavior — this insight
  can't be discovered by any automated feature engineering
  tool; it comes from DOMAIN EXPERTISE.

  This is exactly why the "highest-leverage skill" claim in
  Section 1 is often about DOMAIN UNDERSTANDING as much as
  technical transformation skill.
─────────────────────────────────────────

5. Why More Features Isn't Always Better

Directly connecting back to Module 1, Chapter 4's bias-variance tradeoff — more features generally increase model flexibility (potentially reducing bias) but also increase the risk of overfitting (increasing variance), especially with limited training data.

The Curse of Dimensionality (Preview of Module 6)
─────────────────────────────────────────
  As the NUMBER of features grows, the amount of data needed
  to reliably learn patterns grows EXPONENTIALLY — with too
  many features relative to your dataset size, most of the
  "space" between data points becomes empty, and distance-based
  algorithms (Module 4's KNN) in particular start to behave
  unreliably.

  This is exactly why Module 6 dedicates two full chapters to
  DIMENSIONALITY REDUCTION — reducing feature count while
  preserving useful information.
─────────────────────────────────────────

Feature selection is the direct response to this problem: deliberately removing redundant or unhelpful features, rather than only ever adding more (Sections 2-4).

6. Filter Methods for Feature Selection

Select features based on their statistical relationship to the target, independent of any specific model — fast, and a good first pass.

from sklearn.feature_selection import SelectKBest, f_classif
import numpy as np
 
X = np.random.rand(200, 10)      # 200 samples, 10 features (some genuinely useful, some noise)
y = (X[:, 0] + X[:, 3] > 1).astype(int)      # only features 0 and 3 ACTUALLY matter
 
selector = SelectKBest(score_func=f_classif, k=2)      # keep the TOP 2 features
X_selected = selector.fit_transform(X, y)
 
print(selector.get_support(indices=True))      # likely [0, 3] — correctly identifies the useful ones

Correlation-Based Filtering (Directly from the Data Science Notes)
─────────────────────────────────────────
  Compute each feature's correlation (Data Science Notes, Module
  1, Ch.5) with the target, and drop features with very LOW
  correlation — a simple, fast, but LINEAR-relationship-only
  filtering approach.
─────────────────────────────────────────

Filter methods are fast because they never train a model — but this is also their limitation: they can't detect feature interactions (Section 2) or account for how features behave together within a specific algorithm.

7. Wrapper Methods for Feature Selection

Select features by actually training models with different feature subsets and comparing performance — slower, but accounts for how features interact within the specific algorithm being used.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
 
model = LogisticRegression()
selector = RFE(model, n_features_to_select=2)      # Recursive Feature Elimination
selector.fit(X, y)
 
print(selector.support_)      # boolean mask of SELECTED features
print(selector.ranking_)         # ranking of ALL features by importance

Recursive Feature Elimination (RFE), Conceptually
─────────────────────────────────────────
  1. Train the model using ALL features
  2. Identify the LEAST important feature (by the model's own
     internal importance measure)
  3. REMOVE it, retrain on the remaining features
  4. Repeat until only the desired number of features remain
─────────────────────────────────────────

Wrapper methods are more accurate but far more computationally expensive — they require training many models (one per elimination step), versus a filter method's single pass of statistical tests.

8. Embedded Methods for Feature Selection

Feature selection happens automatically, built into the training process of certain algorithms — the most efficient approach, since it requires no separate selection step at all.

from sklearn.linear_model import Lasso
import numpy as np
 
model = Lasso(alpha=0.1)      # Module 3, Chapter 2 covers Lasso's regularization in depth
model.fit(X, y)
 
print(model.coef_)      # some coefficients will be EXACTLY zero — automatically "selected out"
selected_features = np.where(model.coef_ != 0)[0]
print(f"Features selected by Lasso: {selected_features}")

Why This Is Called "Embedded"
─────────────────────────────────────────
  Lasso regression (Module 3, Ch.2) has a built-in property:
  its regularization penalty can shrink some coefficients ALL
  THE WAY to zero — effectively performing feature selection as
  a NATURAL SIDE EFFECT of training, not a separate step.

  Tree-based models (Module 4's decision trees, Module 5's
  ensembles) similarly provide built-in FEATURE IMPORTANCE
  scores as a natural byproduct of how they're trained.
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

• Feature engineering — creating interaction terms, polynomial features, and domain-specific derived features — is often the single highest-leverage activity in a modeling project, sometimes mattering more than algorithm choice.
• More features aren't automatically better: too many features relative to data size risks overfitting and the curse of dimensionality, motivating deliberate feature selection.
• Filter methods (correlation, statistical tests) are fast but ignore feature interactions and model-specific behavior; wrapper methods (RFE) are more accurate but computationally expensive, since they retrain models repeatedly.
• Embedded methods (Lasso's automatic coefficient shrinkage, tree-based feature importance) perform selection as a natural byproduct of training, requiring no separate selection step.

Concept Check

1. Why might a linear regression model benefit from an explicitly engineered interaction feature, while a decision tree might not need it as much?
2. What's the key trade-off between filter methods and wrapper methods for feature selection?
3. Why is Lasso regression described as having "embedded" feature selection?

Next Chapter

→ Chapter 5: Handling Imbalanced Datasets

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index