Supervised Learning Regression
Simple & Multiple Linear Regression
Nearly every concept covered for MORE complex algorithms
JrCodex·9 min read
Jr Codex ML Notes
Level: Intermediate Prerequisites: Module 2: Data Preparation for ML Time to complete: ~30 minutes
Table of Contents
- The Simplest Possible Model
- Simple Linear Regression
- How the Line Is Actually Found: Least Squares
- Gradient Descent — an Alternative Way to Find the Line
- Multiple Linear Regression
- Interpreting Coefficients
- The Assumptions of Linear Regression
- A Complete Worked Example
- Summary & Next Steps
1. The Simplest Possible Model
Linear regression predicts a continuous target as a weighted sum of input features — the simplest, most interpretable, and often surprisingly effective model in all of supervised learning (Module 1, Chapter 2).
Why Start Here
─────────────────────────────────────────
Nearly every concept covered for MORE complex algorithms
later in this curriculum (loss functions, gradient descent,
regularization, overfitting) is easiest to understand FIRST
in linear regression's simple, transparent setting.
─────────────────────────────────────────
2. Simple Linear Regression
Predicts a target y from a single feature x, fitting the best straight line through the data.
The Model
─────────────────────────────────────────
y = β₀ + β₁x
β₀ (intercept): predicted y when x = 0
β₁ (slope/coefficient): how much y changes per UNIT increase in x
─────────────────────────────────────────
import numpy as np
from sklearn.linear_model import LinearRegression
# Predicting exam score from hours studied
hours_studied = np.array([[1], [2], [3], [4], [5], [6]])
exam_scores = np.array([52, 58, 65, 70, 78, 82])
model = LinearRegression()
model.fit(hours_studied, exam_scores)
print(f"Intercept (β₀): {model.intercept_:.2f}") # ~46.6
print(f"Slope (β₁): {model.coef_[0]:.2f}") # ~6.1
# Predicting a NEW value
predicted_score = model.predict([[7]])
print(f"Predicted score for 7 hours: {predicted_score[0]:.2f}") # ~89.3Reading the Fitted Line
─────────────────────────────────────────
score = 46.6 + 6.1 * hours_studied
Each ADDITIONAL hour of studying is associated with an
estimated 6.1-point increase in exam score, on average
(given this data).
─────────────────────────────────────────
3. How the Line Is Actually Found: Least Squares
The "best" line is defined as the one minimizing the total squared distance between actual and predicted values — the Ordinary Least Squares (OLS) method.
The Loss Function Being Minimized
─────────────────────────────────────────
Sum of Squared Errors (SSE) = Σ (actual_i - predicted_i)²
Linear regression finds the β₀, β₁ that make THIS SUM as
SMALL as possible, across every point in the training data.
─────────────────────────────────────────
import numpy as np
def sum_squared_errors(y_actual, y_predicted):
return np.sum((y_actual - y_predicted) ** 2)
predictions = model.predict(hours_studied)
sse = sum_squared_errors(exam_scores, predictions)
print(f"Sum of squared errors: {sse:.2f}")
# Why SQUARED errors, not just raw differences?
# 1. Squaring makes all errors POSITIVE (a -5 error and a +5 error
# both count as "wrong by 5", not canceling each other out)
# 2. Squaring PENALIZES large errors MORE heavily than small ones —
# an error of 10 contributes 100, while two errors of 5 only
# contribute 25 each (50 total) — encourages consistently
# small errors over occasional huge onesFor simple and multiple linear regression, OLS has a closed-form mathematical solution — no iterative search needed; the exact optimal coefficients can be computed directly via matrix algebra. Section 4 covers the alternative, iterative method used more broadly across ML.
4. Gradient Descent — an Alternative Way to Find the Line
While OLS has a direct formula, gradient descent is the general-purpose optimization technique used across most of machine learning (including neural networks, covered in the Deep Learning Notes) — worth understanding here, in the simplest possible setting.
import numpy as np
def gradient_descent_linear_regression(X, y, learning_rate=0.01, iterations=1000):
"""A from-scratch implementation, for a SINGLE feature."""
m = len(y)
beta0, beta1 = 0.0, 0.0 # start with a random/zero guess
for i in range(iterations):
predictions = beta0 + beta1 * X
error = predictions - y
# The GRADIENT — the direction of STEEPEST INCREASE in error;
# we step in the OPPOSITE direction to REDUCE error
gradient_beta0 = (2/m) * np.sum(error)
gradient_beta1 = (2/m) * np.sum(error * X)
beta0 -= learning_rate * gradient_beta0
beta1 -= learning_rate * gradient_beta1
return beta0, beta1
X = np.array([1, 2, 3, 4, 5, 6], dtype=float)
y = np.array([52, 58, 65, 70, 78, 82], dtype=float)
beta0, beta1 = gradient_descent_linear_regression(X, y, learning_rate=0.05, iterations=5000)
print(f"Gradient descent result: intercept={beta0:.2f}, slope={beta1:.2f}") # converges close to OLS's answerGradient Descent, Conceptually — Directly Connects to the AI Notes' Local Search
─────────────────────────────────────────
Imagine STANDING on a hill (the "loss landscape") where
height = total squared error, and position = the current
β₀, β₁ guess.
Gradient descent repeatedly takes a small STEP DOWNHILL
(in the direction that REDUCES error the fastest), eventually
settling at the BOTTOM (minimum error).
This is EXACTLY the AI Notes' Module 2, Chapter 4 hill-
climbing/local-search idea — just descending instead of
climbing, and moving CONTINUOUSLY instead of jumping
between discrete states.
─────────────────────────────────────────
Why bother with gradient descent when OLS has an exact formula? Because many later algorithms (logistic regression, neural networks) have NO closed-form solution — gradient descent (or its many variants) is often the ONLY practical way to fit them, so understanding it here, where you can also verify it against the exact OLS answer, builds essential intuition.
5. Multiple Linear Regression
Extends the same idea to multiple input features simultaneously.
The Model, Extended
─────────────────────────────────────────
y = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ
Each xᵢ is a DIFFERENT feature; each βᵢ is that feature's
OWN coefficient, learned independently
─────────────────────────────────────────
from sklearn.linear_model import LinearRegression
import numpy as np
# Predicting house price from MULTIPLE features
X = np.array([
[1200, 2, 10], # [sqft, bedrooms, age_years]
[1800, 3, 5],
[2400, 4, 2],
[1500, 3, 15],
])
y = np.array([250000, 340000, 410000, 275000])
model = LinearRegression()
model.fit(X, y)
print(f"Intercept: {model.intercept_:.2f}")
print(f"Coefficients: {model.coef_}") # one coefficient PER feature
new_house = [[2000, 3, 8]]
print(f"Predicted price: {model.predict(new_house)[0]:.2f}")6. Interpreting Coefficients
The genuine, standout strength of linear regression over most later algorithms (Modules 4-5): each coefficient has a direct, precise interpretation.
feature_names = ["sqft", "bedrooms", "age_years"]
for name, coef in zip(feature_names, model.coef_):
print(f"{name}: {coef:+.2f}")
# e.g.:
# sqft: +120.50 → each additional sqft is associated with
# a $120.50 increase in price, HOLDING
# all other features constant
# bedrooms: +5000.00 → each additional bedroom is associated
# with a $5,000 increase, holding
# sqft and age constant
# age_years: -1200.00 → each additional year of age is
# associated with a $1,200 DECREASEThe Crucial Caveat: "Holding All Else Constant"
─────────────────────────────────────────
A coefficient's interpretation ONLY holds when every OTHER
feature stays FIXED — in reality, features are often
CORRELATED with each other (e.g. bigger houses tend to have
MORE bedrooms), which can make individual coefficients harder
to interpret cleanly. This connects directly to Module 2,
Chapter 2's dummy variable trap and multicollinearity
discussion.
─────────────────────────────────────────
This interpretability is EXACTLY why linear regression remains widely used even when more accurate but opaque alternatives exist — directly connecting to the AI Notes' Explainable AI chapter's discussion of interpretable-by-design models.
7. The Assumptions of Linear Regression
Linear regression's mathematical guarantees (and its coefficients' clean interpretation) rely on several assumptions about the data — worth knowing, since violated assumptions can silently produce misleading results.
Key Assumptions
─────────────────────────────────────────
Linearity: the TRUE relationship is actually
(approximately) linear — Chapter 3
covers what to do when it's not
Independence: observations don't influence each
other (violated by, e.g., time
series data with autocorrelation —
Data Science Notes' time series chapter)
Homoscedasticity: the SPREAD of errors is roughly
CONSTANT across all predicted
values (not, e.g., growing
larger for bigger predictions)
Normality of residuals: the ERRORS (actual - predicted)
are approximately normally
distributed (Data Science
Notes, Module 1, Ch.3) —
mainly matters for statistical
inference (confidence
intervals on coefficients),
less for pure prediction
No severe multicollinearity: features aren't extremely
correlated WITH EACH OTHER
(Module 2, Ch.2's dummy
variable trap)
─────────────────────────────────────────
import matplotlib.pyplot as plt
residuals = y - model.predict(X)
# A residual plot is the standard diagnostic check
plt.scatter(model.predict(X), residuals)
plt.axhline(y=0, color="red", linestyle="--")
plt.xlabel("Predicted Values")
plt.ylabel("Residuals")
plt.title("Residual Plot — should show NO clear pattern if assumptions hold")Reading a Residual Plot
─────────────────────────────────────────
Good (assumptions hold): residuals scattered RANDOMLY
around zero, no pattern
Bad (non-linearity): a CURVED pattern in residuals
→ the true relationship isn't
linear (Chapter 3's territory)
Bad (heteroscedasticity): residuals FAN OUT wider as
predictions increase
→ violates the constant-
spread assumption
─────────────────────────────────────────
8. A Complete Worked Example
Combining everything — including Module 2's proper preprocessing pipeline:
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
np.random.seed(42)
n_samples = 200
sqft = np.random.uniform(800, 3500, n_samples)
bedrooms = np.random.randint(1, 6, n_samples)
age = np.random.uniform(0, 40, n_samples)
price = 50000 + 120 * sqft + 8000 * bedrooms - 1000 * age + np.random.normal(0, 20000, n_samples)
X = np.column_stack([sqft, bedrooms, age])
y = price
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, predictions))
r2 = r2_score(y_test, predictions) # Chapter 4 covers these metrics properly
print(f"RMSE: ${rmse:,.2f}")
print(f"R²: {r2:.3f}")
print(f"Learned coefficients: {dict(zip(['sqft', 'bedrooms', 'age'], model.coef_))}")9. Summary & Next Steps
Key Takeaways
- Linear regression predicts a continuous target as a weighted sum of features, fit by minimizing the sum of squared errors (Ordinary Least Squares).
- Gradient descent is a general-purpose, iterative alternative to OLS's exact formula — conceptually identical to the AI Notes' local search/hill climbing, and essential for algorithms without a closed-form solution.
- Each coefficient has a direct interpretation ("each unit increase in this feature is associated with this change in the target, holding others constant") — linear regression's defining strength over more opaque algorithms.
- Linear regression relies on assumptions (linearity, independence, homoscedasticity, limited multicollinearity) — violated assumptions, checkable via residual plots, motivate the techniques in Chapters 2-3.
Concept Check
- Why does Ordinary Least Squares use squared errors rather than raw (signed) differences?
- How does gradient descent conceptually relate to the AI Notes' local search techniques?
- What does a curved pattern in a residual plot suggest about the model's assumptions?
Next Chapter
→ Chapter 2: Regularization — Ridge, Lasso & Elastic Net
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index