Neural Networks From Scratch
Building a Neural Net From Scratch in NumPy
Every piece has now been covered individually: dense layers (Chapter 1), activations (Chapter 2), forward propagation and loss (Chapter 3), backpropagation (Cha
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 5: Gradient Descent & Its Variants Time to complete: ~35 minutes
Table of Contents
- The Goal of This Chapter
- The Dataset
- Designing the Network
- The Full Implementation
- Training the Network
- Verifying It Against PyTorch
- What This Exercise Proves
- Summary & Next Steps
1. The Goal of This Chapter
Every piece has now been covered individually: dense layers (Chapter 1), activations (Chapter 2), forward propagation and loss (Chapter 3), backpropagation (Chapter 4), and gradient descent (Chapter 5). This chapter assembles all five into a single, complete, working neural network — trained entirely in NumPy, with no deep learning framework — to prove the full pipeline is understood end to end before relying on PyTorch's automation for the rest of this curriculum.
2. The Dataset
A small, synthetic binary classification problem — solving the XOR problem (Module 1, Chapter 2) that a single neuron provably cannot solve.
import numpy as np
X = np.array([
[0, 0],
[0, 1],
[1, 0],
[1, 1],
])
y = np.array([0, 1, 1, 0]) # XOR: 1 when inputs DIFFER, 0 when they MATCH3. Designing the Network
Architecture (Following Chapter 1's Anatomy)
─────────────────────────────────────────
Input layer: 2 features (the two binary inputs)
Hidden layer: 4 neurons, ReLU activation (Ch.2)
Output layer: 1 neuron, sigmoid activation
(binary classification, Ch.2)
─────────────────────────────────────────
4. The Full Implementation
import numpy as np
np.random.seed(42)
class NeuralNetwork:
def __init__(self, input_size, hidden_size, output_size):
# Small random weights (Module 3 explains WHY not zero)
self.W1 = np.random.randn(hidden_size, input_size) * 0.5
self.b1 = np.zeros(hidden_size)
self.W2 = np.random.randn(output_size, hidden_size) * 0.5
self.b2 = np.zeros(output_size)
def relu(self, z):
return np.maximum(0, z)
def relu_derivative(self, z):
return (z > 0).astype(float)
def sigmoid(self, z):
return 1 / (1 + np.exp(-z))
def forward(self, x):
self.z1 = self.W1 @ x + self.b1 # Chapter 1's dense layer
self.a1 = self.relu(self.z1) # Chapter 2's activation
self.z2 = self.W2 @ self.a1 + self.b2
self.a2 = self.sigmoid(self.z2)
return self.a2
def backward(self, x, y_true, learning_rate):
# Chapter 4's backpropagation, applied step by step
dL_da2 = -(y_true / self.a2 - (1 - y_true) / (1 - self.a2))
dL_dz2 = dL_da2 * self.a2 * (1 - self.a2) # sigmoid derivative
dL_dW2 = np.outer(dL_dz2, self.a1)
dL_db2 = dL_dz2
dL_da1 = self.W2.T @ dL_dz2
dL_dz1 = dL_da1 * self.relu_derivative(self.z1)
dL_dW1 = np.outer(dL_dz1, x)
dL_db1 = dL_dz1
# Chapter 5's gradient descent weight update
self.W2 -= learning_rate * dL_dW2
self.b2 -= learning_rate * dL_db2
self.W1 -= learning_rate * dL_dW1
self.b1 -= learning_rate * dL_db1
def compute_loss(self, y_true, y_pred, epsilon=1e-15):
y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
return -(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))5. Training the Network
net = NeuralNetwork(input_size=2, hidden_size=4, output_size=1)
epochs = 5000
learning_rate = 0.5
for epoch in range(epochs):
total_loss = 0
for x, y_true in zip(X, y):
prediction = net.forward(x)
total_loss += net.compute_loss(np.array([y_true]), prediction)
net.backward(x, np.array([y_true]), learning_rate)
if epoch % 1000 == 0:
print(f"Epoch {epoch}, Loss: {total_loss[0]:.4f}")
print("\nFinal predictions:")
for x, y_true in zip(X, y):
prediction = net.forward(x)
print(f"Input: {x}, True: {y_true}, Predicted: {prediction[0]:.4f}")Expected Output (Approximately)
─────────────────────────────────────────
Epoch 0, Loss: ~2.7
Epoch 1000, Loss: ~0.05
Epoch 4000, Loss: ~0.01
Final predictions:
Input: [0 0], True: 0, Predicted: ~0.02
Input: [0 1], True: 1, Predicted: ~0.98
Input: [1 0], True: 1, Predicted: ~0.98
Input: [1 1], True: 0, Predicted: ~0.02
─────────────────────────────────────────
The network learns XOR correctly — direct proof that adding a hidden layer with a non-linear activation solved the exact limitation that caused the field's first AI winter (Module 1, Chapter 3).
6. Verifying It Against PyTorch
import torch
import torch.nn as nn
torch.manual_seed(42)
model = nn.Sequential(
nn.Linear(2, 4),
nn.ReLU(),
nn.Linear(4, 1),
nn.Sigmoid(),
)
X_torch = torch.tensor(X, dtype=torch.float32)
y_torch = torch.tensor(y, dtype=torch.float32).unsqueeze(1)
loss_fn = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(5000):
predictions = model(X_torch)
loss = loss_fn(predictions, y_torch)
optimizer.zero_grad() # clear PREVIOUS gradients (Module 3 explains why this matters)
loss.backward() # Chapter 4's backpropagation, fully automated
optimizer.step() # Chapter 5's weight update, fully automated
if epoch % 1000 == 0:
print(f"Epoch {epoch}, Loss: {loss.item():.4f}")
print("\nFinal predictions:")
print(model(X_torch).detach().numpy())Both implementations converge to the same solution — the hand-written NumPy version and PyTorch's automated version are performing the exact same mathematics; PyTorch is simply automating what Sections 4-5 wrote out explicitly.
7. What This Exercise Proves
The Full Pipeline, End to End
─────────────────────────────────────────
Chapter 1 (Dense layers) → self.W1 @ x + self.b1
Chapter 2 (Activations) → self.relu(), self.sigmoid()
Chapter 3 (Forward + Loss) → self.forward(),
self.compute_loss()
Chapter 4 (Backpropagation) → self.backward()
Chapter 5 (Gradient Descent) → the W -= lr *
dW update lines
─────────────────────────────────────────
From this chapter forward, this curriculum uses PyTorch's automated .backward() and built-in optimizers exclusively — but every one of those calls is now understood as shorthand for the exact mechanics implemented by hand here.
8. Summary & Next Steps
Key Takeaways
- A complete neural network is the composition of dense layers, activation functions, a forward pass, a loss function, backpropagation, and gradient descent — each covered individually in this module's prior chapters.
- A 2-layer network with a non-linear activation successfully learns XOR, something a single neuron provably cannot do.
- Hand-implementing backpropagation and gradient descent in NumPy produces results identical to PyTorch's automated equivalent, confirming both are performing the same underlying computation.
- From this point forward, this curriculum relies on PyTorch's automation — understood now as a shorthand for mechanics you've implemented yourself.
Module 2 Complete — What's Next
You've now built a working neural network from first principles. Module 3 addresses a critical follow-up question: plain gradient descent (as implemented here) works on tiny toy examples, but real networks need far more sophisticated training techniques — better optimizers, weight initialization strategies, normalization, and regularization — to train reliably at scale.
Concept Check
- Why does this chapter implement everything in NumPy first, rather than starting directly with PyTorch?
- What specific problem does the hidden layer solve that a single-layer network (Module 1, Chapter 2) cannot?
- What is
optimizer.zero_grad()doing, and why is it called beforeloss.backward()in every training loop?
Next Module
→ Module 3: Training Deep Networks Effectively
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index