Neural Networks From Scratch
The Perceptron & the Multi-Layer Perceptron
Module 1, Chapter 2 introduced the artificial neuron: a weighted sum of inputs, plus a bias, passed through an activation function. This module builds the full
Jr Codex Deep Learning Notes
Level: Beginner–Intermediate Prerequisites: Module 1: Foundations of Deep Learning Time to complete: ~20 minutes
Table of Contents
- Recap: The Single Neuron
- Layers: Input, Hidden, Output
- Fully Connected (Dense) Layers
- Representing a Layer as Matrix Math
- Building an MLP in PyTorch
- How Many Layers, How Wide?
- Summary & Next Steps
1. Recap: The Single Neuron
Module 1, Chapter 2 introduced the artificial neuron: a weighted sum of inputs, plus a bias, passed through an activation function. This module builds the full mathematical picture of how many such neurons, connected together, form a Multi-Layer Perceptron (MLP) — the simplest deep learning architecture, and the direct ancestor of every architecture in this curriculum.
2. Layers: Input, Hidden, Output
The Anatomy of an MLP
─────────────────────────────────────────
Input layer: NOT really a layer of neurons — just the
raw feature values (e.g. pixel values,
or the same tabular features used in
ML Notes, Module 2)
Hidden layer(s): one or more layers of neurons, each
producing a NEW representation of the
input — this is where Module 1,
Section 2's "learned feature hierarchy"
actually happens
Output layer: produces the FINAL prediction — its
shape and activation depend on the
task (Section 5)
─────────────────────────────────────────
Input Hidden Layer 1 Hidden Layer 2 Output
(features) (learns simple (learns complex (prediction)
patterns) patterns)
x1 ──┐ ┌─ h1 ─┐ ┌─ h1' ─┐
x2 ──┼──────► ├─ h2 ─┤ ───────► ├─ h2' ─┤ ──────► y
x3 ──┤ ├─ h3 ─┤ └─ h3' ─┘
x4 ──┘ └─ h4 ─┘
3. Fully Connected (Dense) Layers
In an MLP, every neuron in one layer connects to every neuron in the next layer — this is called a fully connected or dense layer. Each connection has its own weight, so a layer with 4 inputs and 3 hidden neurons has 4×3 = 12 weights, plus 3 biases (one per hidden neuron).
Parameter Count Grows FAST
─────────────────────────────────────────
A layer going from 784 inputs (e.g. a 28x28 pixel image,
flattened) to 128 hidden neurons has:
784 × 128 = 100,352 weights, plus 128 biases
= 100,480 parameters — for JUST one layer
This is why real networks need GPUs (Module 1, Ch.4) and
careful training techniques (Module 3) — the parameter
counts involved are enormous compared to classical ML
models (ML Notes), where a logistic regression model might
have a few dozen parameters total.
─────────────────────────────────────────
4. Representing a Layer as Matrix Math
Computing every neuron's output one at a time would be slow. Instead, an entire layer's computation is expressed as a single matrix multiplication — this is why GPUs (built for fast matrix math) are so well suited to deep learning.
Layer Computation, Vectorized
─────────────────────────────────────────
For a layer with input vector x (size n) and m neurons:
Z = W·x + b
A = f(Z)
W: an (m × n) weight matrix — one ROW per neuron
b: an (m,) bias vector — one bias per neuron
Z: an (m,) vector of weighted sums — one per neuron
f: the activation function, applied ELEMENT-WISE
A: the layer's final output (m,) — fed to the next layer
─────────────────────────────────────────
import numpy as np
def dense_layer(x, W, b, activation_fn):
z = W @ x + b # matrix-vector multiplication — the ENTIRE layer at once
return activation_fn(z)
def relu(z):
return np.maximum(0, z) # covered fully in Chapter 2
x = np.array([0.5, 0.3, 0.9, 0.1]) # 4 input features
W = np.random.randn(3, 4) * 0.1 # 3 neurons, 4 inputs each
b = np.zeros(3)
hidden_output = dense_layer(x, W, b, relu)
print(f"Hidden layer output: {hidden_output}") # 3 values — one per hidden neuron5. Building an MLP in PyTorch
import torch
import torch.nn as nn
class SimpleMLP(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super().__init__()
self.layer1 = nn.Linear(input_size, hidden_size) # a fully connected layer
self.relu = nn.ReLU()
self.layer2 = nn.Linear(hidden_size, output_size)
def forward(self, x):
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
return x
model = SimpleMLP(input_size=4, hidden_size=8, output_size=1)
sample_input = torch.tensor([0.5, 0.3, 0.9, 0.1])
output = model(sample_input)
print(f"Model output: {output}")
# Inspect the number of trainable parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params}")nn.Linear(input_size, hidden_size) is PyTorch's built-in dense/fully-connected layer — it automatically creates and manages the weight matrix and bias vector from Section 4, and (as covered in Module 1, Chapter 4) automatically tracks gradients for training.
6. How Many Layers, How Wide?
Practical Guidance (Refined Further in Module 3)
─────────────────────────────────────────
Start SHALLOW and NARROW: 1-2 hidden layers is often
enough for simple tabular
problems — reflects the ML
Notes' "start simple" workflow
principle (Module 1, Ch.3)
Go DEEPER for complex, images, audio, and text
structured data: benefit from many more
layers (Modules 4-6)
Width (neurons per layer) trades off against
vs depth (number of layers): depth — very deep,
narrow networks and
shallow, very wide
networks BOTH tend
to underperform a
reasonably balanced
architecture
─────────────────────────────────────────
There is no formula that determines the "correct" architecture in advance — like hyperparameter tuning in the ML Notes, Module 7, architecture choice is typically driven by experimentation, established patterns for a given data type (covered per-architecture in Modules 4-6), and validation performance.
7. Summary & Next Steps
Key Takeaways
- An MLP is built from an input layer, one or more hidden layers, and an output layer, where each hidden layer learns a progressively more abstract representation.
- In a fully connected (dense) layer, every neuron connects to every neuron in the next layer, and parameter counts grow quickly as layer sizes increase.
- A layer's entire computation is a single matrix multiplication (
Z = Wx + b), which is exactly why GPU-accelerated matrix math (Module 1, Chapter 4) matters so much for deep learning. - PyTorch's
nn.Linearimplements a dense layer directly, managing weights, biases, and gradient tracking automatically.
Concept Check
- Why is a layer's computation expressed as matrix multiplication rather than a loop over individual neurons?
- If a layer takes 100 inputs and has 50 neurons, how many weight parameters does it have (ignoring biases)?
- What does
nn.Linear(input_size, hidden_size)do internally?
Next Chapter
→ Chapter 2: Activation Functions
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index