Neural Networks From Scratch
Activation Functions
Without an activation function, stacking dense layers (Chapter 1) is mathematically pointless — a composition of purely linear functions is still just a linear
Jr Codex Deep Learning Notes
Level: Beginner–Intermediate Prerequisites: Chapter 1: The Perceptron & the Multi-Layer Perceptron Time to complete: ~20 minutes
Table of Contents
- Why Activation Functions Are Necessary
- Sigmoid
- Tanh
- ReLU
- ReLU Variants
- Softmax — for Multi-Class Output
- Choosing an Activation Function
- Summary & Next Steps
1. Why Activation Functions Are Necessary
Without an activation function, stacking dense layers (Chapter 1) is mathematically pointless — a composition of purely linear functions is still just a linear function, no matter how many layers you stack.
Why Linear Layers Alone Collapse
─────────────────────────────────────────
Layer 1: z1 = W1·x + b1
Layer 2: z2 = W2·z1 + b2 = W2·(W1·x + b1) + b2 = (W2·W1)·x + (W2·b1 + b2)
This is ALGEBRAICALLY IDENTICAL to a single linear layer with
weight matrix (W2·W1) and bias (W2·b1 + b2) — TEN stacked
linear layers would still only be able to represent what ONE
linear layer (i.e. linear regression, from ML Notes) can.
─────────────────────────────────────────
The activation function's non-linearity is what allows depth to actually add representational power — solving problems like XOR (Module 1, Chapter 2) that no linear model can.
2. Sigmoid
The historical default, still used today for binary classification output layers — directly the same function used in logistic regression.
import numpy as np
import torch
def sigmoid(z):
return 1 / (1 + np.exp(-z))
# Squashes ANY real number into the range (0, 1) — interpretable as a probability
print(sigmoid(-5)) # ~0.007 — close to 0
print(sigmoid(0)) # 0.5
print(sigmoid(5)) # ~0.993 — close to 1Sigmoid's Major Problem: Vanishing Gradients
─────────────────────────────────────────
For LARGE positive or negative inputs, sigmoid's curve is
nearly FLAT — its gradient (slope) approaches ZERO.
During backpropagation (Chapter 4), a near-zero gradient
means almost NO learning signal passes backward through
that neuron — in a DEEP network, this compounds across many
layers, and early layers can stop learning entirely. This
is called the VANISHING GRADIENT problem, covered in full
in Module 3, Chapter 6.
─────────────────────────────────────────
3. Tanh
A rescaled version of sigmoid, outputting values in (-1, 1) instead of (0, 1), centered at zero.
def tanh(z):
return np.tanh(z)
print(tanh(-5)) # ~-1.0
print(tanh(0)) # 0.0
print(tanh(5)) # ~1.0Being zero-centered often helps training converge faster than sigmoid, but tanh still suffers from the same vanishing gradient problem at extreme values — it was the standard choice for hidden layers before ReLU (Section 4) took over.
4. ReLU
ReLU (Rectified Linear Unit) is the modern default for hidden layers, and one of the specific algorithmic changes (alongside dropout) credited with AlexNet's 2012 breakthrough (Module 1, Chapter 3).
def relu(z):
return np.maximum(0, z)
print(relu(-3)) # 0
print(relu(0)) # 0
print(relu(3)) # 3 — passes positive values through UNCHANGEDWhy ReLU Solved the Vanishing Gradient Problem (Mostly)
─────────────────────────────────────────
For any POSITIVE input, ReLU's gradient is EXACTLY 1 — no
matter how large the input, the gradient never shrinks
toward zero the way sigmoid/tanh's does. This lets gradients
flow backward through many layers far more effectively,
which is a major reason very deep networks became trainable.
ReLU also computes almost instantly (just a comparison to
zero), unlike sigmoid/tanh which require an exponential —
a meaningful speed advantage at the scale of billions of
neuron activations per training step.
─────────────────────────────────────────
ReLU's Own Problem: "Dying ReLU"
─────────────────────────────────────────
For any NEGATIVE input, ReLU's output AND gradient are both
exactly zero. If a neuron's weights drift such that it
ALWAYS receives negative input, it outputs zero forever and
NEVER receives a gradient to update itself — it's
permanently "dead." Section 5's variants address this.
─────────────────────────────────────────
5. ReLU Variants
def leaky_relu(z, alpha=0.01):
return np.where(z > 0, z, alpha * z) # small NON-ZERO slope for negative inputs
def elu(z, alpha=1.0):
return np.where(z > 0, z, alpha * (np.exp(z) - 1)) # smooth negative curve
def gelu_approx(z):
"""GELU — used in most modern Transformers (Module 6), including BERT/GPT"""
return 0.5 * z * (1 + np.tanh(np.sqrt(2 / np.pi) * (z + 0.044715 * z**3)))Variant Comparison
─────────────────────────────────────────
Leaky ReLU: a small non-zero slope (e.g. 0.01) for
negative inputs — prevents neurons from
fully "dying," at negligible extra cost
ELU: a smooth exponential curve for negative
inputs — can improve training stability,
at slightly higher compute cost
GELU: a smooth, probabilistic variant used by
most modern Transformer architectures
(Module 6) — outperforms plain ReLU in
large language models specifically
─────────────────────────────────────────
6. Softmax — for Multi-Class Output
Unlike the previous functions (applied independently to each neuron), softmax is applied across an entire layer at once, converting a vector of raw scores ("logits") into a valid probability distribution that sums to 1.
def softmax(logits):
exp_logits = np.exp(logits - np.max(logits)) # subtract max for numerical stability
return exp_logits / np.sum(exp_logits)
logits = np.array([2.0, 1.0, 0.1])
probabilities = softmax(logits)
print(probabilities) # e.g. [0.659, 0.242, 0.099]
print(probabilities.sum()) # 1.0 — ALWAYS sums to exactly 1Softmax is used almost universally in the output layer for multi-class classification — directly generalizing sigmoid (binary classification's special case) to any number of classes, paired with cross-entropy loss (covered in Chapter 3).
7. Choosing an Activation Function
A Practical Decision Guide
─────────────────────────────────────────
Hidden layers: ReLU by default; try Leaky ReLU or
GELU if training seems unstable
or many neurons appear "dead"
Binary classification Sigmoid — outputs a single
output layer: probability
Multi-class classification Softmax — outputs a full
output layer: probability distribution
Regression output layer: usually NO activation
(raw linear output) —
the target isn't
bounded to (0,1)
─────────────────────────────────────────
import torch.nn as nn
# PyTorch's built-in versions — used directly in place of hand-written functions
relu = nn.ReLU()
sigmoid = nn.Sigmoid()
softmax = nn.Softmax(dim=-1) # dim=-1 applies softmax across the LAST dimension (the classes)8. Summary & Next Steps
Key Takeaways
- Without a non-linear activation function, stacking any number of dense layers is mathematically equivalent to a single linear layer.
- Sigmoid and tanh suffer from vanishing gradients at extreme input values, which slows or stalls training in deep networks.
- ReLU solved this for positive inputs (constant gradient of 1) and became the modern default for hidden layers, at the cost of the "dying ReLU" problem for negative inputs.
- Softmax converts a layer's raw scores into a probability distribution and is the standard choice for multi-class classification output layers.
Concept Check
- Why does stacking multiple linear layers without activation functions provide no more representational power than a single layer?
- What specific property of ReLU's gradient makes it more resistant to the vanishing gradient problem than sigmoid?
- When would you use no activation function at all on an output layer?
Next Chapter
→ Chapter 3: Forward Propagation & Loss Functions
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index