Weight Initialization

Jr Codex Deep Learning Notes

Level: Intermediate Prerequisites: Module 2: Neural Networks from Scratch Time to complete: ~20 minutes

Why Initialization Isn't an Afterthought
The Zero-Initialization Trap
Random Initialization and Its Own Problems
Xavier/Glorot Initialization
He Initialization
Initialization in PyTorch
Summary & Next Steps

1. Why Initialization Isn't an Afterthought

Module 2's from-scratch implementation initialized weights with small random values (* 0.5) without much explanation. This module opens by explaining exactly why that choice matters — poor initialization can prevent a network from learning at all, regardless of how good the architecture, optimizer, or data are.

2. The Zero-Initialization Trap

A tempting simplification: initialize every weight to zero. This fails completely, for a subtle but important reason.

Why All-Zero Weights Break Training
─────────────────────────────────────────
  If EVERY weight in a layer starts at zero, every neuron in
  that layer computes the EXACT SAME output (zero, or the
  same value after the bias), given the same input.

  During backpropagation (Module 2, Ch.4), every neuron in
  that layer ALSO receives the EXACT SAME gradient — so every
  neuron updates IDENTICALLY, forever. The network is
  mathematically stuck with every neuron in a layer computing
  the IDENTICAL function — called the "SYMMETRY PROBLEM" —
  no matter how long training continues.
─────────────────────────────────────────

import numpy as np
 
# Demonstrating the symmetry problem
W = np.zeros((3, 4))      # all zeros
x = np.array([0.5, 0.3, 0.9, 0.1])
 
z = W @ x
print(z)      # [0. 0. 0.] — every neuron computes the SAME output
# Every neuron will receive the SAME gradient during backprop,
# and update IDENTICALLY forever — they never differentiate

3. Random Initialization and Its Own Problems

Randomizing weights (as Module 2 did) breaks the symmetry problem — but the scale of that randomness matters enormously.

Two Failure Modes From Poor SCALE Choices
─────────────────────────────────────────
  Weights too LARGE:      activations can grow exponentially
                            larger as they pass through each
                            layer — for sigmoid/tanh (Module 2,
                            Ch.2), this pushes activations into
                            the FLAT, vanishing-gradient region
                            immediately
  Weights too SMALL:         activations can shrink toward zero
                                as they pass through each layer
                                — by the final layer of a DEEP
                                network, the signal may have
                                become negligibly small
─────────────────────────────────────────

import numpy as np
 
def simulate_layer_scales(scale, num_layers=10, layer_size=100):
    """Demonstrates how activation scale compounds across layers"""
    activations = np.random.randn(layer_size)
    for layer in range(num_layers):
        W = np.random.randn(layer_size, layer_size) * scale
        activations = np.tanh(W @ activations)
        print(f"Layer {layer+1}: mean abs activation = {np.mean(np.abs(activations)):.6f}")
 
print("Scale too LARGE (1.0):")
simulate_layer_scales(scale=1.0)
print("\nScale too SMALL (0.01):")
simulate_layer_scales(scale=0.01)

Running this reveals activations either saturating (staying near ±1, where tanh's gradient vanishes) or collapsing toward zero — both are symptoms of Module 3, Chapter 6's vanishing/exploding gradient problem, traced back to their root cause here.

4. Xavier/Glorot Initialization

Xavier initialization (also called Glorot initialization, after its author) sets the initial weight scale based on the number of inputs and outputs of a layer, specifically to keep the variance of activations roughly constant across layers.

import numpy as np
 
def xavier_init(fan_in, fan_out):
    """
    fan_in: number of INPUT connections to this layer
    fan_out: number of OUTPUT connections from this layer
    """
    limit = np.sqrt(6 / (fan_in + fan_out))
    return np.random.uniform(-limit, limit, size=(fan_out, fan_in))
 
W = xavier_init(fan_in=100, fan_out=50)
print(f"Weight scale (std): {W.std():.4f}")

Xavier initialization was specifically derived assuming a sigmoid or tanh activation function — it is the correct default when using those activations, but turns out to be a poor fit for ReLU (Section 5).

5. He Initialization

He initialization (named after Kaiming He, its author, and the default in most modern architectures) accounts for a specific property of ReLU: it zeroes out roughly half its inputs (any negative value), so it needs a larger initial weight variance to compensate and keep the signal's overall scale stable.

import numpy as np
 
def he_init(fan_in, fan_out):
    """The standard initialization for ReLU-based networks (the vast majority of this curriculum)"""
    std = np.sqrt(2 / fan_in)
    return np.random.randn(fan_out, fan_in) * std
 
W = he_init(fan_in=100, fan_out=50)
print(f"Weight scale (std): {W.std():.4f}")

Choosing Between Xavier and He
─────────────────────────────────────────
  Xavier/Glorot:    use with sigmoid or tanh activations
  He:                  use with ReLU or its variants (Module 2,
                          Ch.2, Section 5) — the default choice
                          for the vast majority of architectures
                          in Modules 4-6
─────────────────────────────────────────

6. Initialization in PyTorch

PyTorch's layers apply a sensible default initialization automatically — but understanding the underlying scheme matters for diagnosing training problems and for the rare cases requiring a manual override.

import torch
import torch.nn as nn
 
layer = nn.Linear(100, 50)      # PyTorch applies a Kaiming-uniform-based default automatically
 
# Manually applying He/Kaiming initialization, if a specific need arises
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
nn.init.zeros_(layer.bias)      # biases are conventionally initialized to zero — they don't
                                    # suffer from the symmetry problem, since each neuron's
                                    # WEIGHTS already differ
 
print(f"Weight scale (std): {layer.weight.std().item():.4f}")

7. Summary & Next Steps

Key Takeaways

Initializing all weights to zero causes every neuron in a layer to compute identically and update identically forever — the "symmetry problem."
Random initialization breaks symmetry, but the scale of the randomness must be chosen carefully to avoid activations exploding or vanishing across layers.
Xavier/Glorot initialization is derived for sigmoid/tanh activations; He initialization accounts for ReLU's zeroing of negative inputs and is the modern default.
PyTorch layers apply sensible defaults automatically, but understanding the underlying scheme (Xavier vs He) is essential for diagnosing training instability.

Concept Check

Why does initializing all weights to zero prevent a network from learning, even with correct backpropagation?
Why does He initialization use a different formula than Xavier initialization?
What symptom would you observe in activation values across layers if weights were initialized with too large a scale?

Next Chapter

→ Chapter 2: Optimizers — SGD, Momentum, RMSProp, Adam

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index