Convolutional Neural Networks
CNN Architectures
A CNN architecture is a specific arrangement of convolutional layers (Chapter 1), pooling layers, and a handful of dense layers at the end for final classificat
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 1: Convolution & Pooling Operations Time to complete: ~25 minutes
Table of Contents
- Assembling Layers Into a Full Network
- LeNet — the Original CNN
- AlexNet — the 2012 Breakthrough
- VGGNet — Depth Through Simplicity
- The Degradation Problem
- ResNet — Residual Connections
- Modern Efficient Architectures
- Summary & Next Steps
1. Assembling Layers Into a Full Network
A CNN architecture is a specific arrangement of convolutional layers (Chapter 1), pooling layers, and a handful of dense layers at the end for final classification.
The General CNN Pattern
─────────────────────────────────────────
[Conv → Activation → Pool] × several times
(feature extraction — spatial size SHRINKS,
channel depth GROWS, as layers get deeper)
│
▼
Flatten
│
▼
[Dense → Activation] × 1-2 (classification head)
│
▼
Dense (output layer, softmax — Module 2, Ch.2)
─────────────────────────────────────────
This chapter walks through the specific architectures that defined the field's progress, each solving a specific limitation of the one before it.
2. LeNet — the Original CNN
LeNet-5 (1998, Yann LeCun) was the first successful CNN, designed for recognizing handwritten digits — establishing the conv→pool→conv→pool→dense pattern still used today, though at a tiny scale by modern standards.
import torch.nn as nn
class LeNet5(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.features = nn.Sequential(
nn.Conv2d(1, 6, kernel_size=5), nn.Tanh(), nn.AvgPool2d(2),
nn.Conv2d(6, 16, kernel_size=5), nn.Tanh(), nn.AvgPool2d(2),
)
self.classifier = nn.Sequential(
nn.Linear(16 * 4 * 4, 120), nn.Tanh(),
nn.Linear(120, 84), nn.Tanh(),
nn.Linear(84, num_classes),
)
def forward(self, x):
x = self.features(x)
x = x.flatten(1)
return self.classifier(x)LeNet used tanh activations (Module 2 hadn't yet discovered ReLU's advantages) and had only about 60,000 parameters — tiny compared to modern networks, but it proved the core architecture pattern worked.
3. AlexNet — the 2012 Breakthrough
Already introduced in Module 1, Chapter 3, AlexNet scaled LeNet's pattern up dramatically (8 layers, 60 million parameters), and introduced two techniques still standard today:
AlexNet's Key Innovations
─────────────────────────────────────────
ReLU instead of tanh/sigmoid: solved vanishing
gradients (Module 3, Ch.6)
at this scale, and trained
MUCH faster
Dropout (Module 3, Ch.4): regularized the huge
dense layers at the
network's end,
preventing severe
overfitting
Trained on GPUs: made training a
network this large
practical within
days rather than
months
─────────────────────────────────────────
4. VGGNet — Depth Through Simplicity
VGGNet (2014) demonstrated that stacking many small 3×3 convolutional filters (rather than fewer, larger filters like AlexNet used) produced better results, while being conceptually simpler — VGG16 uses only 3×3 convolutions and 2×2 max pooling, repeated 16 layers deep.
Why Small Filters, Stacked, Beat Large Filters
─────────────────────────────────────────
TWO stacked 3x3 convolutions have the SAME effective
"receptive field" (the region of the original image that
influences one output value) as ONE 5x5 convolution — but
with FEWER total parameters, and crucially, TWO non-linear
activations (Module 2, Ch.2) instead of one, giving the
network more representational flexibility for the same
receptive field size.
─────────────────────────────────────────
import torch.nn as nn
def vgg_block(in_channels, out_channels, num_convs):
layers = []
for i in range(num_convs):
layers.append(nn.Conv2d(in_channels if i == 0 else out_channels, out_channels, kernel_size=3, padding=1))
layers.append(nn.ReLU())
layers.append(nn.MaxPool2d(kernel_size=2, stride=2))
return nn.Sequential(*layers)
# VGG16's feature extractor — repeating the SAME simple block pattern
vgg16_features = nn.Sequential(
vgg_block(3, 64, num_convs=2),
vgg_block(64, 128, num_convs=2),
vgg_block(128, 256, num_convs=3),
vgg_block(256, 512, num_convs=3),
vgg_block(512, 512, num_convs=3),
)5. The Degradation Problem
Naively stacking more and more layers eventually stops helping, and even starts hurting — surprisingly, not due to overfitting (Module 3, Chapter 4), but because very deep plain networks become harder to optimize at all, even on the training set itself.
The Counter-Intuitive Finding
─────────────────────────────────────────
A 56-layer plain CNN often achieves WORSE training accuracy
than a comparable 20-layer plain CNN — not because it lacks
capacity (a deeper network can always represent everything
a shallower one can, at minimum, by learning IDENTITY
functions in the extra layers), but because backpropagation
(Module 2, Ch.4) struggles to find that solution through
so many stacked non-linear transformations — connecting
directly to Module 3, Chapter 6's vanishing gradient problem.
─────────────────────────────────────────
6. ResNet — Residual Connections
ResNet (2015) solved the degradation problem with the residual connection idea previewed in Module 3, Chapter 6: instead of forcing each block to learn a full transformation, it learns a residual (a correction) added to the input.
import torch.nn as nn
class ResidualBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.bn1 = nn.BatchNorm2d(channels) # Module 3, Ch.3
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
identity = x # the SHORTCUT — saved before any transformation
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out += identity # THE residual connection — add the ORIGINAL input back
return self.relu(out)Why This Solves the Degradation Problem
─────────────────────────────────────────
If the OPTIMAL transformation for a block is simply "do
nothing" (an identity function), a residual block can learn
this TRIVIALLY by driving conv1/conv2's weights toward zero
— the shortcut passes the input through UNCHANGED. A plain
(non-residual) block would have to learn an EXACT identity
mapping through its non-linear layers, which is much harder
to discover via gradient descent.
This single idea let ResNet scale to 152 layers (and later
variants beyond 1000), FAR deeper than VGG's 16-19 layers,
while still training reliably and improving accuracy.
─────────────────────────────────────────
7. Modern Efficient Architectures
Architectures Worth Knowing (Briefly)
─────────────────────────────────────────
MobileNet: uses "depthwise separable" convolutions —
a cheaper approximation of standard
convolution — designed for mobile/edge
devices with tight compute budgets
EfficientNet: systematically scales network depth,
width, AND input resolution together,
using a principled formula, rather than
scaling any ONE dimension arbitrarily
Vision Transformers apply the Transformer architecture
(ViT): (Module 6) to images directly,
by treating an image as a
sequence of patches — increasingly
competitive with or superior to
CNNs on large datasets
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- CNN architectures follow a general pattern: alternating convolution/activation/pooling layers for feature extraction, followed by dense layers for classification.
- AlexNet (2012) proved deep CNNs work at scale using ReLU and dropout; VGGNet (2014) showed that stacking many small 3x3 filters outperforms fewer large filters.
- Very deep plain networks suffer from a degradation problem — an optimization difficulty, not overfitting — where added depth can hurt even training accuracy.
- ResNet's residual connections solve this by letting each block learn a correction added to its input, allowing gradients to flow through a shortcut path and enabling networks hundreds of layers deep.
Concept Check
- Why do two stacked 3x3 convolutions have an advantage over a single 5x5 convolution with a similar receptive field?
- What specifically is "the degradation problem," and why is it distinct from overfitting?
- How does a residual connection make it easier for a block to learn an identity function?
Next Chapter
→ Chapter 3: Image Classification in Practice
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index