Generative Deep Learning
Autoencoders
Every architecture so far — CNNs (Module 4), RNNs/Transformers (Modules 5-6) — was discriminative: given an input, predict a label or a value. This module cover
JrCodex·6 min read
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Module 7: Transfer Learning & Practical Training Strategies Time to complete: ~20 minutes
Table of Contents
- A Different Kind of Task: Generation
- The Autoencoder Architecture
- The Bottleneck — Forcing Compression
- Training an Autoencoder
- What Autoencoders Are Used For
- Autoencoders in PyTorch
- The Limitation That Motivates VAEs
- Summary & Next Steps
1. A Different Kind of Task: Generation
Every architecture so far — CNNs (Module 4), RNNs/Transformers (Modules 5-6) — was discriminative: given an input, predict a label or a value. This module covers generative models: networks trained to create new data resembling their training distribution, rather than just classify or extract features from existing data.
Discriminative vs Generative, Precisely
─────────────────────────────────────────
Discriminative: learns P(y | x) — "given this input,
what's the most likely label/output?"
(everything through Module 7)
Generative: learns (an approximation of) P(x) —
"what does the DATA ITSELF look
like, well enough to CREATE new,
plausible examples of it?"
─────────────────────────────────────────
2. The Autoencoder Architecture
An autoencoder is trained to reconstruct its own input, after passing it through a deliberately narrow "bottleneck" — forcing the network to learn a compressed representation.
The Autoencoder's Two Halves
─────────────────────────────────────────
Input (e.g. an image)
│
▼
ENCODER (a series of layers, PROGRESSIVELY shrinking —
similar to Module 4's CNN feature extraction)
│
▼
Latent representation (the BOTTLENECK — Section 3)
│
▼
DECODER (a series of layers, PROGRESSIVELY expanding
back to the ORIGINAL size)
│
▼
Reconstructed output (should closely match the ORIGINAL input)
─────────────────────────────────────────
3. The Bottleneck — Forcing Compression
The latent representation — the narrowest point of the network — is dramatically smaller than the input (e.g. compressing a 784-pixel image down to just 32 numbers).
Why the Bottleneck Is the Entire Point
─────────────────────────────────────────
If the latent representation were the SAME size as the
input, the network could trivially learn the IDENTITY
function (just copy the input through unchanged) — learning
NOTHING useful. Forcing the information through a much
SMALLER bottleneck means the encoder MUST learn to keep
only the most essential, informative features — discarding
redundancy and noise — for the decoder to have any hope of
reconstructing the original from so little information.
─────────────────────────────────────────
4. Training an Autoencoder
Notably, an autoencoder requires no labels at all — the "label" for any input is simply the input itself, making this a form of self-supervised learning.
import torch
import torch.nn as nn
class Autoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256), nn.ReLU(),
nn.Linear(256, 64), nn.ReLU(),
nn.Linear(64, latent_dim), # the BOTTLENECK (Section 3)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 64), nn.ReLU(),
nn.Linear(64, 256), nn.ReLU(),
nn.Linear(256, input_dim), nn.Sigmoid(), # sigmoid — pixel values scaled to [0, 1]
)
def forward(self, x):
latent = self.encoder(x)
reconstruction = self.decoder(latent)
return reconstruction, latent
model = Autoencoder()
criterion = nn.MSELoss() # reconstruction loss — how CLOSE is the output to the ORIGINAL input?
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
# A training step — note the ABSENCE of any separate label
for images, _ in train_loader: # the `_` (label) is IGNORED entirely
images_flat = images.view(images.size(0), -1) # flatten to (batch, 784)
reconstructed, latent = model(images_flat)
loss = criterion(reconstructed, images_flat) # comparing OUTPUT to INPUT, not to any external label
optimizer.zero_grad()
loss.backward()
optimizer.step()5. What Autoencoders Are Used For
Practical Applications
─────────────────────────────────────────
Dimensionality reduction: conceptually similar to the ML
Notes' PCA (Module 6), but
capable of learning NON-LINEAR
compressions — often more
powerful than PCA for complex
data like images
Anomaly detection: train on ONLY normal data;
an input that RECONSTRUCTS
poorly (high reconstruction
error) is likely anomalous
— it doesn't match the
patterns the encoder
learned to compress well
Denoising: train the network to
reconstruct a CLEAN
image from a
DELIBERATELY NOISY
input — the network
learns to ignore noise
entirely
Pretraining/ the encoder's learned
representation latent features can
learning: be reused for
OTHER downstream
tasks (Module 7's
transfer learning
concept), especially
valuable when
LABELED data is
scarce
─────────────────────────────────────────
6. Autoencoders in PyTorch
import torch
import torch.nn as nn
class DenoisingAutoencoder(nn.Module):
"""A practical variant — Section 5's denoising application"""
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 128), nn.ReLU(),
nn.Linear(128, latent_dim),
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 128), nn.ReLU(),
nn.Linear(128, input_dim), nn.Sigmoid(),
)
def forward(self, x):
latent = self.encoder(x)
return self.decoder(latent)
model = DenoisingAutoencoder()
def add_noise(images, noise_factor=0.3):
noisy = images + noise_factor * torch.randn_like(images)
return torch.clamp(noisy, 0.0, 1.0) # keep pixel values in a VALID range
# Training: input is NOISY, target is CLEAN — teaches the network to denoise
for images, _ in train_loader:
images_flat = images.view(images.size(0), -1)
noisy_images = add_noise(images_flat)
reconstructed = model(noisy_images)
loss = nn.MSELoss()(reconstructed, images_flat) # compare to the ORIGINAL clean image7. The Limitation That Motivates VAEs
A plain autoencoder's latent space has no guaranteed structure — there's no rule preventing the encoder from mapping similar inputs to wildly different, disconnected regions of the latent space, or from leaving large "gaps" that don't correspond to any realistic input.
The Practical Consequence
─────────────────────────────────────────
If you wanted to GENERATE a brand new image (rather than
just reconstruct an existing one), you'd need to pick a
point in the latent space and run it through the decoder —
but with a plain autoencoder, MOST random points in the
latent space don't correspond to anything realistic, since
nothing forced the latent space to be smooth or well-organized.
This is EXACTLY the limitation that motivates Chapter 2's
Variational Autoencoder — a small but crucial modification
that makes the latent space usable for GENUINE generation.
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Autoencoders are trained to reconstruct their own input through a compressed bottleneck, requiring no labels and learning as a form of self-supervised learning.
- The bottleneck is what forces genuinely useful compression — without it, the network could trivially learn to copy the input unchanged.
- Autoencoders are practically useful for non-linear dimensionality reduction, anomaly detection (via reconstruction error), denoising, and representation pretraining.
- A plain autoencoder's latent space has no guaranteed structure, making it unreliable for generating genuinely new data — the specific limitation that motivates variational autoencoders.
Concept Check
- Why must an autoencoder's latent representation be smaller than its input for training to teach it anything useful?
- How can an autoencoder be used for anomaly detection without ever seeing a labeled anomalous example?
- Why can't a plain autoencoder be reliably used to generate entirely new, realistic data?
Next Chapter
→ Chapter 2: Variational Autoencoders
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index