Generative Deep Learning
GANs
VAEs (Chapter 2) generate data by explicitly modeling a probability distribution and sampling from it. Generative Adversarial Networks (GANs), introduced in 201
JrCodex·7 min read
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 2: Variational Autoencoders Time to complete: ~25 minutes
Table of Contents
- A Completely Different Approach
- The Generator and the Discriminator
- The Adversarial Training Process
- The GAN Loss Function
- Why GANs Are Notoriously Hard to Train
- A GAN in PyTorch
- GANs vs VAEs
- Summary & Next Steps
1. A Completely Different Approach
VAEs (Chapter 2) generate data by explicitly modeling a probability distribution and sampling from it. Generative Adversarial Networks (GANs), introduced in 2014, take an entirely different, remarkably clever approach: pit two networks against each other in a competitive game, where each network's improvement forces the other to improve as well.
2. The Generator and the Discriminator
The Two Competing Networks
─────────────────────────────────────────
Generator (G): takes RANDOM noise as input, and tries
to produce output that looks like it
came from the REAL training data
(structurally similar to a VAE's
decoder, Chapter 2, Section 5)
Discriminator (D): a standard BINARY CLASSIFIER (Module
2, Ch.2-3) that tries to distinguish
REAL data (from the training set)
from FAKE data (produced by the
Generator)
─────────────────────────────────────────
The Adversarial Relationship
─────────────────────────────────────────
Random noise ──► [GENERATOR] ──► fake sample ──┐
├──► [DISCRIMINATOR] ──► real or fake?
Real training data ─────────────────────────────┘
The GENERATOR's goal: FOOL the discriminator — produce
fakes so convincing that D
classifies them as REAL
The DISCRIMINATOR's goal: get BETTER at telling real
from fake, despite G's
improving attempts to fool it
─────────────────────────────────────────
3. The Adversarial Training Process
One Training Iteration, Step by Step
─────────────────────────────────────────
1. Generate a batch of FAKE samples using the Generator's
CURRENT weights
2. Train the DISCRIMINATOR on a MIX of real and fake samples
— a standard binary classification training step
(Module 2, Ch.3), with the discriminator's own weights
updated
3. Train the GENERATOR: generate NEW fake samples, pass them
through the (now slightly better) discriminator, and
update the GENERATOR's weights to make the
discriminator MORE likely to call them "real" —
crucially, the DISCRIMINATOR's weights are FROZEN
during this step
4. REPEAT — both networks continuously improve in response
to each other, ideally converging toward the Generator
producing highly realistic samples
─────────────────────────────────────────
An Analogy: Counterfeiter vs Detective
─────────────────────────────────────────
The Generator is like a COUNTERFEITER, trying to produce
fake currency good enough to pass inspection. The
Discriminator is like a DETECTIVE, trying to catch
counterfeits. As the detective gets better at spotting fakes,
the counterfeiter is FORCED to improve their technique — and
as the counterfeiter improves, the detective must sharpen
their own skills further. Over time, BOTH improve, driven
entirely by this competitive pressure.
─────────────────────────────────────────
4. The GAN Loss Function
import torch
import torch.nn as nn
criterion = nn.BCELoss() # Module 2, Ch.3 — standard binary cross-entropy
def train_discriminator(discriminator, generator, real_samples, optimizer_D, latent_dim):
batch_size = real_samples.size(0)
# Real samples should be classified as 1 (real)
real_labels = torch.ones(batch_size, 1)
real_predictions = discriminator(real_samples)
real_loss = criterion(real_predictions, real_labels)
# Fake samples (from the CURRENT generator) should be classified as 0 (fake)
noise = torch.randn(batch_size, latent_dim)
fake_samples = generator(noise).detach() # detach — do NOT update generator during discriminator's step
fake_labels = torch.zeros(batch_size, 1)
fake_predictions = discriminator(fake_samples)
fake_loss = criterion(fake_predictions, fake_labels)
d_loss = real_loss + fake_loss
optimizer_D.zero_grad()
d_loss.backward()
optimizer_D.step()
return d_loss.item()
def train_generator(discriminator, generator, batch_size, optimizer_G, latent_dim):
noise = torch.randn(batch_size, latent_dim)
fake_samples = generator(noise)
# The generator WANTS the discriminator to output 1 (real) for its fakes
predictions = discriminator(fake_samples)
target_labels = torch.ones(batch_size, 1) # the generator's OWN target — "fool it into saying real"
g_loss = criterion(predictions, target_labels)
optimizer_G.zero_grad()
g_loss.backward() # gradients flow back through the DISCRIMINATOR, but only update the GENERATOR
optimizer_G.step()
return g_loss.item()5. Why GANs Are Notoriously Hard to Train
Common GAN Training Pathologies
─────────────────────────────────────────
Mode collapse: the Generator discovers a SINGLE (or
very few) output(s) that reliably fool
the Discriminator, and stops producing
DIVERSE outputs entirely — technically
"winning" locally, but failing the
actual goal of realistic, VARIED
generation
Training instability: unlike Module 3's training techniques
(which stabilize a SINGLE network's
optimization), a GAN involves TWO
networks improving in response to
EACH OTHER — this adversarial
dynamic can oscillate, diverge, or
get stuck, rather than smoothly
converging the way a single loss
curve (Module 2, Ch.3) typically does
Vanishing gradients if the Discriminator becomes too
for the Generator: GOOD too quickly, it may classify
ALL fakes with near-certainty
(close to 0), providing the
Generator with almost NO useful
gradient signal to improve from
— directly echoing Module 3,
Chapter 6's vanishing gradient
concept, in a NEW context
─────────────────────────────────────────
These challenges are precisely why GANs, despite producing some of the most visually impressive generative results historically, have required substantial specialized technique development (careful architecture choices, alternative loss formulations like Wasserstein GAN) to train reliably — and why diffusion models (Chapter 4) have become preferred for many applications in recent years, offering more stable training.
6. A GAN in PyTorch
import torch
import torch.nn as nn
class Generator(nn.Module):
def __init__(self, latent_dim=100, output_dim=784):
super().__init__()
self.model = nn.Sequential(
nn.Linear(latent_dim, 256), nn.ReLU(),
nn.Linear(256, 512), nn.ReLU(),
nn.Linear(512, output_dim), nn.Tanh(), # tanh — output in [-1, 1]
)
def forward(self, z):
return self.model(z)
class Discriminator(nn.Module):
def __init__(self, input_dim=784):
super().__init__()
self.model = nn.Sequential(
nn.Linear(input_dim, 512), nn.LeakyReLU(0.2), # LeakyReLU (Module 2, Ch.2) — common in GANs
nn.Linear(512, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 1), nn.Sigmoid(),
)
def forward(self, x):
return self.model(x)
generator = Generator()
discriminator = Discriminator()
optimizer_G = torch.optim.Adam(generator.parameters(), lr=0.0002, betas=(0.5, 0.999)) # Module 3, Ch.2
optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=0.0002, betas=(0.5, 0.999))
# A training loop combining Section 4's two functions, alternating steps
for epoch in range(num_epochs):
for real_images, _ in train_loader:
real_images_flat = real_images.view(real_images.size(0), -1)
d_loss = train_discriminator(discriminator, generator, real_images_flat, optimizer_D, latent_dim=100)
g_loss = train_generator(discriminator, generator, real_images_flat.size(0), optimizer_G, latent_dim=100)
print(f"Epoch {epoch}: D loss = {d_loss:.4f}, G loss = {g_loss:.4f}")7. GANs vs VAEs
Choosing Between Them
─────────────────────────────────────────
VAEs (Ch.2): more STABLE, predictable training (a single,
well-behaved loss to monitor); produces
somewhat BLURRIER outputs; provides a
genuinely USEFUL, structured latent space
for other downstream tasks
GANs: historically produced SHARPER, more
realistic-looking outputs (especially
for images); NOTORIOUSLY harder and
less stable to train; the latent space
is less explicitly structured than a
VAE's
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- GANs train two competing networks — a Generator producing fake samples, and a Discriminator distinguishing real from fake — in an adversarial process where each network's improvement forces the other to improve.
- Training alternates between updating the discriminator (on a mix of real and fake data) and updating the generator (to better fool the discriminator), with careful gradient isolation between the two steps.
- GANs are notoriously difficult to train reliably, prone to mode collapse (limited output diversity) and training instability from the adversarial dynamic.
- VAEs offer more stable training and a genuinely structured latent space; GANs have historically produced sharper outputs at the cost of training difficulty.
Concept Check
- What is each network's objective in a GAN, and why does this create a "competitive" training dynamic?
- What is mode collapse, and why does it represent the generator "winning" locally while failing its actual goal?
- What's the key tradeoff between choosing a GAN versus a VAE for a generative task?
Next Chapter
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index