Generative Deep Learning
Diffusion Models
Diffusion models are built on a deceptively simple idea: take a real image, gradually destroy it by adding random noise over many small steps until it becomes p
Jr Codex Deep Learning Notes
Level: Advanced Prerequisites: Chapter 3: GANs Time to complete: ~20 minutes
Table of Contents
- The Current State of the Art
- The Core Idea: Learning to Reverse Noise
- The Forward Process: Adding Noise
- The Reverse Process: Learning to Denoise
- Why Diffusion Trains More Stably Than GANs
- A Simplified Diffusion Training Step
- Text-to-Image Generation — a Brief Orientation
- Summary & Next Steps
1. The Current State of the Art
Diffusion models power most of today's leading image-generation systems (Stable Diffusion, DALL-E, Midjourney, and others), having largely overtaken GANs (Chapter 3) for high-quality image generation, due primarily to more stable training and higher output diversity.
2. The Core Idea: Learning to Reverse Noise
Diffusion models are built on a deceptively simple idea: take a real image, gradually destroy it by adding random noise over many small steps until it becomes pure random noise — then train a network to reverse this process, one small denoising step at a time.
The Two Processes
─────────────────────────────────────────
FORWARD process (Section 3): real image → progressively
NOISIER → ... → pure noise
(a FIXED, non-learned
process — just math)
REVERSE process (Section 4): pure noise → progressively
LESS noisy → ... → a
NEW, realistic image
(THIS is what the
network LEARNS to do)
─────────────────────────────────────────
3. The Forward Process: Adding Noise
import numpy as np
def forward_diffusion_step(x, noise_level):
"""Add a SMALL amount of Gaussian noise — one step of the forward process"""
noise = np.random.randn(*x.shape)
noisy_x = np.sqrt(1 - noise_level) * x + np.sqrt(noise_level) * noise
return noisy_x
def forward_diffusion_full(x_original, num_steps=1000):
"""Applying MANY small noise steps — by the end, barely any of the original signal remains"""
x = x_original.copy()
noise_schedule = np.linspace(0.0001, 0.02, num_steps) # noise level INCREASES gradually
trajectory = [x]
for noise_level in noise_schedule:
x = forward_diffusion_step(x, noise_level)
trajectory.append(x)
return trajectory # x_original -> ... -> nearly pure noiseWhy Make the Forward Process FIXED (Not Learned)
─────────────────────────────────────────
Since ADDING noise is simple, well-understood math (not
something that needs to be learned), the forward process
can be computed DIRECTLY and EXACTLY at any step, without
running through all previous steps sequentially — an
important efficiency detail for training (Section 6).
─────────────────────────────────────────
4. The Reverse Process: Learning to Denoise
The network's actual job: given a noisy image at some step t, predict the noise that was added, so it can be subtracted to move one step back toward the clean original.
The Network's Task, Precisely
─────────────────────────────────────────
Input: a NOISY image at step t, AND the step number t
itself (so the network knows roughly HOW
noisy to expect the input to be)
Output: a PREDICTION of the noise that was added at
this step
Once the noise is predicted, it can be SUBTRACTED (with
appropriate scaling) to produce a SLIGHTLY less noisy image
— repeating this process, starting from PURE random noise,
gradually produces a genuinely NEW, realistic image.
─────────────────────────────────────────
The network used for this noise-prediction task is typically a U-Net (Module 4, Chapter 4's segmentation architecture) — its encoder-decoder structure with skip connections is well suited to processing an image at full resolution while still capturing broader spatial context.
5. Why Diffusion Trains More Stably Than GANs
A Direct Comparison to Chapter 3's Challenges
─────────────────────────────────────────
GANs (Ch.3): TWO networks, competing adversarially —
an inherently unstable dynamic (Ch.3,
Section 5), prone to mode collapse
Diffusion: ONE network, trained with a STANDARD,
well-behaved loss (predicting added
noise is just a REGRESSION problem
— Module 2, Ch.3's MSE loss) — no
adversarial dynamic, no competing
objectives, MUCH closer to the
stable, predictable training already
covered throughout Modules 2-4
─────────────────────────────────────────
This single architectural difference — a standard, single-network training objective instead of an adversarial game — is the primary reason diffusion models have proven easier to train reliably at very large scale than GANs.
6. A Simplified Diffusion Training Step
import torch
import torch.nn as nn
def diffusion_training_step(model, images, num_timesteps=1000):
batch_size = images.size(0)
# Sample a RANDOM timestep for EACH image in the batch
t = torch.randint(0, num_timesteps, (batch_size,))
# Add the CORRESPONDING amount of noise (Section 3) — computed DIRECTLY, no need to iterate
noise = torch.randn_like(images)
noise_level = t.float() / num_timesteps
noisy_images = torch.sqrt(1 - noise_level).view(-1, 1, 1, 1) * images + \
torch.sqrt(noise_level).view(-1, 1, 1, 1) * noise
# The network (typically a U-Net) predicts the NOISE that was added
predicted_noise = model(noisy_images, t)
# A SIMPLE regression loss — predict the noise correctly (Module 2, Ch.3)
loss = nn.functional.mse_loss(predicted_noise, noise)
return loss
# Training loop
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
for images, _ in train_loader:
loss = diffusion_training_step(model, images)
optimizer.zero_grad()
loss.backward()
optimizer.step()7. Text-to-Image Generation — a Brief Orientation
Modern systems like Stable Diffusion extend this chapter's core idea by conditioning the denoising process on a text description — the U-Net receives not just the noisy image and timestep, but also a text embedding (produced by a Transformer text encoder, Module 6), guiding the denoising process toward an image matching the text prompt.
Text-to-Image, Conceptually
─────────────────────────────────────────
Text prompt ("a cat wearing a hat")
│
▼
Text encoder (a Transformer, Module 6) ──► text embedding
│
Random noise ──► [U-Net, conditioned on text embedding] ──►
(repeated over many denoising steps) ──► final image
─────────────────────────────────────────
This combination — a Transformer for understanding text (Module 6) plus a diffusion model for generating images (this chapter) — exemplifies how the architectures covered across this entire curriculum combine in real, state-of-the-art systems, rather than existing in isolation.
8. Summary & Next Steps
Key Takeaways
- Diffusion models learn to reverse a fixed, gradual noise-adding process, training a network to predict and remove noise one small step at a time.
- The forward (noise-adding) process is fixed, simple math; the reverse (denoising) process is what the network learns, typically using a U-Net architecture.
- Diffusion models train with a standard, single-network regression loss, avoiding the adversarial instability that makes GANs difficult to train.
- Text-to-image systems condition the denoising process on text embeddings from a Transformer encoder, combining this module's generative techniques with Module 6's architecture.
Module 8 Complete — What's Next
You've now covered the full landscape of generative deep learning: autoencoders for compression and reconstruction, VAEs for structured, sampleable generation, GANs for adversarial generation, and diffusion models powering today's leading image generation systems. Module 9 shifts to practical engineering: building, training, debugging, and deploying real systems using everything covered in this curriculum.
Concept Check
- Why is the forward (noise-adding) process in a diffusion model not something that needs to be learned?
- What specific task does the denoising network actually learn to perform at each step?
- Why do diffusion models tend to train more stably than GANs?
Next Module
→ Module 9: DL Engineering & Capstone
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index