From Diffusion to Stable Diffusion

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: DL Notes, Module 8, Chapter 4: Diffusion Models Time to complete: ~20 minutes

Table of Contents

1. The Problem with Pixel-Space Diffusion
2. The Latent Diffusion Trick
3. The Three Components
4. The Full Generation Pipeline
5. Running It
6. What Each Component Costs You
7. Summary & Next Steps

1. The Problem with Pixel-Space Diffusion

The DL Notes' diffusion chapter described denoising an image directly. That works, and early models did exactly it. It is also brutally expensive.

Why Denoising Pixels Does Not Scale
─────────────────────────────────────────
  A 512x512 RGB image = 786,432 values.

  Generation requires ~50 sequential passes of a U-Net
  over ALL of them.

  Result: minutes per image on expensive hardware, and
  cost that grows with the SQUARE of resolution.
─────────────────────────────────────────

The insight that fixed this is the reason a modern image model runs on a consumer GPU in seconds.

The Observation
─────────────────────────────────────────
  Most of those 786,432 numbers carry no SEMANTIC
  information. They encode fine texture and
  high-frequency noise that a viewer never consciously
  registers.

  Diffusion — the expensive part — only needs to work
  on the part that carries MEANING.
─────────────────────────────────────────

2. The Latent Diffusion Trick

Latent diffusion runs the entire diffusion process in a compressed latent space (Module 1, Chapter 3) instead of on pixels.

The Compression
─────────────────────────────────────────
  PIXEL SPACE     512 x 512 x 3   = 786,432 values
        │  VAE encoder, 8x downsample per side
        ▼
  LATENT SPACE     64 x  64 x 4   =  16,384 values

  A 48x reduction. The diffusion U-Net now operates on
  48x less data, for roughly 48x less compute per step.
─────────────────────────────────────────

Why the Quality Survives
─────────────────────────────────────────
  The VAE is trained SEPARATELY, once, on a very large
  image corpus, purely to compress and reconstruct.

  It learns to keep structure, colour and object
  identity, and to discard exactly the texture detail
  that it can plausibly REGENERATE on the way back out.

  So the diffusion model never has to spend capacity on
  fine texture — the decoder supplies it.
─────────────────────────────────────────

This is the single architectural idea that separates "a research demo" from "a tool people use." It is also a direct answer to Module 1, Chapter 2's note that VAEs alone produce blurry images: a VAE used purely as a codec, never sampled from directly, has no blurriness problem.

3. The Three Components

Stable Diffusion is three networks, each from a different lineage in your prior study.

Component 1 — TEXT ENCODER (a Transformer)
─────────────────────────────────────────
  Input:   the prompt, tokenised
  Output:  a sequence of text embeddings
  Origin:  CLIP's text tower — Chapter 2
  Frozen:  yes, during diffusion training

Component 2 — U-NET (the denoiser)
─────────────────────────────────────────
  Input:   a noisy LATENT, a timestep, text embeddings
  Output:  a prediction of the noise present
  Origin:  DL Notes, Module 8, Chapter 4
  This is the ONLY part that runs 20-50 times.
  It is also the part that LoRAs modify (Chapter 5).

Component 3 — VAE (the codec)
─────────────────────────────────────────
  Encoder: image ──► latent  (used for img2img, Ch.4)
  Decoder: latent ──► image  (used at the very end)
  Origin:  DL Notes, Module 8, Chapters 1-2
  Runs ONCE per image, not once per step.
─────────────────────────────────────────

Three Curricula, One Model
─────────────────────────────────────────
  Transformer  ──►  DL Notes, Module 6
  U-Net        ──►  DL Notes, Modules 4 and 8
  VAE          ──►  DL Notes, Module 8

  Nothing here is new architecture. What is new is the
  COMPOSITION.
─────────────────────────────────────────

4. The Full Generation Pipeline

Text to Image, End to End
─────────────────────────────────────────
  "a red fox in snow, golden hour"
        │
        ▼
  [1] TEXT ENCODER ──► text embeddings ────────┐
                                               │
  [2] random noise in LATENT space (64x64x4)   │
        │                                      │
        ▼                                      │
  ┌── [3] U-NET denoising loop ────────────────┼──┐
  │      predict noise, conditioned on ────────┘  │
  │      subtract a scheduled portion of it       │
  │      repeat 20-50 times                       │
  └───────────────────┬───────────────────────────┘
                      ▼
              a clean LATENT (64x64x4)
                      │
        ▼ [4] VAE DECODER
              the final image (512x512x3)
─────────────────────────────────────────

Two things to internalise from this diagram, because they explain most practical behaviour:

• Step 3 is the loop. Everything about speed, and every quality/cost trade in Chapter 3, concerns how many times this runs.
• The text embedding enters at every iteration, not once at the start. The prompt steers continuously — which is why guidance strength is a dial, not a switch (Chapter 2).

5. Running It

import torch
from diffusers import StableDiffusionXLPipeline
 
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,        # half precision: ~2x less VRAM, no visible quality loss
    variant="fp16",
).to("cuda")
 
image = pipe(
    prompt="a red fox in deep snow, golden hour, shallow depth of field",
    num_inference_steps=30,           # how many times the U-Net loop runs (Chapter 3)
    guidance_scale=7.0,               # how hard the prompt steers (Chapter 2)
    generator=torch.Generator("cuda").manual_seed(42),      # reproducible NOISE
).images[0]
 
image.save("fox.png")

Seeds Behave Differently Here
─────────────────────────────────────────
  In a TEXT model, temperature=0 is still not
  reproducible (Module 2, Chapter 2).

  In an IMAGE model, the seed determines the STARTING
  NOISE, and the denoising path from a fixed start is
  deterministic on fixed hardware.

  Same seed + same prompt + same settings + same GPU
  ──► the same image, reliably.

  This makes seeds a genuine WORKING TOOL: fix the seed,
  vary one setting, and see that setting's effect in
  isolation.
─────────────────────────────────────────

6. What Each Component Costs You

Useful when a generation is slow, blurry, or out of memory — it tells you which part to look at.

The Two Fixes You Will Reach For Most
─────────────────────────────────────────
  Slow generation      ──► fewer steps, or a scheduler
                           that needs fewer (Chapter 3)

  OOM at the last step ──► pipe.enable_vae_tiling()
                           decodes the latent in tiles
                           instead of all at once
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

• Latent diffusion runs the denoising loop in a ~48x compressed space rather than on pixels, which is what made image generation fast enough to be practical.
• A separately trained VAE acts as a codec: it discards texture the decoder can regenerate, so the U-Net never spends capacity on it.
• Stable Diffusion composes three familiar architectures — a Transformer text encoder, a U-Net denoiser, and a VAE — with only the U-Net running repeatedly.
• Text conditioning is applied at every denoising step, not once, which is why prompt influence is a continuously tunable strength.

Concept Check

1. A VAE alone produces blurry images, yet Stable Diffusion contains one and is not blurry. Resolve the apparent contradiction.
2. Which component runs 30 times for a 30-step generation, and which run exactly once? What does that imply about where to optimise?
3. Why are image seeds a practical experimental tool while text-model seeds are not?

Next Chapter

→ Chapter 2: CLIP and Text Conditioning

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index