Generative AI

The Generative AI Landscape

The Family of Generative Models

Every generative model solves the same problem — sample from P(data) — and they differ only in how they make that tractable. Modelling the joint distribution of

JrCodex·6 min read

Jr Codex Generative AI Notes

Level: Beginner Prerequisites: Chapter 1: What Generative AI Is, and What It Isn't Time to complete: ~20 minutes

Table of Contents

  1. Four Families, One Goal
  2. Autoregressive Models
  3. Variational Autoencoders
  4. Generative Adversarial Networks
  5. Diffusion Models
  6. The Decision Map
  7. Summary & Next Steps

1. Four Families, One Goal

Every generative model solves the same problem — sample from P(data) — and they differ only in how they make that tractable. Modelling the joint distribution of a million-pixel image directly is impossible, so each family picks a different simplification.

The Four Strategies
─────────────────────────────────────────
  AUTOREGRESSIVE  Break the joint distribution into a
                  chain of conditionals. Generate one
                  piece at a time.

  VAE             Learn a compressed latent space with a
                  known shape, then sample from that.

  GAN             Skip probability entirely. Train a
                  generator to fool a critic.

  DIFFUSION       Learn to reverse a gradual corruption
                  process, one denoising step at a time.
─────────────────────────────────────────

This chapter is a map, not a derivation. The mechanics of each family are worked through in Deep Learning Module 8. The purpose here is to know which one you are reaching for, and why.

2. Autoregressive Models

The dominant family for text and code, and the one behind every LLM you have used.

The Idea
─────────────────────────────────────────
  P(sentence) = P(w1) x P(w2|w1) x P(w3|w1,w2) x ...

  Each token is predicted from everything before it.
  Generation = repeat this prediction, feeding each
  output back in as input.
─────────────────────────────────────────
Best atText, code, anything with natural sequential order
StrengthExact likelihood, stable training, strong coherence over long spans
WeaknessGeneration is inherently serial — token n needs token n-1, so it cannot be parallelised at inference
Covered inNLP & LLM Notes, Module 5, Chapter 1

That serial weakness is exactly why a 500-word LLM response streams in over several seconds while a diffusion image of far greater raw size arrives in one batch of parallel steps.

3. Variational Autoencoders

A VAE compresses data into a latent space that is structured — deliberately shaped like a simple, samplable distribution.

The Idea
─────────────────────────────────────────
  ENCODER:  data ──► a distribution in latent space
  DECODER:  a point in latent space ──► data

  The training loss forces the latent space to stay
  smooth and centred, so ANY point you sample from it
  decodes to something plausible.
─────────────────────────────────────────
Best atCompression, anomaly detection, representation learning
StrengthA clean, continuous, interpretable latent space
WeaknessBlurry output — the reconstruction loss averages over possibilities
Covered inDL Notes, Module 8, Chapter 2

VAEs are rarely used alone for generation today, but they have not gone away: Stable Diffusion contains a VAE, using it to shrink images before the expensive part of generation. Module 3, Chapter 1 picks this up directly.

4. Generative Adversarial Networks

Two networks in competition: a generator producing fakes, and a discriminator trying to spot them.

The Idea
─────────────────────────────────────────
  noise ──► GENERATOR ──► fake sample
                              │
       real sample ───────────┼──► DISCRIMINATOR ──► real/fake?
                              │
  Generator improves by fooling the discriminator.
  Discriminator improves by catching the generator.
  Neither ever "wins" — the equilibrium is the goal.
─────────────────────────────────────────
Best atSharp images, fast single-pass generation, style/domain transfer
StrengthOne forward pass per sample — far faster than diffusion at inference
WeaknessNotoriously unstable training; mode collapse; no likelihood estimate
Covered inDL Notes, Module 8, Chapter 3

GANs dominated image generation from roughly 2016 to 2021 and were then largely displaced by diffusion. They remain competitive where inference speed matters more than peak quality — real-time face swapping, super-resolution, upscaling.

5. Diffusion Models

The current state of the art for images, and increasingly for audio and video.

The Idea
─────────────────────────────────────────
  FORWARD (fixed, not learned):
     image ──► + noise ──► + noise ──► ... ──► pure noise

  REVERSE (learned):
     pure noise ──► denoise ──► denoise ──► ... ──► image

  The network's only job: given a noisy image and a
  timestep, predict the noise that was added.
─────────────────────────────────────────
Best atImages, audio, video; anything needing high fidelity and diversity
StrengthStable training (a plain regression loss), excellent quality and coverage
WeaknessSlow — sampling needs many sequential network passes
Covered inDL Notes, Module 8, Chapter 4

The trade against GANs is direct: diffusion buys quality and training stability with inference cost. Much of the engineering in modern image tools — fewer sampling steps, better schedulers, distillation — is about clawing that cost back.

6. The Decision Map

In practice you rarely choose a family from first principles; the modality chooses for you.

Which Family, by Modality
─────────────────────────────────────────
  Text, code               ──► Autoregressive
  Images                   ──► Diffusion (GAN if speed is critical)
  Audio, speech            ──► Diffusion or autoregressive
  Video                    ──► Diffusion
  Compression, anomalies   ──► VAE
─────────────────────────────────────────

The More Useful Trade-off Table
─────────────────────────────────────────
                  Quality  Speed   Training  Latent
                                   stability space
  Autoregressive    high    slow      high     no
  VAE               low     fast      high     yes
  GAN               high    fast      LOW      yes
  Diffusion         high    slow      high     yes
─────────────────────────────────────────

Two things worth noticing. First, no family wins every column — which is why all four are still in production use somewhere. Second, real systems combine them: Stable Diffusion is a VAE plus a diffusion U-Net plus a Transformer text encoder, three families in one pipeline.

7. Summary & Next Steps

Key Takeaways

  • All four families sample from P(data); they differ in the simplification that makes it tractable — chaining conditionals, structuring a latent space, adversarial training, or reversing noise.
  • Autoregressive models own text because language has a natural order; their serial generation is the reason LLM output streams.
  • Diffusion displaced GANs for images by trading inference speed for training stability and quality; GANs survive where speed dominates.
  • VAEs are rarely the whole system now but remain a component — notably inside Stable Diffusion.

Concept Check

  1. Why is autoregressive generation inherently serial, and what practical consequence does that have for users?
  2. GANs are faster at inference than diffusion models. Why did diffusion still displace them for image generation?
  3. A VAE alone produces blurry images. Why is a VAE still useful inside a modern image-generation pipeline?

Next Chapter

→ Chapter 3: Latent Space — The Unifying Idea

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index