Generative AI

Image Generation

CLIP and Text Conditioning

Chapter 1 established that text embeddings enter the U-Net at every denoising step. It did not explain the hard part: why does an embedding of the English word

JrCodex·8 min read

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Chapter 1: From Diffusion to Stable Diffusion Time to complete: ~20 minutes

Table of Contents

  1. The Question This Chapter Answers
  2. CLIP — Two Encoders, One Space
  3. Contrastive Training
  4. How Text Enters the U-Net
  5. Classifier-Free Guidance
  6. Why Prompts Fail in Predictable Ways
  7. Summary & Next Steps

1. The Question This Chapter Answers

Chapter 1 established that text embeddings enter the U-Net at every denoising step. It did not explain the hard part: why does an embedding of the English word "fox" bias a numerical denoising process toward fox-shaped pixels?

Nothing about the two is obviously related. The answer is a model called CLIP, and it is the piece that makes text-to-image possible at all.

2. CLIP — Two Encoders, One Space

CLIP (Contrastive Language–Image Pre-training) is two separate encoders trained to place matching text and images at the same location in a shared embedding space.

The Architecture
─────────────────────────────────────────
  "a red fox in snow"
        │
        ▼
  TEXT ENCODER (a Transformer)  ──►  vector  ──┐
                                               │
                                          SAME SPACE
                                               │
  IMAGE ENCODER (a ViT)         ──►  vector  ──┘
        ▲
        │
  [photo of a red fox in snow]

  Training goal: these two vectors should be CLOSE.
─────────────────────────────────────────

import torch
from transformers import CLIPModel, CLIPProcessor
 
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
proc  = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
 
captions = ["a red fox in snow", "a red car in snow", "a bowl of soup"]
inputs = proc(text=captions, images=fox_photo, return_tensors="pt", padding=True)
 
with torch.no_grad():
    out = model(**inputs)
    img = out.image_embeds / out.image_embeds.norm(dim=-1, keepdim=True)   # NORMALISE, so
    txt = out.text_embeds  / out.text_embeds.norm(dim=-1, keepdim=True)    # dot = cosine
    for caption, score in zip(captions, (img @ txt.T)[0]):
        print(f"{score:.3f}  {caption}")
 
# 0.312  a red fox in snow      ← the matching caption sits CLOSEST
# 0.241  a red car in snow      ← shares colour and setting, differs in OBJECT
# 0.118  a bowl of soup

The middle score is the interesting one: "car" and "fox" share every word except the subject, so the gap between 0.312 and 0.241 is CLIP demonstrating that it encodes object identity, not just colour and setting.

This is Module 1, Chapter 3's point about conditioning made concrete: two latent spaces, deliberately fused into one. Once text and images live in the same space, a text vector is a valid description of a region of image space — and that is precisely what a conditioning signal needs to be.

3. Contrastive Training

CLIP never learned to caption or to draw. It learned only to match, on roughly 400 million image–text pairs scraped from the web.

One Training Batch
─────────────────────────────────────────
  Take N image-caption pairs. Encode all of them.
  Compute every image-to-text similarity: an N x N grid.

               txt1   txt2   txt3   txt4
        img1  [ ✓ ]    x      x      x
        img2    x    [ ✓ ]    x      x
        img3    x      x    [ ✓ ]    x
        img4    x      x      x    [ ✓ ]

  PUSH UP the N matching pairs on the diagonal.
  PUSH DOWN all N² - N mismatched pairs.
─────────────────────────────────────────

Why This Simple Objective Is So Powerful
─────────────────────────────────────────
  To separate "a red fox in snow" from "a red car in
  snow", the encoders must genuinely represent OBJECT
  IDENTITY, not just colour and setting.

  Scaled to 400M web pairs, the pressure to distinguish
  captions forces both encoders to learn compositional
  visual semantics — style, medium, lighting, mood,
  count, relation — because captions mention all of it.

  Consequence: CLIP understands "in the style of a
  woodcut" because thousands of captions said so.
─────────────────────────────────────────

And Why Its Biases Are Inherited
─────────────────────────────────────────
  CLIP learned from uncurated web captions. Whatever
  those captions associate — occupations with genders,
  descriptors with ethnicities, "beautiful" with a
  narrow range of appearances — becomes geometry in the
  shared space.

  Every image model built on CLIP inherits that
  geometry. Module 6, Chapter 4 takes this up directly.
─────────────────────────────────────────

4. How Text Enters the U-Net

The mechanism is cross-attention — the same operation from DL Notes, Module 6, with the query and key/value coming from different sources.

Cross-Attention in a Denoising Block
─────────────────────────────────────────
  QUERIES     come from the IMAGE latent
              "what should this spatial region become?"

  KEYS/VALUES come from the TEXT embeddings
              "here are the concepts on offer"

  Each spatial position in the latent attends over the
  prompt's tokens and pulls in whichever ones are
  relevant to it.
─────────────────────────────────────────

What This Buys: Spatial Selectivity
─────────────────────────────────────────
  Prompt: "a red fox on a blue rug"

  The region that is becoming the animal attends
  strongly to "red" and "fox".
  The region that is becoming the floor attends
  strongly to "blue" and "rug".

  Different parts of the image are steered by
  DIFFERENT parts of the prompt. That is why a prompt
  is compositional at all, rather than a single global
  style knob.
─────────────────────────────────────────

This also explains a whole class of failures. When cross-attention maps for "red" leak into the rug region, you get a red rug and a blue fox — the well-known attribute binding problem, and the reason Chapter 4's ControlNet exists as a spatial control that does not rely on attention alone.

5. Classifier-Free Guidance

Conditioning alone produces images that are only loosely on-prompt. Classifier-free guidance (CFG) amplifies the prompt's effect, and it is the single most impactful dial in image generation.

The Mechanism
─────────────────────────────────────────
  At EVERY denoising step, predict the noise TWICE:

    n_cond   = U-Net(latent, t, prompt embedding)
    n_uncond = U-Net(latent, t, EMPTY embedding)

  Then extrapolate AWAY from the unconditional one:

    n_final = n_uncond + g * (n_cond - n_uncond)

  (n_cond - n_uncond) is literally "the direction the
  prompt pulls in". g is guidance_scale.
─────────────────────────────────────────

@torch.no_grad()
def denoise_step(unet, latent, t, cond_emb, uncond_emb, guidance_scale):
    """One CFG-guided denoising step — the formula above, in code."""
    both = torch.cat([uncond_emb, cond_emb])              # BATCH the two passes together
    latent_in = torch.cat([latent] * 2)                   # so it is one U-Net call, not two
 
    noise_uncond, noise_cond = unet(latent_in, t, encoder_hidden_states=both).sample.chunk(2)
 
    # EXTRAPOLATE away from the unconditional prediction:
    return noise_uncond + guidance_scale * (noise_cond - noise_uncond)

Batching the two passes into one call is why CFG costs roughly 2x the compute rather than 2x the wall-clock time on a GPU with capacity to spare — the work doubles, the number of kernel launches does not.

Reading the Scale
─────────────────────────────────────────
  g = 1    no guidance; the raw conditional prediction.
           Diverse, often ignores much of the prompt.

  g = 7    the usual default. Balanced.

  g = 15   heavy extrapolation. Prompt adherence rises,
           then saturation, contrast and artefacts rise
           with it, and diversity collapses.

  g = 30   burnt, oversaturated, deep-fried. The
           extrapolation has left the region of latent
           space where the decoder produces natural
           images.
─────────────────────────────────────────

The Cost, and Why It Matters
─────────────────────────────────────────
  CFG runs the U-Net TWICE per step. Generation with
  guidance is ~2x the compute of generation without it.

  Every "turbo", "lightning" or distilled model you see
  is largely an attempt to recover this 2x — plus the
  step count — by training a model that needs neither.
─────────────────────────────────────────

6. Why Prompts Fail in Predictable Ways

Understanding the mechanism turns three common frustrations from mysteries into expected behaviour.

Failure                  Mechanistic Cause
─────────────────────────────────────────
  Long prompts lose      CLIP's text encoder has a hard
  their ending           token limit (77 in the original
                         CLIP). Text beyond it is
                         TRUNCATED — silently.

  "no cars" puts cars    CLIP embeds concepts, not
  in the image           logical negation. "no cars"
                         embeds NEAR "cars". Negation
                         belongs in the NEGATIVE prompt
                         (Chapter 3), which works by
                         replacing the unconditional
                         term in the CFG formula above.

  Counting fails         Nothing in contrastive training
  ("five apples")        rewards exact count. Captions
                         rarely enumerate, so the
                         embedding for "five" is weakly
                         grounded.
─────────────────────────────────────────

Each of these is a property of CLIP's training objective, not a prompt-writing skill issue — which is why the fixes in Chapter 3 and Chapter 4 are mechanisms, not better wording.

7. Summary & Next Steps

Key Takeaways

  • CLIP trains a text encoder and an image encoder to place matching pairs at the same point in one shared space, which is what makes a text vector a usable image-space instruction.
  • Contrastive matching over 400M web pairs forces genuine compositional visual semantics — and imports the biases of those captions along with them.
  • Text steers the image through cross-attention, letting different spatial regions attend to different prompt tokens; leakage between them causes attribute-binding errors.
  • Classifier-free guidance extrapolates along the conditional-minus-unconditional direction; it doubles compute, and pushing the scale too high leaves the natural-image region entirely.

Concept Check

  1. CLIP cannot generate an image or write a caption. Why is it nevertheless essential to a text-to-image model?
  2. Write out what guidance_scale does to the noise prediction, and explain why very high values produce oversaturated images.
  3. Why does the prompt "a street with no cars" tend to produce cars, and where does the fix actually apply?

Next Chapter

→ Chapter 3: Prompting Image Models

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index