Generative AI

Image Generation

Prompting Image Models

The instincts from NLP Module 7 actively mislead here. An LLM prompt is an instruction to a model trained to follow instructions. An image prompt is a caption f

JrCodex·7 min read

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Chapter 2: CLIP and Text Conditioning Time to complete: ~20 minutes

Table of Contents

  1. Image Prompts Are Not Text Prompts
  2. The Six-Slot Structure
  3. Negative Prompts
  4. Steps and Schedulers
  5. Seeds and Systematic Iteration
  6. Resolution and Aspect Ratio
  7. Summary & Next Steps

1. Image Prompts Are Not Text Prompts

The instincts from NLP Module 7 actively mislead here. An LLM prompt is an instruction to a model trained to follow instructions. An image prompt is a caption fed to an encoder trained only to match captions to pictures (Chapter 2).

The Consequences of That Difference
─────────────────────────────────────────
  LLM PROMPT                IMAGE PROMPT
  ─────────────────────────────────────────
  Instructions work         Descriptions work
  "Please write..."         "a photograph of..."

  Negation works            Negation does NOT work
  "do not mention X"        use a NEGATIVE prompt

  Long context helps        ~77 tokens, then truncated

  Politeness is neutral     Politeness is WASTED TOKENS
                            from a hard budget

  Reasoning step by step    No reasoning happens at all
─────────────────────────────────────────

Write what a good caption for the image you want would say. That single reframe fixes most beginner image prompts.

2. The Six-Slot Structure

Prompts that work consistently tend to fill the same six slots, roughly in order of importance.

The Template
─────────────────────────────────────────
  [1] MEDIUM      photograph, oil painting, 3D render,
                  pencil sketch, watercolour

  [2] SUBJECT     an elderly fisherman mending a net
                  (the most specific thing you say)

  [3] SETTING     on a weathered wooden dock at dawn

  [4] LIGHTING    soft golden hour light, backlit

  [5] COMPOSITION close-up portrait, shallow depth of
                  field, 35mm, rule of thirds

  [6] STYLE       in the style of documentary
                  photography, muted colour palette
─────────────────────────────────────────

Assembled
─────────────────────────────────────────
  "photograph of an elderly fisherman mending a net on
   a weathered wooden dock at dawn, soft golden hour
   light, close-up portrait, shallow depth of field,
   35mm, documentary style, muted colour palette"
─────────────────────────────────────────

Why Medium Comes First
─────────────────────────────────────────
  Position matters: earlier tokens carry more weight,
  and everything after ~77 tokens is discarded entirely.

  Medium is also the single highest-variance choice.
  "photograph" and "oil painting" of the same subject
  are further apart than any two subjects sharing a
  medium — so it belongs where it cannot be truncated.
─────────────────────────────────────────

Specificity Beats Adjective Stacking
─────────────────────────────────────────
  Weak:    "a beautiful amazing stunning gorgeous
              highly detailed masterpiece landscape"
              — near-synonyms compete for the same
              embedding region and add no information

  Strong:  "a limestone valley at dusk, low mist over
              a river, single bare oak in the
              foreground"
              — every phrase constrains something
              DIFFERENT
─────────────────────────────────────────

3. Negative Prompts

A negative prompt describes what you want absent. It works through the CFG mechanism from Chapter 2 rather than through any notion of negation.

The Mechanism
─────────────────────────────────────────
  Standard CFG:
     n_final = n_uncond + g * (n_cond - n_uncond)
     where n_uncond uses an EMPTY prompt

  With a negative prompt, the empty prompt is REPLACED:
     n_uncond = U-Net(latent, t, NEGATIVE embedding)

  So generation extrapolates AWAY from the negative
  prompt's region of the space. It is a direction of
  repulsion, not a filter.
─────────────────────────────────────────

image = pipe(
    prompt="photograph of a red fox in deep snow, golden hour, 35mm",
    negative_prompt="blurry, low quality, watermark, text, extra limbs, "
                    "oversaturated, cartoon, illustration",
    guidance_scale=7.0,
    num_inference_steps=30,
).images[0]
Using It Well
─────────────────────────────────────────
  DO   name concrete visual defects
       (blurry, watermark, jpeg artifacts, extra
        fingers, harsh shadows)

  DO   push away an unwanted MEDIUM
       (cartoon, illustration, 3D render) when you
       want a photograph

  DON'T write sentences — it is a bag of concepts,
        not a request

  DON'T overload it. A 40-term negative prompt repels
        the image from so much of the space that
        quality and diversity both drop.
─────────────────────────────────────────

4. Steps and Schedulers

num_inference_steps sets how many times the denoising loop runs. The scheduler (or sampler) decides how much noise to remove at each of those steps.

The Returns Curve
─────────────────────────────────────────
  quality
     ▲        ┌───────────────────────────
     │      ┌─┘  diminishing returns
     │    ┌─┘
     │  ┌─┘
     │ ┌┘
     │┌┘
     └┴──┬────┬────┬────┬────┬────► steps
        10   20   30   50   100

  Below ~15: visibly unresolved, muddy.
  20-35:     the working range for most schedulers.
  Above 50:  cost with no perceptible gain.
─────────────────────────────────────────

Schedulers, Practically
─────────────────────────────────────────
  DPM++ 2M Karras   the default recommendation.
                    Excellent quality at 20-30 steps.

  Euler a           "ancestral" — injects fresh noise
                    each step, so output keeps changing
                    with step count and never fully
                    converges. Creative, less
                    predictable.

  DDIM              deterministic and fast; useful when
                    you need a reproducible baseline.

  LCM / Turbo       distilled for 1-8 steps. Enormous
                    speedup, some quality cost. Use
                    guidance_scale near 1.0 — these are
                    trained not to need CFG.
─────────────────────────────────────────

from diffusers import DPMSolverMultistepScheduler
 
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True,
)
image = pipe(prompt=prompt, num_inference_steps=25, guidance_scale=7.0).images[0]
The One Trap
─────────────────────────────────────────
  ANCESTRAL schedulers (names ending in "a", e.g.
  Euler a) add noise at every step. Raising steps
  does not refine the SAME image — it produces a
  DIFFERENT one.

  So you cannot use them to A/B step count. Use a
  deterministic scheduler for that comparison.
─────────────────────────────────────────

5. Seeds and Systematic Iteration

Chapter 1 noted that a fixed seed gives a reproducible image. This is what turns prompting from guesswork into a controlled experiment.

import torch
 
BASE = "photograph of a red fox in deep snow, golden hour, 35mm"
 
# Step 1 — find a COMPOSITION you like: fix the prompt, vary the seed.
for seed in range(8):
    img = pipe(prompt=BASE, num_inference_steps=25, guidance_scale=7.0,
               generator=torch.Generator("cuda").manual_seed(seed)).images[0]
    img.save(f"explore_{seed}.png")
 
# Step 2 — LOCK that seed, then change exactly one thing at a time.
g = torch.Generator("cuda").manual_seed(3)          # the composition you chose
for scale in (4.0, 7.0, 10.0, 13.0):
    pipe(prompt=BASE, guidance_scale=scale, num_inference_steps=25,
         generator=torch.Generator("cuda").manual_seed(3)).images[0].save(f"cfg_{scale}.png")
The Two-Phase Workflow
─────────────────────────────────────────
  PHASE 1 — EXPLORE
    Fix everything, vary the seed. You are shopping for
    a composition, not tuning quality.

  PHASE 2 — REFINE
    Lock the seed. Change ONE variable per batch —
    prompt wording, guidance, steps, negative prompt.

  Changing the seed and the prompt together tells you
  nothing about either.
─────────────────────────────────────────

6. Resolution and Aspect Ratio

Generate at the Model's Native Resolution
─────────────────────────────────────────
  SD 1.5   trained at 512 x 512
  SDXL     trained at ~1024 x 1024

  Generating far from the native resolution produces a
  characteristic failure: DUPLICATED SUBJECTS. Ask
  SD 1.5 for 512x1024 and you often get two heads, or
  two horizons.

  Cause: the U-Net's receptive field was tuned for a
  specific canvas size. A taller canvas looks to it
  like room for another subject.
─────────────────────────────────────────

The Standard Fix: Generate Then Upscale
─────────────────────────────────────────
  1. Generate at native resolution
  2. Upscale 2-4x with a dedicated upscaler, or with
     img2img at low strength (Chapter 4)

  This is faster AND better than generating large
  directly — and it is why "hires fix" exists in
  every image UI.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • An image prompt is a caption, not an instruction: description works, negation and reasoning do not, and roughly 77 tokens is the hard budget.
  • The six-slot structure — medium, subject, setting, lighting, composition, style — puts the highest-variance and least-truncatable choices first.
  • Negative prompts replace the unconditional term in CFG, repelling generation from a region rather than filtering anything out.
  • Fix the seed to explore compositions, then lock it and change one variable at a time; ancestral schedulers break this by re-randomising every step.

Concept Check

  1. Why is "a highly detailed, beautiful, stunning masterpiece" a weaker prompt than three concrete descriptive clauses of the same length?
  2. Explain, in terms of the CFG formula, what a negative prompt actually changes.
  3. You asked SDXL for a 512x1536 image and got two subjects stacked vertically. What caused it and what is the standard remedy?

Next Chapter

→ Chapter 4: Controlling and Editing Images

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index