Generative AI

Image Generation

Controlling and Editing Images

Text is a low-bandwidth control surface. No caption reliably specifies "this exact pose, this exact composition, this person's face." Every technique in this ch

JrCodex·8 min read

Jr Codex Generative AI Notes

Level: Intermediate–Advanced Prerequisites: Chapter 3: Prompting Image Models Time to complete: ~25 minutes

Table of Contents

  1. Beyond Text-Only Control
  2. img2img — Starting from an Image
  3. Inpainting — Editing a Region
  4. Outpainting — Extending the Canvas
  5. ControlNet — Imposing Structure
  6. IP-Adapter — Conditioning on an Image
  7. Composing the Controls
  8. Summary & Next Steps

1. Beyond Text-Only Control

Text is a low-bandwidth control surface. No caption reliably specifies "this exact pose, this exact composition, this person's face." Every technique in this chapter adds a second conditioning channel alongside the prompt.

The Control Surfaces, Mapped
─────────────────────────────────────────
  WHAT YOU WANT TO FIX        USE
  ─────────────────────────────────────────
  Overall colour and layout   img2img
  One region, rest untouched  inpainting
  More canvas around it       outpainting
  Exact pose / edges / depth  ControlNet
  Subject or style identity   IP-Adapter (this ch.)
                              LoRA / DreamBooth (Ch.5)
─────────────────────────────────────────

2. img2img — Starting from an Image

Instead of starting the denoising loop from pure noise, start it from a partially noised real image.

The Mechanism
─────────────────────────────────────────
  Text-to-image:  pure noise ──► 30 steps ──► image

  img2img:        real image
                    │ VAE-encode to a latent
                    │ add noise up to strength s
                    ▼
                  noisy latent ──► (1-s) x 30 steps
                    ▼
                  new image

  STRENGTH is literally "how far back toward noise do
  we push the original before denoising forward again".
─────────────────────────────────────────

from diffusers import StableDiffusionXLImg2ImgPipeline
from PIL import Image
 
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0", torch_dtype=torch.float16).to("cuda")
 
result = pipe(
    prompt="an oil painting of a harbour at dusk, thick impasto brushwork",
    image=Image.open("photo.jpg").convert("RGB").resize((1024, 1024)),
    strength=0.55,          # THE dial: 0 = unchanged, 1 = the original is ignored
    guidance_scale=7.0,
).images[0]
Reading Strength
─────────────────────────────────────────
  0.15 - 0.30   subtle: denoise, upscale-refine, minor
                  cleanup. Composition preserved exactly.

  0.40 - 0.60   the useful middle: restyle while keeping
                  layout, colour blocking and pose.

  0.70 - 0.85   heavy reinterpretation. Only broad
                  composition survives.

  0.95 +        effectively text-to-image. The input
                  contributes almost nothing.
─────────────────────────────────────────

Note the free side-effect: strength also controls cost. At strength=0.4 only 40% of the steps run, so an img2img pass is proportionally cheaper than a full generation.

3. Inpainting — Editing a Region

Inpainting regenerates a masked region while holding everything else fixed.

The Mechanism
─────────────────────────────────────────
  At EVERY denoising step:

    1. denoise the whole latent normally
    2. OVERWRITE the unmasked area with the correctly
       noised version of the ORIGINAL

  The masked region is generated fresh, but at every
  step it sees the real surroundings — which is why the
  result blends rather than sitting in a rectangle.
─────────────────────────────────────────

from diffusers import StableDiffusionXLInpaintPipeline
 
pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1", torch_dtype=torch.float16).to("cuda")
 
result = pipe(
    prompt="a vase of white peonies",       # describe the FINAL region, not the edit
    image=base_image,
    mask_image=mask,                        # WHITE = regenerate, BLACK = keep
    strength=0.99,
    num_inference_steps=30,
).images[0]
Three Rules That Fix Most Bad Inpaints
─────────────────────────────────────────
  1. DESCRIBE THE RESULT, NOT THE ACTION.
     ✗ "remove the car"
     ✓ "empty asphalt road, wet, reflective"
     There is no delete operation — you can only
     specify what should be there instead.

  2. FEATHER THE MASK.
     A hard-edged mask produces a visible seam. Blur
     the mask by 8-20 pixels.

  3. MASK GENEROUSLY.
     Include a margin of surrounding context. A mask
     tight to the object leaves its shadow, its
     reflection, and its outline behind.
─────────────────────────────────────────

4. Outpainting — Extending the Canvas

Outpainting is inpainting where the mask is outside the original image.

The Setup
─────────────────────────────────────────
  ┌───────────────────────────┐
  │░░░░░░ generate ░░░░░░░░░░░│
  │░░░┌───────────────────┐░░░│
  │░░░│                   │░░░│
  │░░░│  original image   │░░░│  ░ = masked (white)
  │░░░│      (kept)       │░░░│    = generate here
  │░░░└───────────────────┘░░░│
  │░░░░░░░░░░░░░░░░░░░░░░░░░░░│
  └───────────────────────────┘

  Extend in STEPS of 25-30% per pass, not all at once.
  Each pass gives the model real pixels to anchor to;
  one large extension gives it too little context and
  drifts into unrelated content.
─────────────────────────────────────────

5. ControlNet — Imposing Structure

Every technique so far controls content. ControlNet controls geometry — pose, edges, depth, layout — independently of what the prompt describes.

The Architecture
─────────────────────────────────────────
  A trainable COPY of the U-Net's encoder runs
  alongside the frozen original, taking a CONTROL IMAGE
  as its input. Its outputs are added into the main
  U-Net at each corresponding block.

     control image (an edge map, a pose skeleton)
           │
           ▼
     ControlNet copy ──► residuals ──┐
                                     ▼
     prompt ──► frozen U-Net ────► combined ──► output

  Key design point: the connections start at ZERO
  weight, so an untrained ControlNet is a no-op. It can
  only ADD structure, never break the base model.
─────────────────────────────────────────

The Common Conditioning Types
─────────────────────────────────────────
  Canny edges   strictest. Preserves precise outlines.
                Use for: restyling while keeping exact
                shapes; product images.

  Depth         preserves 3D layout, allows surface
                detail to change freely. Use for:
                re-lighting, changing materials.

  OpenPose      a human skeleton only. Body ignored, so
                clothing, physique and setting are free.
                Use for: character posing.

  Scribble      loosest. A rough sketch becomes a
                composition. Use for: layout blocking.

  Segmentation  region-by-region semantic layout.
                Use for: architectural and scene design.
─────────────────────────────────────────

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
import cv2, numpy as np
 
controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet,
    torch_dtype=torch.float16).to("cuda")
 
edges = cv2.Canny(np.array(source_image), 100, 200)          # extract the STRUCTURE
control = Image.fromarray(np.stack([edges] * 3, axis=-1))
 
result = pipe(
    prompt="a brass and walnut art-deco lamp, studio product photo",
    image=control,
    controlnet_conditioning_scale=0.7,      # 0.5-0.8 typical; 1.0 often too rigid
    num_inference_steps=30,
).images[0]
Tuning the Conditioning Scale
─────────────────────────────────────────
  Too LOW (< 0.4)   structure is ignored; you have an
                    expensive text-to-image call

  Too HIGH (> 0.9)  the control image dominates, the
                    prompt is starved, and artefacts
                    from the edge map bleed into the
                    output

  0.5 - 0.8 is the working band. If the prompt is being
  ignored, LOWER this before rewriting the prompt.
─────────────────────────────────────────

6. IP-Adapter — Conditioning on an Image

Sometimes the reference you have is a style or a subject, not a geometry. IP-Adapter accepts an image as a second prompt.

The Idea
─────────────────────────────────────────
  Encode the reference image with CLIP's IMAGE encoder
  (Chapter 2), project it into the same space the text
  embeddings occupy, and give the U-Net a SECOND
  cross-attention path for it.

  Result: "generate something with THIS look" without
  training anything.
─────────────────────────────────────────
IP-AdapterLoRA / DreamBooth (Chapter 5)
Training neededNone — zero-shotYes, minutes to hours
Fidelity to subjectGoodExcellent
Setup costOne reference image10–30 images + a training run
Best forStyle transfer, quick likeness, one-off referencesA recurring character, brand, or product

The practical rule: reach for IP-Adapter first. If the identity is not faithful enough after tuning its scale, and you need it repeatedly, then train a LoRA.

7. Composing the Controls

These stack, and the combinations are where the real work happens.

Recipes
─────────────────────────────────────────
  Product on a new background
     ControlNet (canny, 0.8) + inpainting on the
     background region

  A character in a new pose
     ControlNet (openpose, 0.7) + IP-Adapter for the
     face + a subject LoRA (Chapter 5)

  Restyle a photo, keep the layout exactly
     ControlNet (depth, 0.6) + img2img at strength 0.5

  Widen a photo for a banner
     Outpaint in two 25% passes, then img2img at
     strength 0.2 over the whole result to unify it
─────────────────────────────────────────

The Ordering Principle
─────────────────────────────────────────
  Apply controls from MOST to LEAST structural:

    geometry (ControlNet)
      ──► region (inpaint mask)
        ──► identity (IP-Adapter, LoRA)
          ──► content and finish (prompt, img2img)

  Adding more controls narrows the space the model can
  sample from. Stack four strong ones and you will get
  artefacts — because you have left almost nowhere
  valid to sample.
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

  • img2img starts denoising from a partially noised real image; strength sets how far back toward noise it goes, and therefore how much of the original survives.
  • Inpainting re-noises only the masked region while re-imposing the real surroundings at every step — so describe the desired result, feather the mask, and mask generously.
  • ControlNet adds a zero-initialised trainable copy of the U-Net encoder to impose geometry (edges, depth, pose) independently of the prompt; 0.5–0.8 is the useful conditioning band.
  • IP-Adapter conditions on a reference image with no training and is the right first attempt before committing to a LoRA.

Concept Check

  1. Why is strength=0.4 in img2img both a quality setting and a cost saving?
  2. "Remove the car" is a poor inpainting prompt. Explain why in terms of what inpainting actually does, and give a better one.
  3. Your ControlNet output follows the edge map perfectly but ignores the prompt entirely. Which parameter is wrong and in which direction?

Next Chapter

→ Chapter 5: Personalization — LoRA, DreamBooth & Textual Inversion

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index