Personalization: LoRA, DreamBooth & Textual Inversion

Jr Codex Generative AI Notes

Level: Advanced Prerequisites: Chapter 4: Controlling and Editing Images; NLP Notes, Module 6, Chapter 2 Time to complete: ~25 minutes

Table of Contents

1. The Problem: Teaching a Model One New Thing
2. Three Methods, Three Places to Intervene
3. Textual Inversion
4. DreamBooth
5. LoRA — The Practical Default
6. Building a Training Set
7. Using and Combining LoRAs
8. Summary & Next Steps

1. The Problem: Teaching a Model One New Thing

No amount of prompt engineering will produce your specific dog, your company's product, or an illustration style that exists only in your studio. The model has never seen them, and Chapter 2 established that a prompt can only navigate to regions CLIP already knows.

Chapter 4's IP-Adapter approximates this without training. When approximation is not enough, you have to change the model — and the question becomes how little you can change.

The Constraint That Shapes Everything
─────────────────────────────────────────
  You have 5-30 images.
  The model was trained on hundreds of millions.

  Naive fine-tuning on 20 images will make the model
  produce ONLY those 20 images, for every prompt. This
  is CATASTROPHIC FORGETTING, and it is the failure
  mode all three methods are designed around.
─────────────────────────────────────────

2. Three Methods, Three Places to Intervene

The methods differ in where in the pipeline they inject the new concept.

Where Each One Acts
─────────────────────────────────────────
  prompt ──► TEXT ENCODER ──► U-NET ──► VAE ──► image
                  ▲             ▲
                  │             │
        TEXTUAL INVERSION    LoRA / DREAMBOOTH
        adds a new token     modifies the denoiser
        embedding only       weights themselves
─────────────────────────────────────────

3. Textual Inversion

The most conservative approach: freeze the entire model and learn only a new word.

The Mechanism
─────────────────────────────────────────
  Add one new token, <my-style>, to the vocabulary.
  Initialise its embedding randomly.

  Train: given your images and captions containing
  <my-style>, do gradient descent on the EMBEDDING
  ONLY. Every network weight stays frozen.

  Result: a single vector that points at the region of
  CLIP space where your concept lives.
─────────────────────────────────────────

Its Strength and Its Ceiling
─────────────────────────────────────────
  STRENGTH   The model is completely unchanged, so
             forgetting is IMPOSSIBLE. Tiny files, and
             several can be loaded at once.

  CEILING    A single vector can only point at concepts
             the model can ALREADY render. It works
             well for a STYLE ("painted like this") and
             poorly for a specific FACE, because no
             existing region of the space contains it.
─────────────────────────────────────────

4. DreamBooth

The opposite extreme: fine-tune the whole U-Net, using two ideas to prevent forgetting.

Idea 1 — A RARE TOKEN IDENTIFIER
─────────────────────────────────────────
  Bind the subject to a token with little prior
  meaning: "sks", "ohwx", "zwx".

  ✗ "a photo of a dog"      overwrites ALL dogs
  ✓ "a photo of sks dog"    binds to sks, leaves the
                            general concept of "dog"
                            available as context

Idea 2 — PRIOR PRESERVATION LOSS
─────────────────────────────────────────
  Before training, generate ~200 images of the generic
  class ("a photo of a dog") using the ORIGINAL model.

  During training, alternate:
    - your images, captioned "a photo of sks dog"
    - the generated images, captioned "a photo of a dog"

  The second stream continuously reminds the model what
  dogs in general look like, so learning sks does not
  destroy the class.
─────────────────────────────────────────

DreamBooth gives the highest fidelity, but produces a multi-gigabyte model per subject and cannot be combined with another. In practice it has been almost entirely superseded by LoRA trained with the same two ideas.

5. LoRA — The Practical Default

LoRA is the same technique the NLP Notes applied to LLMs, applied to the U-Net's attention layers. The theory is identical; only the target changes.

The Recap, in Image Terms
─────────────────────────────────────────
  A weight update ΔW is approximated as B x A, where
  A and B are thin matrices of rank r.

  For a 1024 x 1024 attention weight at r = 16:
     full ΔW  = 1,048,576 parameters
     B x A    =    32,768 parameters   (~3%)

  Train A and B, keep W frozen. At inference:
     W_effective = W + scale * (B x A)
─────────────────────────────────────────

Which Layers, and Why It Matters
─────────────────────────────────────────
  Applied to the CROSS-ATTENTION layers — exactly the
  layers where text meets image (Chapter 2).

  That is the point: a LoRA does not teach the model
  new visual primitives. It re-routes which existing
  visual capabilities a given phrase reaches.
─────────────────────────────────────────

# Training (with the diffusers example script)
# accelerate launch train_dreambooth_lora_sdxl.py \
#   --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
#   --instance_data_dir="./my_images" \
#   --instance_prompt="a photo of sks terrier" \
#   --with_prior_preservation --class_prompt="a photo of a terrier" \
#   --rank=16 --learning_rate=1e-4 --max_train_steps=1200
 
# Inference
pipe.load_lora_weights("./output", weight_name="pytorch_lora_weights.safetensors")
image = pipe(
    prompt="a photo of sks terrier wearing a knitted scarf, studio lighting",
    cross_attention_kwargs={"scale": 0.8},      # LoRA strength: 0 = off, 1 = full
).images[0]

Hyperparameters That Actually Matter
─────────────────────────────────────────
  rank (r)         4-8   styles (styles are simple)
                   16-32 subjects and faces
                   >64   rarely helps, overfits fast

  learning rate    1e-4 is a solid default for r=16

  steps            roughly 100 x (number of images).
                   20 images ──► ~2000 steps.
                   Save checkpoints every 250 steps and
                   pick by EYE — the best checkpoint is
                   almost never the last.

  scale at         0.6-0.9. If output looks burnt or
  inference        every image is identical, LOWER this
                   before retraining.
─────────────────────────────────────────

6. Building a Training Set

Data quality dominates every hyperparameter here. This is where most failed LoRAs are lost.

For a SUBJECT (person, pet, product)
─────────────────────────────────────────
  Count       15-25 images. More is not better if it
              is more of the SAME.

  Vary        angle, distance, lighting, background,
              clothing/context

  Hold fixed  ONLY the subject itself

  Exclude     other people, heavy filters, motion blur,
              anything cropped through the subject
─────────────────────────────────────────

For a STYLE
─────────────────────────────────────────
  Count       20-50 images

  Vary        SUBJECT MATTER as widely as possible —
              portraits, landscapes, objects, interiors

  Hold fixed  the rendering style

  Why: the model must learn "this look" as separable
  from "this content". If every training image is a
  portrait, the LoRA will learn to draw portraits.
─────────────────────────────────────────

The Captioning Rule
─────────────────────────────────────────
  CAPTION WHAT SHOULD VARY. OMIT WHAT SHOULD BE BAKED IN.

  Anything you describe becomes a controllable
  variable. Anything you leave undescribed gets
  absorbed into the trigger token.

  Subject LoRA — you want pose and setting controllable:
     "sks terrier sitting on a wooden porch, daylight"

  Style LoRA — you want the style baked in, always on:
     "mystyle, a lighthouse on a cliff"
     (describe the content, NOT the style)
─────────────────────────────────────────

That rule is the single most useful thing in this chapter, and the most commonly got wrong.

7. Using and Combining LoRAs

pipe.load_lora_weights("./subject_lora", adapter_name="terrier")
pipe.load_lora_weights("./style_lora",   adapter_name="woodcut")
pipe.set_adapters(["terrier", "woodcut"], adapter_weights=[0.9, 0.6])
 
image = pipe(prompt="mystyle, a photo of sks terrier on a hillside").images[0]

Combining Rules
─────────────────────────────────────────
  KEEP THE TOTAL near 1.0-1.5. Two LoRAs at 1.0 each
  usually produces artefacts — both are pulling the
  same weights in different directions.

  ONE SUBJECT ONLY. Two subject LoRAs will blend into
  a single hybrid, not place two individuals in a
  scene. For that, generate separately and composite
  with inpainting (Chapter 4).

  Subject + style is the reliable combination. Weight
  the subject higher.
─────────────────────────────────────────

Diagnosing a Bad LoRA
─────────────────────────────────────────
  Every output is identical, ignores the prompt
     ──► OVERFIT. Fewer steps, an earlier checkpoint,
         or lower inference scale.

  Subject is barely recognisable
     ──► UNDERFIT, or too few varied images. More
         steps, or raise rank.

  Backgrounds all look like the training photos
     ──► You did not caption the backgrounds, so they
         were baked into the trigger. Re-caption.

  Style LoRA only produces portraits
     ──► Training set lacked subject variety.
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

• All personalization methods work around catastrophic forgetting; they differ in where they intervene — the token embedding, the full U-Net, or low-rank adapters.
• DreamBooth's two contributions, a rare trigger token and prior preservation loss, are now applied through LoRA rather than through full fine-tuning.
• LoRA modifies cross-attention layers, so it re-routes which visual capabilities a phrase reaches rather than teaching new visual primitives.
• Caption what should vary and omit what should be baked in — this single rule explains most LoRA successes and failures.

Module 3 Complete — What's Next

You can now generate, steer, edit, and personalise images. Module 4 applies the same conceptual toolkit — latent spaces, conditioning, guidance — to audio and video, where the extra dimension of time changes what is hard.

Concept Check

1. Why does textual inversion work well for a painting style but poorly for a specific person's face?
2. What is prior preservation loss preventing, and how does the generated-class-images stream prevent it?
3. Your style LoRA reproduces the training images' backgrounds no matter what you prompt. Which captioning mistake caused it?

Next Module

→ Module 4: Audio, Video & Multimodal

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index