Generative AI

Audio Video And Multimodal

Video Generation

Video is not a harder version of image generation. It is a different problem, and the reason is easy to state.

JrCodex·7 min read

Jr Codex Generative AI Notes

Level: Intermediate–Advanced Prerequisites: Chapter 2: Music & Audio Generation Time to complete: ~20 minutes

Table of Contents

  1. The Consistency Problem
  2. Spatiotemporal Latents
  3. Attention Across Time
  4. The Modes of Video Generation
  5. Working Around the Limits
  6. An Honest Assessment
  7. Summary & Next Steps

1. The Consistency Problem

Video is not a harder version of image generation. It is a different problem, and the reason is easy to state.

Why Frame-by-Frame Fails
─────────────────────────────────────────
  Generate 120 frames independently, each an excellent
  image, and you get 120 DIFFERENT scenes.

  The character's jacket changes colour. The background
  building gains a window. Shadows fall from a new
  direction each frame.

  Every individual frame is correct. The video is
  unwatchable.
─────────────────────────────────────────

What Consistency Actually Requires
─────────────────────────────────────────
  IDENTITY      the same face, clothes and objects
                across every frame

  PHYSICS       plausible motion — things fall, do not
                teleport, and keep their momentum

  ILLUMINATION  one coherent light source over time

  OCCLUSION     an object hidden behind another comes
                back UNCHANGED

  Human viewers detect violations of all four
  instantly and unforgivingly — far more sensitively
  than they detect flaws in a still image.
─────────────────────────────────────────

That last point is the crux: our tolerance for error in video is far lower than in images, at exactly the moment the technical problem got harder.

2. Spatiotemporal Latents

The fix starts by extending Module 3's latent compression into the time dimension.

From 2D to 3D Compression
─────────────────────────────────────────
  IMAGE VAE      compresses HEIGHT and WIDTH
                 512x512x3  ──►  64x64x4

  VIDEO VAE      compresses HEIGHT, WIDTH and TIME
                 16x512x512x3  ──►  4x64x64x4
                    │                │
                    16 frames        4 latent frames

  Temporal compression works for the same reason
  spatial compression does: consecutive frames are
  overwhelmingly REDUNDANT. Most pixels do not change.
─────────────────────────────────────────

The Key Consequence
─────────────────────────────────────────
  The model generates the WHOLE CLIP as one object in a
  single denoising process — not frame by frame.

  Consistency is therefore not enforced afterwards. It
  is a property of generating one joint sample, in the
  same way that a single image's left and right halves
  are automatically consistent.
─────────────────────────────────────────

3. Attention Across Time

Inside the denoising network, attention is extended so that positions can see other times, not only other places.

Three Attention Patterns
─────────────────────────────────────────
  SPATIAL       within one frame. Standard image
                self-attention.

  TEMPORAL      one spatial position attending to the
                SAME position across all frames.
                Cheap. Keeps a pixel's colour stable
                over time.

  FULL 3D       every position attending to every
                position in every frame.
                Expensive — cost grows with the SQUARE
                of (frames x pixels) — but the only
                pattern that handles large motion,
                where a subject MOVES to a different
                spatial position.
─────────────────────────────────────────

Why Videos Are Short
─────────────────────────────────────────
  Attention cost is quadratic in sequence length (DL
  Notes, Module 6). In video the sequence is
  frames x height x width.

  Doubling clip LENGTH more than quadruples cost.

  This is not an engineering oversight to be optimised
  away. It is the reason clips are measured in seconds,
  and the central research problem in the field.
─────────────────────────────────────────

4. The Modes of Video Generation

TEXT-TO-VIDEO
─────────────────────────────────────────
  Prompt only. Maximum freedom, minimum control.
  Composition, framing and pacing are all a lottery.

IMAGE-TO-VIDEO
─────────────────────────────────────────
  A still image becomes the first frame; the model
  generates motion forward from it.

  Far more controllable, and the workhorse in
  practice: use Module 3's full toolkit to get exactly
  the frame you want — prompt, ControlNet, LoRA,
  inpainting — then animate it.

  Composition becomes a SOLVED image problem instead
  of an unsolved video problem.

VIDEO-TO-VIDEO
─────────────────────────────────────────
  Restyle existing footage while keeping its motion.
  The source supplies structure and timing, so
  consistency is largely inherited rather than
  generated.
─────────────────────────────────────────

Image-to-video is the recommended default. It converts the hardest part of the problem into one you already know how to solve.

import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import export_to_video
 
pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16, variant="fp16",
).to("cuda")
pipe.enable_model_cpu_offload()        # video latents are LARGE — offload or run out of VRAM
 
frames = pipe(
    first_frame,                       # produced with Module 3's full toolkit
    num_frames=25,
    motion_bucket_id=127,              # motion STRENGTH: higher = more movement AND artefacts
    noise_aug_strength=0.02,           # how far the first frame may drift; keep LOW
    decode_chunk_size=8,               # decode frames in chunks, not all at once
    generator=torch.Generator("cuda").manual_seed(7),
).frames[0]
 
export_to_video(frames, "shot_01.mp4", fps=7)
def chain_clips(pipe, first_frame, hops=3, **kw):
    """Extend length by feeding each clip's LAST frame into the next."""
    frame, all_frames = first_frame, []
    for _ in range(hops):
        clip = pipe(frame, **kw).frames[0]
        all_frames.extend(clip)
        frame = clip[-1]               # the handoff — and where DRIFT accumulates
    return all_frames                  # expect visible degradation past ~3-4 hops

5. Working Around the Limits

Motion Fails or Is Too Subtle
─────────────────────────────────────────
  Most models expose a motion strength parameter.
  Raising it increases movement AND artefacts together.

  Describe motion explicitly and simply: "the camera
  slowly pushes in", "steam rises from the cup". Vague
  prompts produce drifting, aimless motion.

Clips Are Too Short
─────────────────────────────────────────
  Chain them: take the LAST frame of clip N as the
  FIRST frame of clip N+1.

  This works, and it drifts. Quality degrades with each
  hop as small errors compound. Three or four hops is
  usually the practical ceiling before the subject has
  visibly changed.

The Character Changes Between Shots
─────────────────────────────────────────
  There is no reliable cross-clip identity mechanism.

  Best available approach: generate a consistent
  CHARACTER SHEET with a subject LoRA (Module 3,
  Chapter 5), use those stills as first frames, and
  keep every shot short.

Faces and Hands Distort in Motion
─────────────────────────────────────────
  Both are high-detail, high-articulation regions.
  Keep them further from camera, or avoid shots where
  a hand manipulates an object.
─────────────────────────────────────────

6. An Honest Assessment

This is the fastest-moving area in this curriculum, and the most oversold. Specific rather than general claims:

Reliable Now
─────────────────────────────────────────
  - 3-10 second clips from a controlled first frame
  - Ambient motion: cloth, water, smoke, foliage,
    gentle camera moves
  - Restyling existing footage
  - B-roll, backgrounds, transitions, loops

Not Reliable Now
─────────────────────────────────────────
  - A named character consistent across a sequence
  - Dialogue with correct lip sync
  - Precise physical interaction (pouring, catching,
    a hand picking something up)
  - Legible text on screen
  - Anything over ~30 seconds without visible drift
  - Matching an exact duration for an edit

The Production Reality
─────────────────────────────────────────
  Usable output typically requires generating MANY
  candidates and selecting — often 10-30 for one
  usable shot.

  Budget for that ratio. A plan that assumes one
  generation per shot will miss by an order of
  magnitude on both cost and time.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Independently generated frames produce 120 different scenes; consistency must come from generating the clip as one joint sample, not from post-processing.
  • Video VAEs compress time as well as space, exploiting the redundancy between consecutive frames, so the denoiser works on a whole spatiotemporal latent.
  • Attention cost is quadratic in frames × pixels, which is the structural reason clips are seconds long rather than minutes.
  • Image-to-video is the practical default: it turns composition into a solved image problem and leaves only motion to the video model.

Concept Check

  1. Why is consistency an emergent property of spatiotemporal generation rather than something enforced after the fact?
  2. Explain why doubling clip length more than quadruples generation cost.
  3. You need a 40-second sequence featuring one recognisable character. Describe the approach you would actually take, and state plainly which part remains unreliable.

Next Chapter

→ Chapter 4: Multimodal Models

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index