Capstone: A Multimodal Content Studio

Jr Codex Generative AI Notes

Level: Advanced Prerequisites: All previous modules Time to complete: ~30 minutes reading; the build is a multi-session project

Table of Contents

1. What You Are Building
2. The Architecture
3. Stage 1 — The Script
4. Stage 2 — The Images
5. Stage 3 — The Narration
6. The Pipeline, Assembled
7. Evaluation and Guardrails
8. Extensions
9. Summary & Next Steps

1. What You Are Building

A service that turns a topic into a narrated illustrated segment: a structured script, one image per scene in a consistent style, and synchronised narration.

The Contract
─────────────────────────────────────────
  IN   a topic, an audience, a target duration,
       a visual style

  OUT  a script (structured, per scene)
       one image per scene, visually consistent
       narration audio per scene
       a manifest tying it together, with provenance
─────────────────────────────────────────

Why This Capstone
─────────────────────────────────────────
  It exercises every module:

  Module 2   structured script generation, budgeting
  Module 3   consistent image generation
  Module 4   narration and script-for-speech
  Module 5   evaluating output with no ground truth
  Module 6   provenance, disclosure, rights
  Module 7   async jobs, caching, cost control

  And it has the property that makes generative
  projects hard: three stages that must agree with
  each other.
─────────────────────────────────────────

2. The Architecture

The Pipeline
─────────────────────────────────────────
  POST /projects  ──►  validate, enqueue, return id
        │
        ▼
  [1] SCRIPT      one structured LLM call
        │              (Module 2, Chapter 3)
        ▼
  [2] STYLE BIBLE derived ONCE from the script
        │              (the consistency mechanism)
        ▼
  [3] IMAGES      one per scene, IN PARALLEL
        │              (Module 3)
        ▼
  [4] NARRATION   one per scene, IN PARALLEL
        │              (Module 4, Chapter 1)
        ▼
  [5] ASSEMBLE    manifest + provenance + timings
        │              (Module 6, Chapter 3)
        ▼
  GET /projects/{id}  ──►  status and results
─────────────────────────────────────────

Two Structural Decisions
─────────────────────────────────────────
  STAGES ARE SEQUENTIAL, ITEMS ARE PARALLEL.
  Scene 4's image does not depend on scene 3's, so
  generate them concurrently. But no image can start
  before the script and style bible exist.

  EACH STAGE IS SEPARATELY RETRYABLE.
  A failed image at scene 7 must not re-run the
  script and the other six images. Persist stage
  output before moving on.
─────────────────────────────────────────

3. Stage 1 — The Script

from pydantic import BaseModel, Field
from typing import Literal
 
class Scene(BaseModel):
    index: int
    narration: str = Field(description="Spoken text. Write for SPEECH: expand "
                                       "abbreviations and numerals, use short "
                                       "sentences, punctuate for pauses.")
    visual: str = Field(description="What is SEEN. A concrete visual description "
                                    "with no text or lettering in the image.")
    seconds: float = Field(ge=3, le=25)
 
class Script(BaseModel):
    title: str
    audience: Literal["beginner", "practitioner", "executive"]
    scenes: list[Scene] = Field(min_length=3, max_length=12)
 
    @property
    def duration(self) -> float:
        return sum(s.seconds for s in self.scenes)
 
def write_script(client, topic, audience, target_seconds):
    return client.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {"role": "system", "content":
                "You write short narrated explainers. Each scene pairs ONE spoken "
                "idea with ONE concrete visual. Narration is written to be READ "
                "ALOUD. Visuals never contain text or lettering."},
            {"role": "user", "content":
                f"Topic: {topic}\nAudience: {audience}\nTarget: {target_seconds}s"},
        ],
        response_format=Script, temperature=0.7,
    ).choices[0].message.parsed

The Two Instructions Carrying the Design
─────────────────────────────────────────
  "Narration is written to be READ ALOUD."
    Module 4, Chapter 1: with no prosody API,
    punctuation and spelled-out numbers ARE the
    control surface. Getting this in the script stage
    is far cheaper than fixing audio later.

  "Visuals never contain text or lettering."
    Module 1, Chapter 4: image models render text
    unreliably. Design the constraint OUT of the
    system rather than fighting it downstream.
─────────────────────────────────────────

4. Stage 2 — The Images

The hard requirement is consistency: twelve images that look like one piece of work.

The Style Bible Pattern
─────────────────────────────────────────
  Derive ONE style specification from the script,
  then append it verbatim to EVERY scene prompt.

  Fix: medium, palette, lighting, composition, finish
  Vary: subject only

  Also fix the SEED per project. Same seed + same
  style suffix ──► a coherent visual language across
  scenes (Module 3, Chapter 3).
─────────────────────────────────────────

STYLE_BIBLE = (
    "flat vector illustration, limited palette of deep teal, warm sand and off-white, "
    "soft even lighting, generous negative space, centred composition, "
    "no text, no lettering, no watermark"
)
 
NEGATIVE = ("photorealistic, 3d render, cluttered, harsh shadows, text, letters, "
            "watermark, signature, extra limbs, blurry, oversaturated")
 
def render_scene(pipe, scene: Scene, project_seed: int):
    return pipe(
        prompt=f"{scene.visual}, {STYLE_BIBLE}",     # SUBJECT varies, STYLE is constant
        negative_prompt=NEGATIVE,
        num_inference_steps=28,
        guidance_scale=6.5,
        generator=torch.Generator("cuda").manual_seed(project_seed + scene.index),
    ).images[0]

Why seed + index Rather Than One Seed
─────────────────────────────────────────
  A single identical seed for every scene produces
  near-identical COMPOSITIONS — the same layout twelve
  times.

  project_seed + index gives each scene its own
  composition while keeping the whole project
  reproducible from one number: regenerate the entire
  set exactly, or change the project seed to reshuffle
  all compositions at once.
─────────────────────────────────────────

5. Stage 3 — The Narration

def narrate(client, scene: Scene, voice: str, out_dir: str) -> str:
    path = f"{out_dir}/scene_{scene.index:02d}.mp3"
    with client.audio.speech.with_streaming_response.create(
        model="tts-1-hd", voice=voice, input=scene.narration, response_format="mp3",
    ) as response:
        response.stream_to_file(path)
    return path
 
def verify_narration(whisper_model, path: str, expected: str) -> float:
    """Round-trip WER — Module 5, Chapter 2's cheap intelligibility check."""
    segments, _ = whisper_model.transcribe(path)
    heard = " ".join(s.text for s in segments)
    return word_error_rate(normalise(expected), normalise(heard))

The Round-Trip Check Earns Its Place
─────────────────────────────────────────
  Transcribe the generated audio and compare it to the
  script. A WER above ~10% means the TTS mangled
  something — usually an acronym, a number, or an
  unusual proper noun.

  This is an automatic, cheap ASSERTION (Module 5's
  rung 1) on a modality that otherwise requires
  listening to everything.
─────────────────────────────────────────

6. The Pipeline, Assembled

import asyncio
 
async def build_project(ctx, job: Job, topic: str, audience: str, seconds: int):
    budget = ctx.budget.check(job.user_id, estimated_cents=estimate(seconds))
 
    # STAGE 1 — sequential, and everything downstream depends on it
    script = await ctx.run_stage(job, "script",
                                 lambda: write_script(ctx.llm, topic, audience, seconds))
 
    if not passes_assertions(script):                    # rung 1 — before spending on media
        raise StageFailed("script failed validation")
 
    # STAGES 2 AND 3 — parallel across scenes, and parallel with each other
    images, audio = await asyncio.gather(
        ctx.run_stage(job, "images",
                      lambda: asyncio.gather(*[render(ctx, s, job.seed) for s in script.scenes])),
        ctx.run_stage(job, "audio",
                      lambda: asyncio.gather(*[speak(ctx, s) for s in script.scenes])),
    )
 
    # STAGE 5 — provenance and manifest (Module 6, Chapter 3)
    manifest = assemble(script, images, audio, job)
    attach_c2pa(manifest, generator="jrcodex-studio", model_versions=ctx.versions)
    ctx.budget.record(job.user_id, actual_cents=job.cost_cents)
    return manifest

Note the Ordering
─────────────────────────────────────────
  The script assertion runs BEFORE any image or audio
  generation.

  Script generation costs a fraction of a cent. Twelve
  images and twelve narrations cost orders of
  magnitude more. VALIDATE THE CHEAP STAGE FIRST — the
  single most effective cost control in a multi-stage
  generative pipeline.
─────────────────────────────────────────

7. Evaluation and Guardrails

Assertions — every run, must pass
─────────────────────────────────────────
  □ script parses; 3-12 scenes; durations within range
  □ total duration within 20% of target
  □ every image is the expected dimensions
  □ every narration file exists and is non-silent
  □ round-trip WER < 10% per scene
  □ no banned terms in prompts (Module 6, Chapter 1)

Metrics — tracked per run
─────────────────────────────────────────
  CLIPScore per scene       does the image match its
                            visual description?
  Style consistency         mean pairwise CLIP image
                            embedding similarity
                            across scenes — HIGH is
                            good here, unusually
  Cost and latency          per stage, so you know
                            which stage to optimise
  Regeneration rate         per scene, from real users

Judge — before each release
─────────────────────────────────────────
  A rubric over 30 saved projects (Module 5, Ch.3):
    coherence   do the scenes tell one story? (0-2)
    pacing      does narration fit the duration? (0-2)
    fit         does each image match its scene? (0-2)
    audience    is the level right? (0-2)
─────────────────────────────────────────

Guardrails
─────────────────────────────────────────
  □ per-user daily budget, checked before stage 1
  □ per-project ceiling: max scenes, max regenerations
  □ input moderation on the topic
  □ output moderation on images before display
  □ C2PA on every asset; AI involvement disclosed
  □ degraded path: script-only output if image
    generation is unavailable
─────────────────────────────────────────

That last guardrail is the Chapter 1 principle in practice — a script with no images is a usable partial result; an error page is not.

8. Extensions

In Rough Order of Difficulty
─────────────────────────────────────────
  1. Scene regeneration — re-render one scene without
     touching the rest. Tests whether your stage
     persistence is real.

  2. Style presets — several style bibles the user
     picks from, with a preview.

  3. Timed assembly — cut a video with real audio
     durations rather than estimates, using Ken Burns
     moves over the stills.

  4. Brand consistency — a LoRA for a recurring
     character or house style (Module 3, Chapter 5).

  5. Image-to-video — animate each still (Module 4,
     Chapter 3). Expect to generate several candidates
     per scene.

  6. Translation — regenerate narration in other
     languages, reusing the images unchanged. The
     cheapest large feature on this list.
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

• Stages run sequentially and items within a stage run in parallel; persist each stage's output so a single failure does not re-run the whole pipeline.
• Consistency across images comes from a fixed style bible appended to every prompt plus a project seed offset per scene — style fixed, subject varied.
• Validate the cheapest stage before spending on the expensive ones; a script assertion costs nothing and can save twenty-four media generations.
• Round-trip WER turns narration quality into an automatic assertion, and style-consistency similarity is the rare metric where a high value is the goal.

Concept Check

1. Why use project_seed + scene.index rather than one seed for every scene, or a random seed per scene?
2. Which single ordering decision in the pipeline does the most for cost control, and why?
3. Image generation is down. What does your service return, and what principle from Chapter 1 does that follow?

Next Chapter

→ Chapter 4: Where to Go Next

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index