Audio Video And Multimodal
Multimodal Models
"Multimodal" covers three genuinely different capabilities that are frequently conflated.
JrCodex·6 min read
Jr Codex Generative AI Notes
Level: Intermediate Prerequisites: Chapter 3: Video Generation; NLP Notes, Module 5, Chapter 4 Time to complete: ~20 minutes
Table of Contents
- Input, Output, and Any-to-Any
- Vision as Input, in Practice
- What Vision Models Are Good and Bad At
- Unified Any-to-Any Models
- Pipeline or Unified Model?
- Summary & Next Steps
1. Input, Output, and Any-to-Any
"Multimodal" covers three genuinely different capabilities that are frequently conflated.
The Three Meanings
─────────────────────────────────────────
MULTIMODAL INPUT many modalities in, TEXT out
"describe this chart"
Mature and widely available.
MULTIMODAL OUTPUT text in, another modality out
Modules 3 and 4 — separate
specialist models.
ANY-TO-ANY one model, any modality in and
out, in a single conversation.
Emerging.
─────────────────────────────────────────
The NLP Notes covered multimodal input — images become visual tokens projected into the text embedding space and processed by the same decoder. This chapter takes that as given and focuses on what it means to use, and on the third category.
2. Vision as Input, in Practice
import base64
from openai import OpenAI
client = OpenAI()
def encode(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "List every line item and its amount from this receipt."},
{"type": "image_url", "image_url": {
"url": f"data:image/jpeg;base64,{encode('receipt.jpg')}",
"detail": "high", # "high" tiles the image for more tokens and more detail
}},
],
}],
response_format=LineItems, # combine with Module 2, Chapter 3 — structure the output
)Images Cost Tokens
─────────────────────────────────────────
An image is converted into visual tokens and billed
like text. A high-detail image can cost as much as
several pages of prose.
detail="low" one small fixed cost. Enough for
"what is this a picture of?"
detail="high" the image is tiled and each tile
encoded. Necessary for reading small
text, tables, or fine detail.
Choose deliberately. Sending 200 product photos at
high detail is a real bill.
─────────────────────────────────────────
The pairing worth remembering: vision input plus structured output (Module 2, Chapter 3) is the single most useful multimodal pattern in production. It turns any document, screenshot, or photograph into schema-validated data.
3. What Vision Models Are Good and Bad At
Reliable
─────────────────────────────────────────
Description and classification of scenes
Reading clear printed text (OCR-quality and better)
Extracting fields from documents and forms
Explaining charts, diagrams and UI screenshots
Comparing two images qualitatively
Describing images for accessibility
Unreliable
─────────────────────────────────────────
COUNTING beyond a handful of objects
PRECISE SPATIAL relations ("is A left of B?")
Reading dense handwriting
Exact colour or measurement values
Anything requiring pixel-level precision
─────────────────────────────────────────
The Reason for the Split
─────────────────────────────────────────
An image is compressed into a few hundred visual
tokens (Module 1, Chapter 3's compression, again).
That is ample for SEMANTICS and lossy for
ENUMERATION and GEOMETRY.
Rule of thumb: if the answer requires looking at
every object individually, use a purpose-built
detection model (DL Notes, Module 4, Chapter 4)
and let the language model reason over ITS output.
─────────────────────────────────────────
4. Unified Any-to-Any Models
The frontier: one model that natively takes and produces multiple modalities, rather than orchestrating specialists.
Pipeline vs Unified
─────────────────────────────────────────
PIPELINE (today's default)
speech ──► Whisper ──► text ──► LLM ──► text
──► TTS ──► speech
Each arrow is a separate model, and each hop
DISCARDS what text cannot carry: tone, hesitation,
sarcasm, emphasis, background sound.
UNIFIED
speech ──► ONE MODEL ──► speech
Audio in and audio out as tokens in a single
model. Tone and timing survive because they were
never transcribed away.
─────────────────────────────────────────
What Unification Actually Buys
─────────────────────────────────────────
LATENCY one forward pass instead of three
sequential models — the difference
between a 2-second and a 300ms reply,
which is the difference between a
transaction and a conversation
NUANCE it can hear that you sounded uncertain
and respond to it
INTERRUPTION real-time duplex conversation becomes
possible at all
─────────────────────────────────────────
The Honest Caveats
─────────────────────────────────────────
- A specialist still beats the generalist on its own
task. Whisper transcribes better; a dedicated
diffusion model renders better images.
- Unified models are harder to debug: you cannot
inspect the intermediate text, because there
isn't one.
- Cost and availability are less favourable, and the
controls (Module 3's seeds, guidance, ControlNet)
are largely absent.
─────────────────────────────────────────
5. Pipeline or Unified Model?
Decision Guide
─────────────────────────────────────────
Need to LOG or AUDIT the intermediate text?
──► PIPELINE. There is no transcript otherwise.
Need sub-second conversational latency?
──► UNIFIED. A pipeline cannot get there.
Need maximum quality in one modality?
──► PIPELINE with a specialist for that step.
Need fine control — seeds, guidance, ControlNet?
──► PIPELINE. Unified models expose little.
Does tone, emotion or timing carry meaning?
──► UNIFIED, or a pipeline that explicitly
extracts and passes those features along.
─────────────────────────────────────────
The Practical Default in 2026
─────────────────────────────────────────
Build the PIPELINE first.
It is cheaper, debuggable, swappable component by
component, and every hop is inspectable — which
matters enormously when something goes wrong, and
which Module 7 depends on.
Move to a unified model when a specific requirement
— usually latency or vocal nuance — makes the
pipeline's losses unacceptable.
─────────────────────────────────────────
6. Summary & Next Steps
Key Takeaways
- "Multimodal" means three different things: many modalities in, another modality out, or one model doing both — only the first is fully mature.
- Vision input plus structured output is the highest-value multimodal pattern in production, turning documents and screenshots into validated data.
- Vision models are strong on semantics and weak on counting and geometry, because an image is compressed to a few hundred tokens.
- Pipelines discard everything text cannot carry; unified models preserve tone and cut latency but lose specialist quality, fine control, and inspectability.
Module 4 Complete — What's Next
You have now seen every major modality and the trade-offs each one imposes. One question has been deferred throughout: how do you tell whether any of this output is actually good? Module 5 takes that on directly.
Concept Check
- Why does a speech-to-speech pipeline lose information that a unified model retains, and give a case where that loss matters?
- Your model miscounts the items on a warehouse shelf. Explain why, and describe the architecture you would use instead.
- Under what specific circumstance would you choose a pipeline even though a unified model is available and faster?
Next Module
→ Module 5: Evaluating Generative Output
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index