Building And Shipping Gen AI
Cost, Latency & Scaling
Optimisation effort is routinely spent in the wrong order. This is the correct one.
JrCodex·7 min read
Jr Codex Generative AI Notes
Level: Advanced Prerequisites: Chapter 1: Architecture of a Gen AI Feature Time to complete: ~20 minutes
Table of Contents
- The Levers, in Order of Impact
- Not Generating at All
- Model Routing
- Prompt and Output Reduction
- Latency Is a Separate Problem
- Budgets and Guardrails
- Summary & Next Steps
1. The Levers, in Order of Impact
Optimisation effort is routinely spent in the wrong order. This is the correct one.
Ordered by Typical Savings
─────────────────────────────────────────
1. DON'T GENERATE cache, dedupe, precompute
──► up to 100% on hits
2. USE A SMALLER MODEL routing by task difficulty
──► 80-95%
3. SHRINK THE PROMPT retrieve instead of pasting,
compact history
──► 50-90%
4. SHRINK THE OUTPUT cap max_tokens, request
concise formats
──► 20-50% of the expensive
half
5. BATCH provider batch APIs for
non-urgent work
──► ~50%
6. TUNE PARAMETERS fewer diffusion steps,
distilled models
──► 20-40%
─────────────────────────────────────────
The Common Mistake
─────────────────────────────────────────
Teams start at lever 6 — shaving diffusion steps,
trimming prompt wording — because it feels like
engineering.
Lever 1 usually saves more than 3 through 6
combined, and takes an afternoon.
─────────────────────────────────────────
2. Not Generating at All
Three Kinds of Cache
─────────────────────────────────────────
EXACT MATCH
Hash the full request (prompt + params + model).
Identical request ──► stored result.
Trivially correct. Hit rates are surprisingly
high on real traffic — users repeat themselves,
and so do retries.
SEMANTIC MATCH
Embed the request; return a cached result if
similarity exceeds a threshold.
Much higher hit rate, and it WILL eventually
return a subtly wrong answer. Use only where
approximate is acceptable, and set the threshold
conservatively (≥ 0.95).
PROVIDER PROMPT CACHE
The long-prefix mechanism from Module 2,
Chapter 4. Free, once the prompt is ordered
static-first.
─────────────────────────────────────────
import hashlib, json
def request_key(model: str, messages: list, **params) -> str:
payload = json.dumps({"model": model, "messages": messages, "params": params},
sort_keys=True) # ORDER-STABLE — or the hash is useless
return hashlib.sha256(payload.encode()).hexdigest()
def generate(cache, client, model, messages, ttl=86_400, **params):
key = request_key(model, messages, **params)
if hit := cache.get(key):
return hit, True # report the hit — you need the rate
result = client.chat.completions.create(model=model, messages=messages, **params)
cache.set(key, result, ttl=ttl)
return result, FalseAlso Under "Don't Generate"
─────────────────────────────────────────
PRECOMPUTE generate popular content offline, on
the cheap batch tier
DEDUPE collapse concurrent identical requests
into one upstream call, and fan the
result out
SHORT-CIRCUIT many "generative" features have a
deterministic answer for common inputs.
Check that first. An empty input does
not need a model call to be rejected.
─────────────────────────────────────────
3. Model Routing
The highest-leverage lever after caching, and the most under-used.
The Premise
─────────────────────────────────────────
Most production traffic is EASY. Classification,
extraction, short rewrites, routing decisions — a
small model handles all of it at a fraction of the
cost, often faster.
Sending everything to a frontier model is paying
frontier prices for a majority of trivial requests.
─────────────────────────────────────────
def route(task_type: str, input_tokens: int) -> str:
if task_type in {"classify", "extract", "route", "moderate"}:
return "small" # structured output makes quality VERIFIABLE here
if task_type == "summarize" and input_tokens < 4_000:
return "small"
if task_type in {"reason", "code", "analyze"}:
return "large"
return "medium"
def generate_with_escalation(clients, task_type, messages, validator):
tier = route(task_type, count_tokens(messages))
result = clients[tier].create(messages=messages)
if not validator(result) and tier != "large":
return clients["large"].create(messages=messages) # ESCALATE only on FAILURE
return resultWhy Escalation-on-Failure Works
─────────────────────────────────────────
Try cheap first, verify, escalate only if the check
fails.
If the small model succeeds 85% of the time and
costs 5% as much, the blended cost is roughly
0.85 x 0.05 + 0.15 x 1.05 ≈ 20% of always-large.
The prerequisite is a VALIDATOR — which is exactly
what Module 2, Chapter 3's schemas and Module 5's
assertions give you. Cost control and evaluation
turn out to be the same investment.
─────────────────────────────────────────
4. Prompt and Output Reduction
Input Side
─────────────────────────────────────────
RETRIEVE, DON'T PASTE 5 relevant chunks instead
of 200 pages. Cheaper AND
more accurate (Module 2,
Chapter 4).
COMPACT HISTORY summarise old turns rather
than resending them.
TRIM FEW-SHOT EXAMPLES measure whether 8 examples
beat 3. Usually they do
not.
DROP UNUSED TOOLS every tool definition is
input tokens on EVERY call.
─────────────────────────────────────────
Output Side
─────────────────────────────────────────
Output typically costs several times more than
input per token, so it is worth disproportionate
attention.
SET max_tokens always, on every call
ASK FOR CONCISION "answer in under 100 words"
genuinely works
PREFER STRUCTURE JSON with short keys beats
prose that must be parsed,
in both tokens and
reliability
DON'T ASK FOR REASONING you will discard. Chain-of-
thought is expensive; use it
where it improves the
answer, not by default.
─────────────────────────────────────────
5. Latency Is a Separate Problem
Cost and latency optimisations overlap but are not the same, and conflating them wastes effort.
Where Latency Actually Comes From
─────────────────────────────────────────
QUEUE TIME your own backlog. Fix with worker
capacity, not with model choice.
TIME TO FIRST dominated by INPUT length. A long
TOKEN prompt is slow before it is
expensive. Prompt caching helps
here more than anywhere.
GENERATION proportional to OUTPUT length,
because generation is serial
(Module 1, Chapter 2).
Different causes, different fixes. Always separate
queue time from model time in your metrics
(Chapter 1).
─────────────────────────────────────────
Perceived vs Actual
─────────────────────────────────────────
STREAM shows progress; time-to-first-
token becomes the number that
matters, not total time
PARALLELISE independent calls concurrently
— the whole point of map-reduce
(Module 2, Chapter 4)
SPECULATE start likely work before the
user asks. Costs money on
wrong guesses; measure the hit
rate before committing.
SHOW REAL PROGRESS for async jobs, a genuine
step count beats a spinner.
Diffusion gives you step
progress for free — use it.
─────────────────────────────────────────
6. Budgets and Guardrails
The Four Limits Every System Needs
─────────────────────────────────────────
PER REQUEST max_tokens, max steps, max retries,
max input size. Bounds the worst
single call.
PER USER daily and monthly caps. The single
most important control — it bounds
abuse, injection loops, and
enthusiasm alike.
PER FEATURE a spend ceiling that DISABLES the
feature rather than the account,
falling back to the degraded path
(Chapter 1).
GLOBAL a total daily kill switch. Rarely
fires. Ends the incident when it does.
─────────────────────────────────────────
class Budget:
def __init__(self, store, daily_cents=500):
self.store, self.daily_cents = store, daily_cents
def check(self, user_id, estimated_cents):
spent = self.store.spend_today(user_id)
if spent + estimated_cents > self.daily_cents:
raise BudgetExceeded(f"daily limit reached ({spent/100:.2f} used)")
return True
def record(self, user_id, actual_cents):
self.store.add_spend(user_id, actual_cents) # record ACTUAL, not the estimate —
# estimates drift and the gap compoundsEstimate Before, Record After
─────────────────────────────────────────
Check the budget with an ESTIMATE before the call,
so you can refuse cheaply.
Record the ACTUAL cost from the usage field after,
so the ledger is truthful.
A system that budgets on estimates alone drifts,
and always in the direction of overspending.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Optimise in order: not generating at all, then routing to a smaller model, then shrinking prompt and output — parameter tuning is last and least.
- Escalation-on-failure routing needs a validator, which means cost control and evaluation are the same investment.
- Latency has three distinct sources — queue time, input length, output length — with three different fixes; measure them separately.
- Every system needs per-request, per-user, per-feature and global limits; budget against an estimate before the call and record the actual cost after.
Concept Check
- Why does caching typically save more than every prompt-shortening technique combined?
- Work through why escalation-on-failure lands near 20% of always-large cost, and state what it depends on.
- A feature has acceptable total latency but users complain it feels slow. Which component would you look at, and which fix applies?
Next Chapter
→ Chapter 3: Capstone — A Multimodal Content Studio
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index