Building And Shipping Gen AI
Architecture of a Gen AI Feature
What Makes This Different from a Normal Endpoint
JrCodex·7 min read
Jr Codex Generative AI Notes
Level: Advanced Prerequisites: Module 6, Chapter 4; NLP Notes, Module 9, Chapter 3 Time to complete: ~25 minutes
Table of Contents
- Four Properties That Change the Design
- Choosing Sync, Streaming, or Async
- The Async Job Pattern
- Designing for Non-Determinism
- Failure and Degradation
- Observability
- Summary & Next Steps
1. Four Properties That Change the Design
What Makes This Different from a Normal Endpoint
─────────────────────────────────────────
SLOW 1-60+ seconds, not 50ms. Long enough
that a synchronous HTTP request is often
the wrong shape entirely.
EXPENSIVE real marginal cost per call. A retry
loop bug is a bill, not just load.
UNRELIABLE non-deterministic output, upstream rate
limits, and content filters that fire
unpredictably.
UNVERIFIABLE no exact expected result to assert
against (Module 5).
─────────────────────────────────────────
Every pattern in this chapter follows from one of those four.
2. Choosing Sync, Streaming, or Async
The Decision Rule
─────────────────────────────────────────
Expected latency < 1s ──► SYNCHRONOUS
classification, extraction, short completions
1-30s AND text output ──► STREAMING
chat, drafting, summaries. The user sees progress
immediately, so the wait is tolerable.
> 30s, OR non-text output ──► ASYNCHRONOUS
images, video, audio, batch jobs. There is no
partial result to show, so hold nothing open.
Any output the user might ──► ASYNCHRONOUS
return to later regardless of duration
─────────────────────────────────────────
The Mistake to Avoid
─────────────────────────────────────────
Holding an HTTP request open for a 45-second image
generation.
It breaks on every proxy, load balancer and mobile
network timeout in the path; it pins a server
worker for 45 seconds; and if the connection drops,
the user has paid for an image nobody can retrieve.
That last consequence is the decisive one.
─────────────────────────────────────────
3. The Async Job Pattern
The Shape
─────────────────────────────────────────
POST /generate ──► validate, enqueue, return a
job_id immediately (202)
WORKER ──► pulls the job, calls the model,
writes the result to storage,
updates job status
GET /jobs/{id} ──► status: queued | running |
succeeded | failed, plus the
result URL when done
or a WEBSOCKET / SSE channel pushing status changes
─────────────────────────────────────────
from enum import Enum
from dataclasses import dataclass, field
import uuid, time
class Status(str, Enum):
QUEUED = "queued"; RUNNING = "running"
SUCCEEDED = "succeeded"; FAILED = "failed"
@dataclass
class Job:
id: str = field(default_factory=lambda: str(uuid.uuid4()))
status: Status = Status.QUEUED
request_hash: str = "" # for caching and idempotency (Chapter 2)
result_url: str | None = None
error: str | None = None
cost_cents: float = 0.0 # ATTRIBUTE COST PER JOB — you cannot control
attempts: int = 0 # what you do not measure
created_at: float = field(default_factory=time.time)
def submit(store, queue, user_id, params):
h = hash_request(params)
if cached := store.find_succeeded_by_hash(h): # IDENTICAL request already done
return cached # return it; do not pay twice
job = Job(request_hash=h)
store.save(job, user_id)
queue.push(job.id)
return jobThree Things This Buys You
─────────────────────────────────────────
RESILIENCE a dropped connection costs nothing —
the job continues and the result is
retrievable
BACKPRESSURE the queue absorbs traffic spikes
instead of hammering a rate-limited
upstream
ATTRIBUTION cost, latency and attempts are recorded
per job, which is what makes Chapter 2
possible at all
─────────────────────────────────────────
4. Designing for Non-Determinism
Show the Variance Instead of Hiding It
─────────────────────────────────────────
Users accept variation when the interface implies
it. They report a BUG when it implies determinism.
DO generate 4 variants and let the user pick
DO offer a visible "regenerate" affordance
DO show and let users reuse the seed (Module 3)
DO keep the history so an earlier result is
recoverable
DON'T present a single output as THE answer
DON'T silently replace a previous result
─────────────────────────────────────────
The Regeneration Rate Is a Product Metric
─────────────────────────────────────────
How often users hit regenerate is the cheapest,
most honest quality signal you have — Module 5's
rung 5.
Rising regeneration rate after a deploy is a
regression, whatever your offline metrics say.
Instrument it on day one. Retrofitting it means
losing the baseline you needed.
─────────────────────────────────────────
5. Failure and Degradation
The Failure Modes, and the Right Response
─────────────────────────────────────────
RATE LIMITED (429) retry with backoff and
jitter; then fall back to a
secondary provider
CONTENT FILTERED do NOT retry — it will fire
again. Explain to the user
(Module 6, Chapter 4)
TIMEOUT retry once with a smaller
request (fewer steps, shorter
max_tokens)
MALFORMED OUTPUT repair loop with the
validation error fed back
(Module 2, Chapter 3), capped
at 2-3 attempts
PROVIDER DOWN circuit-break, and serve the
degraded path
─────────────────────────────────────────
Define the Degraded Path Before You Need It
─────────────────────────────────────────
For EVERY generative feature, decide in advance what
happens when generation is unavailable:
- a smaller/cheaper model
- a different provider
- a cached or template result
- the feature hidden, with the rest of the product
working normally
The unacceptable answer is an error page for a
feature that was optional to begin with. Generative
features should almost always fail SOFT.
─────────────────────────────────────────
class CircuitBreaker:
def __init__(self, threshold=5, reset_after=60):
self.failures, self.opened_at = 0, None
self.threshold, self.reset_after = threshold, reset_after
def allows(self):
if self.opened_at is None:
return True
if time.time() - self.opened_at > self.reset_after:
self.opened_at, self.failures = None, 0 # HALF-OPEN: let one through to test
return True
return False # fail fast — do not queue behind a
# dead provider and burn timeouts
def record(self, ok: bool):
if ok:
self.failures, self.opened_at = 0, None
else:
self.failures += 1
if self.failures >= self.threshold:
self.opened_at = time.time()6. Observability
Log Per Request — Non-Negotiable
─────────────────────────────────────────
request_id, user_id, feature
model name AND exact version
prompt hash (not the raw prompt, unless retention
policy permits it)
input tokens, output tokens, cost
latency, split into queue time and model time
finish_reason, filter results
attempts, and the final outcome
cache hit or miss
─────────────────────────────────────────
The Four Dashboards Worth Having
─────────────────────────────────────────
COST per feature, per user, per day. With an
alert on daily spend, because the failure
mode here is financial and silent.
LATENCY p50 and p95, queue time separated from
model time — they have different fixes.
QUALITY assertion pass rate, regeneration rate,
judge scores on a scheduled run.
ERRORS by TYPE, not just count. Rate limits,
filters, timeouts and malformed output are
four different problems and one aggregate
error rate hides which you have.
─────────────────────────────────────────
The Alert Most Teams Miss
─────────────────────────────────────────
Alert on COST PER USER PER DAY exceeding a
threshold.
A prompt-injected loop, a retry bug, or one
enthusiastic user can produce a bill an order of
magnitude above plan before anything else fires —
because nothing is DOWN. Latency is fine, errors
are zero, and the system is working perfectly.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Generative features are slow, expensive, unreliable and unverifiable — every architectural decision here follows from one of those four properties.
- Choose sync under a second, streaming for 1–30s text, and async for everything else; holding an HTTP connection open for a 45-second generation loses results users have already paid for.
- Design interfaces that expose variance — variants, regenerate, visible seeds, history — and treat regeneration rate as a first-class quality metric.
- Define the degraded path before launch: generative features should fail soft, and cost per user per day deserves an alert because that failure is silent.
Concept Check
- Why is async the right shape for image generation even when it usually completes in 20 seconds?
- Which failure mode should never be retried, and what should happen instead?
- Your dashboards show normal latency, zero errors, and a 40x cost spike. What is the likely cause and which alert should have caught it?
Next Chapter
→ Chapter 2: Cost, Latency & Scaling
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index