Evaluation Production And Capstone
LLMs in Production
ML Notes, Module 9 and DL Notes, Module 9 covered general production concerns — deployment, monitoring, drift. LLM systems share these concerns but add several
JrCodex·7 min read
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 2: Evaluating LLMs — Benchmarks, Human Eval, LLM-as-Judge Time to complete: ~25 minutes
Table of Contents
- Extending the ML/DL Notes' Production Discipline
- Cost Management
- Latency Optimization
- Caching Strategies
- Monitoring an LLM System in Production
- Guardrails and Safety Filtering
- A Production LLM Architecture
- Summary & Next Steps
1. Extending the ML/DL Notes' Production Discipline
ML Notes, Module 9 and DL Notes, Module 9 covered general production concerns — deployment, monitoring, drift. LLM systems share these concerns but add several LLM-specific ones this chapter covers: per-token cost, generation latency, and prompt-injection safety.
2. Cost Management
Unlike a classical ML model's near-negligible per-prediction cost (ML Notes, Module 9, Chapter 2), LLM API costs scale directly with tokens processed — both input (prompt) and output (generated) tokens.
def estimate_cost(input_tokens, output_tokens, input_price_per_1k, output_price_per_1k):
return (input_tokens / 1000) * input_price_per_1k + (output_tokens / 1000) * output_price_per_1k
# A RAG query (Module 8) with a LARGE retrieved context is often more expensive
# than the actual GENERATED response, given how INPUT tokens (the context) can dominate
rag_cost = estimate_cost(input_tokens=2000, output_tokens=150, input_price_per_1k=0.01, output_price_per_1k=0.03)
print(f"Estimated cost: ${rag_cost:.4f}")Practical Cost-Reduction Strategies
─────────────────────────────────────────
Use a SMALLER model where sufficient: Module 5, Ch.4's
model landscape —
not every task
needs the LARGEST,
most expensive
model
Reduce CONTEXT size (Module 8): retrieve FEWER,
more PRECISELY
relevant chunks
(Module 8,
Ch.3's
re-ranking)
rather than
padding the
prompt with
excess context
Cache repeated queries (Section 4)
(Section 4):
Batch requests where processing
possible: MULTIPLE
requests
together
can be
more
cost-
efficient
than
one at
a time,
for
self-
hosted
models
(DL
Notes,
Module
9, Ch.1)
─────────────────────────────────────────
3. Latency Optimization
Where Latency Comes From
─────────────────────────────────────────
Time to first token: how long before the model starts
producing ANY output — dominated
by PROMPT processing (longer
context = SLOWER, given
self-attention's quadratic
scaling, DL Notes Module 6 Ch.5)
Time per output token: how fast SUBSEQUENT tokens are
generated — the autoregressive
loop (Module 5, Ch.1) means
THIS accumulates linearly with
RESPONSE length
─────────────────────────────────────────
# Streaming responses — showing tokens AS they're generated, rather than
# waiting for the FULL response — directly improves PERCEIVED latency,
# even though TOTAL generation time is unchanged
def stream_response(client, prompt):
stream = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
stream=True, # returns tokens INCREMENTALLY, as they're generated
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True) # DISPLAY immediatelyKV Caching, Revisited (Module 5, Chapter 1's Preview)
─────────────────────────────────────────
Production inference systems cache each layer's key/value
vectors (DL Notes, Module 6, Ch.1) from PREVIOUS generation
steps, so generating the NEXT token only requires computing
attention for the SINGLE newest token, not re-processing the
ENTIRE sequence from scratch — this is what makes practical,
responsive LLM serving feasible at all.
─────────────────────────────────────────
4. Caching Strategies
import hashlib
import json
class LLMResponseCache:
def __init__(self):
self.cache = {}
def _make_key(self, prompt, model, temperature):
key_data = json.dumps({"prompt": prompt, "model": model, "temperature": temperature}, sort_keys=True)
return hashlib.sha256(key_data.encode()).hexdigest()
def get_or_generate(self, client, prompt, model="gpt-4", temperature=0.0):
key = self._make_key(prompt, model, temperature)
if key in self.cache:
return self.cache[key] # AVOID a redundant, costly API call entirely
response = client.chat.completions.create(
model=model, messages=[{"role": "user", "content": prompt}], temperature=temperature,
)
result = response.choices[0].message.content
self.cache[key] = result
return resultWhy Caching Requires temperature=0 (or Careful Handling)
─────────────────────────────────────────
Module 5, Chapter 1, Section 5's SAMPLING-based decoding
strategies (temperature, top-k/top-p) are, BY DESIGN,
non-deterministic — the SAME prompt can produce DIFFERENT
outputs on different calls. Caching works cleanly ONLY for
DETERMINISTIC generation (temperature=0, greedy decoding) —
caching NON-deterministic outputs means later "cache hits"
return a RANDOM PAST response, not necessarily representative
of what a FRESH call would produce.
─────────────────────────────────────────
5. Monitoring an LLM System in Production
Directly extending ML Notes, Module 9, Chapter 3's monitoring/drift discussion with LLM-specific signals.
What to Monitor, Specifically for LLM Systems
─────────────────────────────────────────
Token usage & cost TRACKING actual spend against
(Section 2): budget, per feature/user
Latency percentiles (ML Notes' general
(p50/p95/p99): monitoring discipline,
applied to Section 3's
latency concerns)
Output quality drift periodic LLM-as-judge
(Chapter 2, Section 5): (Chapter 2, Sec.5)
sampling of PRODUCTION
outputs, watching
for QUALITY
degradation over time
RAG-specific: retrieval
precision/recall
(Chapter 2,
Sec.7) on a
SAMPLE of
production queries
Refusal/safety RATE of the
trigger rate: model
declining
to answer
(Module 6,
Ch.3's
alignment)
— unusual
SPIKES may
indicate a
prompt/
usage
pattern
shift
worth
investigating
─────────────────────────────────────────
6. Guardrails and Safety Filtering
Directly connecting to Module 8, Chapter 4, Section 6's agent safety risks and Module 6, Chapter 3's alignment discussion — production systems typically add explicit input/output filtering LAYERS, rather than relying entirely on the model's own alignment.
def apply_input_guardrails(user_input):
"""A simplified illustration — real systems use dedicated moderation APIs/models"""
blocked_patterns = ["ignore previous instructions", "disregard your guidelines"] # basic
# prompt-injection
# detection
# (Module 8, Ch.4)
for pattern in blocked_patterns:
if pattern.lower() in user_input.lower():
return False, "Input flagged for review"
return True, None
def apply_output_guardrails(client, generated_response):
"""Using a moderation-specific model/API to check GENERATED output before showing it to a user"""
moderation_result = client.moderations.create(input=generated_response)
if moderation_result.results[0].flagged:
return "I cannot provide that response."
return generated_responseDefense in Depth
─────────────────────────────────────────
Relying on Module 6, Chapter 3's RLHF alignment ALONE is
insufficient for production systems — explicit input
filtering (blocking known prompt-injection patterns) and
output filtering (moderation checks BEFORE showing a
response to a user) provide an ADDITIONAL, independent
safety layer, catching cases where the MODEL's own alignment
training didn't generalize perfectly.
─────────────────────────────────────────
7. A Production LLM Architecture
Putting the Whole Curriculum Together
─────────────────────────────────────────
User request
│
▼
Input guardrails (Section 6)
│
▼
Cache check (Section 4) — return CACHED response if available
│
▼
RAG retrieval (Module 8, Ch.1-3) — if the system needs
grounded knowledge
│
▼
LLM generation (Module 5) — possibly via an AGENT loop
(Module 8, Ch.4-5) for multi-step tasks
│
▼
Output guardrails (Section 6)
│
▼
Response streamed to user (Section 3)
│
▼
Logging & monitoring (Section 5) — feeding back into ONGOING
evaluation (Chapter 2)
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- LLM API costs scale with tokens processed, requiring active management through smaller models where sufficient, reduced context size, and caching.
- Latency is dominated by prompt processing time (quadratic with context length) and per-token generation time; streaming and KV caching are the standard mitigations.
- Response caching requires deterministic generation (temperature=0), since caching non-deterministic sampled outputs produces misleading "cache hits."
- Production LLM systems need explicit input/output guardrails as a defense-in-depth layer, rather than relying entirely on the model's own RLHF-based alignment.
Concept Check
- Why does a RAG query with a large retrieved context often cost more than the generated response itself?
- Why must response caching be paired with deterministic (temperature=0) generation to work correctly?
- Why are explicit guardrails necessary in production, even for a model that has already been aligned via RLHF?
Next Chapter
→ Chapter 4: Capstone — End-to-End RAG Application
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index