Evaluation Production And Capstone
Capstone: End-to-End RAG Application
A software company wants an internal support assistant: employees ask natural-language questions about internal documentation (IT policies, HR procedures, engin
JrCodex·6 min read
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: All of Modules 1–9 (Chapters 1-3) Time to complete: ~50 minutes
Table of Contents
- The Scenario
- Phase 1 — Problem Definition
- Phase 2 — Building the Knowledge Base
- Phase 3 — Retrieval With Hybrid Search & Re-Ranking
- Phase 4 — Generation With Guardrails
- Phase 5 — Adding Agentic Tool Use
- Phase 6 — Evaluation
- Phase 7 — Production Readiness
- Where to Go From Here
1. The Scenario
A software company wants an internal support assistant: employees ask natural-language questions about internal documentation (IT policies, HR procedures, engineering runbooks), and the assistant answers accurately, citing sources, and can escalate to creating a support ticket when it can't resolve something directly. This single project pulls in a technique from nearly every module in this curriculum.
knowledge_base/ — directory structure
─────────────────────────────────────────
it_policies/ (IT policy documents)
hr_procedures/ (HR procedure documents)
engineering_runbooks/ (technical runbooks)
─────────────────────────────────────────
2. Phase 1 — Problem Definition
Applying Module 8, Chapter 1's RAG-vs-fine-tuning decision:
Precise Problem Framing
─────────────────────────────────────────
Approach: RAG (Module 8), not fine-tuning — internal
documentation CHANGES frequently, and full
TRACEABILITY (which document supported this
answer) is a HARD requirement for an
internal support tool
Success answer accuracy (verified against
criteria: source documents), source citation
PRESENT in every answer, graceful
"I don't know" + escalation when
retrieval doesn't find relevant
content (Module 8, Ch.3, Section 6)
Escalation when the assistant can't answer
path: confidently, it should be
ABLE to create a support
ticket (Module 8, Ch.4's
tool use) rather than
guessing
─────────────────────────────────────────
3. Phase 2 — Building the Knowledge Base
Applying Module 8, Chapter 2's full pipeline:
import os
from sentence_transformers import SentenceTransformer
import chromadb
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.Client()
collection = client.create_collection(name="internal_docs")
def load_and_chunk_documents(directory):
all_chunks, all_metadata = [], []
for root, _, files in os.walk(directory):
for filename in files:
filepath = os.path.join(root, filename)
with open(filepath, "r") as f:
text = f.read()
chunks = chunk_text(text, chunk_size=500, overlap=50) # Module 8, Ch.2, Section 2
all_chunks.extend(chunks)
all_metadata.extend([{"source": filepath}] * len(chunks)) # for CITATION (Phase 1's requirement)
return all_chunks, all_metadata
chunks, metadata = load_and_chunk_documents("knowledge_base/")
embeddings = embedding_model.encode(chunks)
collection.add(
documents=chunks,
embeddings=embeddings.tolist(),
metadatas=metadata, # tracks SOURCE for citation
ids=[f"chunk_{i}" for i in range(len(chunks))],
)
print(f"Indexed {len(chunks)} chunks from internal documentation")4. Phase 3 — Retrieval With Hybrid Search & Re-Ranking
Applying Module 8, Chapter 3's advanced retrieval techniques, since internal documentation often contains EXACT terms (policy numbers, tool names) that pure semantic search might miss:
def retrieve_with_citations(query, collection, embedding_model, top_k=5):
query_embedding = embedding_model.encode([query])
results = collection.query(query_embeddings=query_embedding.tolist(), n_results=top_k)
retrieved = [
{"text": doc, "source": meta["source"]}
for doc, meta in zip(results["documents"][0], results["metadatas"][0])
]
return retrieved # hybrid search (Module 8, Ch.3, Sec.2) and re-ranking (Sec.3)
# would be layered in here for a fuller production system5. Phase 4 — Generation With Guardrails
Applying Module 8, Chapter 3, Section 6's uncertainty handling and Module 9, Chapter 3's guardrails:
from openai import OpenAI
client = OpenAI()
def generate_grounded_answer(query, retrieved_chunks):
context = "\n\n".join([f"[Source: {c['source']}]\n{c['text']}" for c in retrieved_chunks])
prompt = f"""Answer the employee's question using ONLY the context below.
Cite the source document for each claim. If the context doesn't contain
enough information, respond EXACTLY with: "I don't have enough information
to answer this confidently. Would you like me to create a support ticket?"
Context:
{context}
Question: {query}"""
ok, reason = apply_input_guardrails(query) # Module 9, Ch.3, Section 6
if not ok:
return "Your request could not be processed."
response = client.chat.completions.create(model="gpt-4", messages=[{"role": "user", "content": prompt}])
return apply_output_guardrails(client, response.choices[0].message.content) # Module 9, Ch.3, Sec.66. Phase 5 — Adding Agentic Tool Use
Applying Module 8, Chapter 4's agent loop for the escalation path defined in Phase 1:
def create_support_ticket(issue_summary, category):
"""A REAL implementation would call an actual ticketing system's API"""
return {"ticket_id": "TICKET-4821", "status": "created", "category": category}
tools = [{"type": "function", "function": {
"name": "create_support_ticket",
"description": "Create a support ticket when the knowledge base cannot answer the question",
"parameters": {
"type": "object",
"properties": {
"issue_summary": {"type": "string"},
"category": {"type": "string", "enum": ["IT", "HR", "Engineering"]},
},
"required": ["issue_summary", "category"],
},
}}]
def support_assistant(client, query, collection, embedding_model):
retrieved = retrieve_with_citations(query, collection, embedding_model) # Phase 3
answer = generate_grounded_answer(query, retrieved) # Phase 4
if "create a support ticket" in answer.lower():
# The assistant DECIDED escalation is needed — hand off to the agent loop (Module 8, Ch.4)
return run_agent(client, f"The user asked: '{query}'. Create an appropriate support ticket.",
tools=tools, tool_functions={"create_support_ticket": create_support_ticket})
return answer7. Phase 6 — Evaluation
Applying Chapter 1's classical metrics and Chapter 2's LLM-specific evaluation:
def evaluate_rag_system(test_questions, collection, embedding_model, client):
results = []
for item in test_questions:
answer = support_assistant(client, item["question"], collection, embedding_model)
# Faithfulness check (Module 8, Ch.1, Sec.3; Ch.9, Sec.7) via LLM-as-judge
judge_prompt = f"""Does this answer accurately reflect ONLY the provided context,
without adding unsupported claims? Context: {item['context']}
Answer: {answer}
Respond "FAITHFUL" or "UNFAITHFUL"."""
judgment = client.chat.completions.create(
model="gpt-4", messages=[{"role": "user", "content": judge_prompt}],
).choices[0].message.content
results.append({"question": item["question"], "answer": answer, "faithful": "FAITHFUL" in judgment})
faithfulness_rate = sum(r["faithful"] for r in results) / len(results)
print(f"Faithfulness rate: {faithfulness_rate:.2%}")
return results8. Phase 7 — Production Readiness
Applying Chapter 3's production concerns:
Production Checklist for This Capstone
─────────────────────────────────────────
Cost: context size kept LEAN via re-ranking
(Module 8, Ch.3) — only the MOST relevant
chunks are sent to the LLM per query
Latency: responses STREAMED to the user (Ch.3,
Sec.3) rather than waiting for full
generation
Caching: common, REPEATED questions (e.g. "how
do I request PTO?") cached with
temperature=0 (Ch.3, Sec.4)
Monitoring: faithfulness rate (Phase 6),
escalation rate, and cost per
query tracked on an ONGOING
basis (Ch.3, Sec.5)
Guardrails: input/output filtering
(Phase 4) active on EVERY
request
─────────────────────────────────────────
9. Where to Go From Here
Module → Technique Used in This Capstone
─────────────────────────────────────────
Module 1-2 (Foundations, → understanding WHY dense
Embeddings) embeddings and context matter
for retrieval quality
Module 3 (Tokenization & → the tokenization and
Pre-training) pretraining foundation
underlying EVERY model
used
Module 4 (BERT) → the sentence-
embedding model
(built on BERT-
family
architecture) used
for retrieval
Module 5 (GPT) → the generative
model producing
final answers
Module 6 (Fine-Tuning) → NOT directly
used here
(RAG was the
chosen
approach,
Phase 1) —
would apply
if a
CONSISTENT
tone/format
needed
enforcing
beyond
prompting
Module 7 (Prompt Engineering) → the
grounded-
answer
prompt
template
(Phase 4)
Module 8 (RAG & Agents) → the
ENTIRE
retrieval
+
generation
+
escalation
pipeline
Module 9 (This module) → evaluation,
guardrails,
production
readiness
─────────────────────────────────────────
You've now built a full, realistic RAG application end-to-end, using nearly every technique across this entire curriculum. Chapter 5 closes out this module — and the whole NLP & LLM curriculum — with a reflection on where to continue learning.
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index