Rag And Agents
Building a RAG Pipeline
Documents are typically too long to embed as a single unit meaningfully (a whole book vs. a single sentence would produce vastly different, hard-to-compare embe
JrCodex·6 min read
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 1: Why RAG — The Limits of Parametric Knowledge Time to complete: ~30 minutes
Table of Contents
- The Four Stages of a RAG Pipeline
- Stage 1: Chunking Documents
- Stage 2: Generating Embeddings
- Stage 3: Storing Embeddings in a Vector Database
- Stage 4: Retrieval via Similarity Search
- Putting It All Together
- Summary & Next Steps
1. The Four Stages of a RAG Pipeline
The Complete Pipeline
─────────────────────────────────────────
1. CHUNKING: split source documents into manageable
pieces (Section 2)
2. EMBEDDING: convert each chunk into a dense
vector representation (Section 3)
3. STORAGE: store embeddings in a specialized
database for fast similarity
search (Section 4)
4. RETRIEVAL: given a query, find the MOST
SIMILAR stored chunks
(Section 5)
─────────────────────────────────────────
2. Stage 1: Chunking Documents
Documents are typically too long to embed as a single unit meaningfully (a whole book vs. a single sentence would produce vastly different, hard-to-compare embeddings) — they're split into smaller chunks first.
def chunk_text(text, chunk_size=500, overlap=50):
"""A simple FIXED-SIZE chunking strategy, with OVERLAP between chunks"""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap # OVERLAP ensures information near a chunk
# boundary isn't lost entirely to ONE chunk
return chunks
document = "..." * 1000 # a long document
chunks = chunk_text(document, chunk_size=500, overlap=50)
print(f"Split into {len(chunks)} chunks")Why Overlap Matters
─────────────────────────────────────────
Without overlap, a sentence spanning the boundary between
two chunks gets SPLIT and loses coherence in BOTH resulting
chunks. A modest overlap (e.g. 10% of chunk size) ensures
content near boundaries appears WHOLE in at least ONE chunk.
─────────────────────────────────────────
Chunking Strategy Tradeoffs
─────────────────────────────────────────
SMALL chunks: more PRECISE retrieval (less irrelevant
text bundled in), but may LACK enough
context to be individually meaningful
LARGE chunks: more CONTEXT per chunk, but retrieval
becomes LESS precise (a query might
match a chunk where only ONE small
part is actually relevant)
Semantic chunking split at NATURAL boundaries
(an alternative): (paragraphs, sections) rather
than a FIXED character count
— often produces more
COHERENT chunks, at the cost
of variable chunk sizes
─────────────────────────────────────────
3. Stage 2: Generating Embeddings
Each chunk is converted into a dense vector — directly building on Module 2's embedding concepts, but at the level of entire passages rather than individual words.
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer("all-MiniLM-L6-v2") # a model specifically
# trained to produce
# good SENTENCE/PASSAGE
# embeddings
chunk_embeddings = embedding_model.encode(chunks)
print(f"Embedding shape: {chunk_embeddings.shape}") # (num_chunks, 384) — 384-dimensional
# vectors, one per chunkWhy NOT Just Use BERT's [CLS] Token Directly (Module 4)
─────────────────────────────────────────
BERT (Module 4) was pretrained via MLM, NOT specifically to
produce GOOD passage-level similarity representations — its
raw [CLS] embeddings often perform POORLY for retrieval
tasks specifically. Sentence-embedding models (like the one
above) are FURTHER trained (Module 6's fine-tuning concept)
SPECIFICALLY so that semantically SIMILAR passages end up
CLOSE together in embedding space — directly optimizing for
exactly what RAG's retrieval step needs.
─────────────────────────────────────────
4. Stage 3: Storing Embeddings in a Vector Database
A vector database stores embeddings alongside their original text, optimized specifically for fast similarity search across potentially millions of vectors.
import chromadb
client = chromadb.Client()
collection = client.create_collection(name="knowledge_base")
collection.add(
documents=chunks,
embeddings=chunk_embeddings.tolist(),
ids=[f"chunk_{i}" for i in range(len(chunks))],
)Why a SPECIALIZED Database, Not Just a List of Vectors
─────────────────────────────────────────
A naive approach — comparing a query's embedding against
EVERY stored embedding one by one — becomes impractically
slow at scale (millions of chunks). Vector databases use
APPROXIMATE nearest-neighbor search algorithms (e.g. HNSW —
Hierarchical Navigable Small World graphs) that find
EXTREMELY similar vectors MUCH faster than exhaustive
comparison, at a small, usually acceptable cost to perfect
accuracy.
─────────────────────────────────────────
5. Stage 4: Retrieval via Similarity Search
def retrieve_relevant_chunks(query, collection, embedding_model, top_k=3):
query_embedding = embedding_model.encode([query])
results = collection.query(
query_embeddings=query_embedding.tolist(),
n_results=top_k, # retrieve the top_k MOST SIMILAR chunks
)
return results["documents"][0]
query = "What is the return policy for opened electronics?"
relevant_chunks = retrieve_relevant_chunks(query, collection, embedding_model)
print(relevant_chunks)Cosine Similarity — the Standard Comparison Metric
─────────────────────────────────────────
Vector databases typically use COSINE SIMILARITY (the same
metric from Module 2, Chapter 3, Section 6) to rank stored
chunks by relevance to the query embedding — the SAME
distributional-hypothesis-based similarity concept from
Module 2, now applied to entire passages rather than single
words.
─────────────────────────────────────────
6. Putting It All Together
def build_rag_pipeline(documents, embedding_model, collection):
"""Stages 1-3 — run ONCE, when the knowledge base is built or updated"""
all_chunks = []
for doc in documents:
all_chunks.extend(chunk_text(doc))
embeddings = embedding_model.encode(all_chunks)
collection.add(
documents=all_chunks,
embeddings=embeddings.tolist(),
ids=[f"chunk_{i}" for i in range(len(all_chunks))],
)
def answer_with_rag(query, collection, embedding_model, llm_client):
"""Stage 4 + generation — run for EVERY user query"""
relevant_chunks = retrieve_relevant_chunks(query, collection, embedding_model)
context = "\n\n".join(relevant_chunks)
prompt = f"""Answer the question using ONLY the context below. If the answer
isn't present, say "I don't have that information."
Context:
{context}
Question: {query}"""
response = llm_client.chat.completions.create(
model="gpt-4", messages=[{"role": "user", "content": prompt}],
)
return response.choices[0].message.contentThe Full Pipeline's Two Distinct Phases
─────────────────────────────────────────
INDEXING (build_rag_pipeline): runs ONCE (or whenever the
knowledge base changes) —
chunking, embedding,
storage (Sections 2-4)
QUERYING (answer_with_rag): runs for EVERY user
question — retrieval
+ generation (Section
5 + Module 5's
generation)
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- A RAG pipeline has four stages: chunking documents, generating embeddings, storing them in a vector database, and retrieving the most relevant chunks for a query.
- Chunking involves a real tradeoff between precision (smaller chunks) and context (larger chunks), with overlap preventing information loss at chunk boundaries.
- Sentence-embedding models are specifically fine-tuned to produce good passage-similarity representations, unlike BERT's raw [CLS] token which wasn't optimized for this purpose.
- Vector databases use approximate nearest-neighbor search to make similarity retrieval practical at scale, avoiding an impractical exhaustive comparison against every stored vector.
Concept Check
- Why does chunk overlap help prevent information loss at chunk boundaries?
- Why do RAG pipelines typically use specialized sentence-embedding models rather than BERT's raw [CLS] output?
- What are the two distinct phases of a RAG pipeline, and when does each one run?
Next Chapter
→ Chapter 3: Advanced RAG Techniques
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index