Rag And Agents
Why RAG: The Limits of Parametric Knowledge
Everything an LLM "knows" from Module 3's pretraining and Module 6's fine-tuning is encoded implicitly within its billions of weights — called parametric knowle
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Module 7: Prompt Engineering Time to complete: ~20 minutes
Table of Contents
- Parametric Knowledge — What's "Baked In"
- Three Fundamental Limitations
- Hallucination
- The Alternative: Non-Parametric Knowledge
- RAG's Core Idea
- RAG vs Fine-Tuning — a Direct Comparison
- Summary & Next Steps
1. Parametric Knowledge — What's "Baked In"
Everything an LLM "knows" from Module 3's pretraining and Module 6's fine-tuning is encoded implicitly within its billions of weights — called parametric knowledge, since it lives in the model's parameters, not in any separately inspectable database.
2. Three Fundamental Limitations
Why Parametric Knowledge Alone Isn't Enough
─────────────────────────────────────────
1. KNOWLEDGE CUTOFF: a model only knows what existed
in its training data — it has
NO knowledge of events,
documents, or facts that
emerged AFTER its training
data was collected (Module 3,
Chapter 3, Section 6)
2. NO ACCESS TO PRIVATE/ a model trained on public
PROPRIETARY DATA: internet text has NEVER
seen YOUR company's
internal documents,
YOUR specific customer
data, or any other
information that wasn't
part of its training
corpus
3. IMPERFECT RECALL AT even for information
SCALE: genuinely present
in training data,
a model's recall
of SPECIFIC facts
(exact dates,
precise figures)
can be unreliable
— it learned
STATISTICAL
patterns, not a
perfectly indexed
database
─────────────────────────────────────────
3. Hallucination
Hallucination — an LLM generating text that is fluent and confident-sounding, but factually incorrect or entirely fabricated — is a direct consequence of these limitations.
Why Hallucination Happens, Mechanically
─────────────────────────────────────────
A decoder-only model (Module 5, Chapter 1) generates the
NEXT token by predicting what's STATISTICALLY plausible,
given everything so far — it has NO built-in mechanism to
distinguish "I have HIGH confidence this fact is TRUE" from
"this SOUNDS like a plausible continuation, but I'm
essentially GUESSING." When asked about something outside
its knowledge (Section 2), it doesn't necessarily "know that
it doesn't know" — it may confidently generate a
PLAUSIBLE-sounding but FABRICATED answer instead.
─────────────────────────────────────────
# A hallucination is INDISTINGUISHABLE in surface fluency from a correct answer —
# this is precisely what makes it dangerous in high-stakes applications
prompt = "What was the exact revenue figure Acme Corp reported in their Q3 2024 earnings call?"
# If "Acme Corp" is fictional, or the model never saw this SPECIFIC data, it may
# STILL generate a specific-sounding number, confidently and fluently, with
# NOTHING in the output signaling "I am uncertain" or "I am fabricating this"4. The Alternative: Non-Parametric Knowledge
Non-parametric knowledge lives OUTSIDE the model's weights — in an external, separately maintained, inspectable source (a document collection, a database) that can be searched and retrieved at inference time.
Parametric vs Non-Parametric Knowledge
─────────────────────────────────────────
Parametric (Sections 1-3): FIXED once training ends;
CANNOT be updated without
retraining or fine-tuning
(Module 6); INSPECTING what
the model "knows" requires
indirect probing
Non-parametric: UPDATABLE instantly — add,
edit, or remove documents
without touching the
model at all; fully
INSPECTABLE — you can
directly read exactly
what information is
available
─────────────────────────────────────────
5. RAG's Core Idea
Retrieval-Augmented Generation (RAG) combines both: retrieve relevant information from a non-parametric knowledge source, then provide it to the LLM as context (Module 7, Chapter 1, Section 5) — letting the model's parametric knowledge handle language understanding and generation, while the RETRIEVED text supplies the specific, up-to-date, or proprietary facts.
The Core RAG Loop
─────────────────────────────────────────
User query
│
▼
RETRIEVE relevant documents/passages from an external
knowledge source (Chapter 2 covers HOW)
│
▼
AUGMENT the prompt — insert the retrieved text as context
(Module 7, Ch.1, Sec.5)
│
▼
GENERATE a response, using the LLM's language ability
PLUS the retrieved, grounding information
─────────────────────────────────────────
# A minimal, conceptual illustration of RAG's structure
def rag_query(user_question, knowledge_base):
relevant_docs = retrieve_relevant_documents(user_question, knowledge_base) # Chapter 2
prompt = f"""Answer the question using ONLY the following context. If the
answer isn't in the context, say you don't know.
Context:
{relevant_docs}
Question: {user_question}"""
return llm_generate(prompt) # Module 5, Chapter 1's generation6. RAG vs Fine-Tuning — a Direct Comparison
When Each Approach Fits Better
─────────────────────────────────────────
RAG fits better for: FREQUENTLY changing information
(news, prices, inventory);
PROPRIETARY documents that
shouldn't be baked into weights;
needing FULL TRACEABILITY (which
SPECIFIC document supported this
answer?)
Fine-tuning (Module 6) teaching a specific STYLE,
fits better for: FORMAT, or behavior; tasks
requiring deep DOMAIN
REASONING patterns, not
just domain FACTS; when
latency matters (RAG adds
a retrieval step BEFORE
generation even begins)
Often COMBINED: fine-tune a model for
DESIRED BEHAVIOR/
FORMAT (Module 6),
THEN use RAG (this
module) to supply
current, specific
FACTS at inference
time
─────────────────────────────────────────
This directly extends Module 6, Chapter 4, Section 2's decision order — RAG sits BEFORE fine-tuning in that hierarchy, since it requires no training at all and directly solves the knowledge-cutoff/proprietary-data problem that fine-tuning is a comparatively expensive and imperfect way to address.
7. Summary & Next Steps
Key Takeaways
- Parametric knowledge — everything encoded in a model's weights — is fixed at training time, has no access to private data, and suffers from a hard knowledge cutoff.
- Hallucination occurs because a decoder-only model generates statistically plausible continuations without an inherent mechanism to signal genuine uncertainty.
- Non-parametric knowledge lives in an external, updatable, inspectable source, and RAG combines it with an LLM's language ability by retrieving relevant context and inserting it into the prompt.
- RAG and fine-tuning solve different problems and are often combined: fine-tuning shapes behavior and format, while RAG supplies current, specific, or proprietary facts.
Module 8 (Partial) — What's Next
You now understand exactly why RAG exists and what fundamental problem it solves. Chapter 2 covers how to actually build one: chunking documents, generating embeddings, storing them in a vector database, and retrieving the most relevant passages for a given query.
Concept Check
- Why can't a model's knowledge cutoff be fixed simply by asking it nicely in the prompt?
- Why does hallucination happen mechanically, given how a decoder-only model generates text?
- Give an example of when fine-tuning would be a better fit than RAG, and vice versa.
Next Chapter
→ Chapter 2: Building a RAG Pipeline
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index