Agentic AI

Agent Memory

Why Agents Need Memory

Every apparent memory an agent has is something

JrCodex·7 min read

Jr Codex Agentic AI Notes

Level: Intermediate Prerequisites: Module 2, Chapter 1: The LLM as a Reasoning Engine Time to complete: ~20 minutes

Table of Contents

  1. The Stateless Model Problem
  2. Two Different Failures
  3. The Four Memory Types
  4. What Deserves to Be Remembered
  5. The Memory Interface
  6. Summary & Next Steps

1. The Stateless Model Problem

The Core Fact
─────────────────────────────────────────
  The model has NO memory. None.

  Every apparent memory an agent has is something
  YOUR CODE put into the context before the call.

  "The agent remembers my name" means: your code
  retrieved the name and pasted it into the prompt.
  Nothing else is happening.
─────────────────────────────────────────

This is the same statelessness noted in the Generative AI Notes, but the consequences are sharper for agents. A chat application resends a conversation. An agent must decide, at every step, which of a potentially unbounded history is worth the context it costs.

Memory Is a SELECTION Problem
─────────────────────────────────────────
  Not "how do I store things" — storage is easy.

  "Given a fixed context budget and a growing pile
   of history, what goes in THIS step's prompt?"

  Every memory design in this module is an answer to
  that question.
─────────────────────────────────────────

2. Two Different Failures

Memory problems look similar and have entirely different fixes. Separating them is the first diagnostic step.

FAILURE 1 — WITHIN a run
─────────────────────────────────────────
  Step 14 of a long task. The agent has forgotten
  the original goal, re-runs a tool it already ran,
  and contradicts a decision it made at step 4.

  Cause: the context grew past what the model
  attends to reliably (Module 2, Chapter 1).

  Fix: SHORT-TERM memory — trimming, summarisation,
  scratchpads. Chapter 2.

FAILURE 2 — ACROSS runs
─────────────────────────────────────────
  The user explained their timezone yesterday. Today
  the agent asks again. It makes the same tool
  mistake it made last week, and the week before.

  Cause: nothing persisted. Every run starts blank.

  Fix: LONG-TERM memory — a store outside the
  context. Chapter 3.
─────────────────────────────────────────

The Diagnostic
─────────────────────────────────────────
  "Did the agent ever know this?"

  YES, earlier in this run  ──► short-term problem
  YES, in a previous run    ──► long-term problem
  NO, never                 ──► not a memory problem
                                at all — a tool or
                                retrieval problem
─────────────────────────────────────────

3. The Four Memory Types

Borrowed from cognitive science, and genuinely useful because each maps to a different storage mechanism.

WORKING MEMORY
─────────────────────────────────────────
  What: the current run's context — goal, recent
        steps, latest results
  Where: the context window itself
  Lifetime: one run
  Chapter 2

SEMANTIC MEMORY
─────────────────────────────────────────
  What: FACTS. "The user's timezone is IST."
        "Acme is on the enterprise plan."
  Where: a key-value store or vector store
  Lifetime: until contradicted
  Chapter 3

EPISODIC MEMORY
─────────────────────────────────────────
  What: EVENTS. "On 12 Aug the user asked for a
        refund; it was approved."
  Where: a vector store over past interactions
  Lifetime: long, with decay
  Chapter 3

PROCEDURAL MEMORY
─────────────────────────────────────────
  What: HOW to do things. "For refunds over $500,
        check the fraud flag first."
  Where: the system prompt, few-shot examples, or a
         retrieved playbook
  Lifetime: until the process changes
  Chapter 3
─────────────────────────────────────────

Why the Distinction Is Practical
─────────────────────────────────────────
  They have different UPDATE rules.

  A fact is OVERWRITTEN when it changes — one true
  timezone at a time.
  An event is APPENDED — history does not get
  rewritten.
  A procedure is VERSIONED and reviewed.

  Storing all three in one undifferentiated vector
  store is the most common memory design mistake,
  because it forces one update rule onto three kinds
  of thing.
─────────────────────────────────────────

4. What Deserves to Be Remembered

Storing everything is as bad as storing nothing: retrieval degrades, contradictions accumulate, and the context fills with noise.

WORTH STORING
─────────────────────────────────────────
  Stated user preferences and constraints
    "always use metric", "I'm in IST"
  Durable entity facts
    plan tier, account owner, region
  Decisions and their reasons
    "we chose Postgres because of PostGIS"
  Outcomes and lessons
    "the export tool times out above 10k rows"
  Corrections
    "no, the invoice date, not the due date"

NOT WORTH STORING
─────────────────────────────────────────
  Full conversation transcripts
    huge, low signal, retrieval-hostile
  Anything derivable from a tool call
    do not cache the account balance; fetch it
  One-off task details
    "summarise THIS document" is not a preference
  The agent's own intermediate reasoning
    it will re-derive it, and it is often wrong
─────────────────────────────────────────

The Test
─────────────────────────────────────────
  "Would a competent colleague write this down, or
   just look it up again?"

  Colleagues note preferences, decisions and hard-won
  lessons. They do not transcribe meetings, and they
  do not memorise numbers they can query.
─────────────────────────────────────────

5. The Memory Interface

Whatever the backing store, agent memory needs four operations. Defining them explicitly keeps the design honest.

from dataclasses import dataclass, field
from typing import Literal
import time
 
MemoryKind = Literal["semantic", "episodic", "procedural"]
 
@dataclass
class MemoryItem:
    content: str
    kind: MemoryKind
    key: str | None = None                 # semantic facts are keyed, so they can be REPLACED
    created_at: float = field(default_factory=time.time)
    last_used: float = field(default_factory=time.time)
    uses: int = 0
    source: str = ""                       # WHERE it came from — needed to audit a bad memory
 
class Memory:
    def write(self, item: MemoryItem) -> None:
        """Store. For semantic items, REPLACE any existing item with the same key."""
 
    def read(self, query: str, k: int = 5, kind: MemoryKind | None = None) -> list[MemoryItem]:
        """Retrieve the k most relevant items, optionally of one kind only."""
 
    def update(self, key: str, content: str) -> None:
        """Correct a fact in place, preserving its key."""
 
    def forget(self, key: str | None = None, older_than: float | None = None) -> int:
        """Delete by key or by age. Returns how many were removed."""
Why forget() Is Not Optional
─────────────────────────────────────────
  Three reasons, all of which will bite you:

  CORRECTNESS   stale facts are worse than no facts.
                An agent confidently using last
                quarter's pricing is worse than one
                that looks it up.

  PRIVACY       users can ask for their data to be
                deleted, and you must be able to
                comply. A memory system with no
                delete path is a compliance problem.

  RETRIEVAL     precision degrades as the store
                grows. Pruning is maintenance, not
                loss.
─────────────────────────────────────────

6. Summary & Next Steps

Key Takeaways

  • The model has no memory; every apparent memory is something your code selected and placed into the context, which makes memory a selection problem under a fixed budget.
  • Within-run and across-run forgetting look alike but need different fixes — ask whether the agent ever knew the fact, and when.
  • The four types differ in update rule: facts are overwritten, events appended, procedures versioned, working memory discarded — so one undifferentiated store is a design mistake.
  • Store preferences, durable facts, decisions, lessons and corrections; never store transcripts or anything a tool can fetch.

Concept Check

  1. An agent asks for the user's timezone every session despite being told each time. Which failure is this, and which chapter addresses it?
  2. Why is storing semantic facts and episodic events in the same undifferentiated store a problem?
  3. Give three distinct reasons a memory system needs a forget operation.

Next Chapter

→ Chapter 2: Short-Term Memory

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index