Agent Memory
Short-Term Memory
Every step, you assemble a prompt from more material than will fit. Making that allocation explicit is the whole of short-term memory design.
JrCodex·8 min read
Jr Codex Agentic AI Notes
Level: Intermediate Prerequisites: Chapter 1: Why Agents Need Memory Time to complete: ~25 minutes
Table of Contents
- The Context Budget
- Strategy 1 — Trimming
- Strategy 2 — Summarising
- Strategy 3 — The Scratchpad
- Compressing Tool Results
- Assembling the Context
- Summary & Next Steps
1. The Context Budget
Every step, you assemble a prompt from more material than will fit. Making that allocation explicit is the whole of short-term memory design.
The Allocation
─────────────────────────────────────────
┌─────────────── context window ───────────────┐
│ system │ tools │ scratchpad │ history │ out │
│ ~5% │ ~10% │ ~10% │ ~55% │ ~20% │
└───────────────────────────────────────────────┘
system + tools FIXED. Same every step.
scratchpad SMALL, curated, always present.
history the ELASTIC part — what gets cut.
out RESERVED. Never let input eat it.
─────────────────────────────────────────
from dataclasses import dataclass
@dataclass
class ContextBudget:
window: int = 128_000
reserve_output: int = 4_000
scratchpad_max: int = 6_000
def history_allowance(self, fixed_tokens: int, scratchpad_tokens: int) -> int:
return self.window - self.reserve_output - fixed_tokens - scratchpad_tokens
def fit(messages, allowance, enc, keep_recent=4):
"""Drop from the MIDDLE — never the newest, never the anchor."""
if total(messages, enc) <= allowance:
return messages, []
head, tail = messages[:1], messages[-keep_recent:] # first turn + latest steps
middle, dropped = [], []
for m in reversed(messages[1:-keep_recent]): # newest-first through the middle
if total(head + [m] + middle + tail, enc) <= allowance:
middle.insert(0, m)
else:
dropped.insert(0, m)
return head + middle + tail, dropped # `dropped` feeds summarisation2. Strategy 1 — Trimming
The simplest strategy: keep the most recent N steps, discard the rest.
What It Gets Right and Wrong
─────────────────────────────────────────
RIGHT cheap, deterministic, no extra LLM call,
and the recent steps are usually the
relevant ones.
WRONG drops the ORIGINAL GOAL, which lives at
the very start. An agent that trims its
first message has forgotten what it is
doing — the single worst failure available.
─────────────────────────────────────────
The Fix Is Structural
─────────────────────────────────────────
Never trim positionally alone. PIN the messages
that must survive:
PINNED system prompt, original goal, active
plan, user constraints
ELASTIC intermediate steps and tool results
Trim only from the elastic set. `fit()` above does
this by keeping `head` and `tail` unconditionally.
─────────────────────────────────────────
3. Strategy 2 — Summarising
Replace dropped steps with a generated summary rather than discarding them outright.
COMPACT_PROMPT = """Summarise these agent steps for a colleague taking over mid-task.
Preserve EXACTLY:
- every fact discovered, with its source tool
- every decision made, and why
- every tool that FAILED, and how it failed
- every open question
Omit: reasoning that led nowhere, restatements of the goal.
Write as terse bullet points, not prose.
STEPS:
{steps}"""
def compact(client, dropped_messages, prior_summary=None):
body = render(dropped_messages)
if prior_summary: # ROLL UP — do not stack summaries
body = f"PREVIOUS SUMMARY:\n{prior_summary}\n\nNEW STEPS:\n{body}"
return client.chat.completions.create(
model="gpt-4o-mini", # a cheap model is fine here
messages=[{"role": "user", "content": COMPACT_PROMPT.format(steps=body)}],
temperature=0,
).choices[0].message.contentThe Two Details That Matter
─────────────────────────────────────────
"EVERY TOOL THAT FAILED, AND HOW"
Without this line, summarisation erases failure
history — and the agent cheerfully retries the
tool that already failed three times. This is a
real and common regression.
ROLL UP, DO NOT STACK
Summarising a summary compounds loss. Always
fold new steps into the PREVIOUS summary to
produce ONE current summary, rather than
accumulating a chain of them.
─────────────────────────────────────────
The Trade
─────────────────────────────────────────
Costs an extra LLM call and is LOSSY — each
compaction can drop a detail permanently.
Worth it when runs are long. Unnecessary below
roughly ten steps, where trimming with pinning is
sufficient.
─────────────────────────────────────────
4. Strategy 3 — The Scratchpad
The most effective technique in this chapter, and the least used. Maintain a small structured state object outside the message history.
from dataclasses import dataclass, field
@dataclass
class Scratchpad:
goal: str
facts: dict = field(default_factory=dict) # what has been ESTABLISHED
done: list = field(default_factory=list) # completed sub-tasks
todo: list = field(default_factory=list) # remaining sub-tasks
failed: dict = field(default_factory=dict) # tool -> why it failed
open_questions: list = field(default_factory=list)
def render(self) -> str:
parts = [f"GOAL: {self.goal}"]
if self.facts:
parts.append("ESTABLISHED:\n" + "\n".join(f" {k}: {v}"
for k, v in self.facts.items()))
if self.done:
parts.append("DONE:\n" + "\n".join(f" ✓ {d}" for d in self.done))
if self.todo:
parts.append("REMAINING:\n" + "\n".join(f" · {t}" for t in self.todo))
if self.failed:
parts.append("DO NOT RETRY:\n" + "\n".join(f" ✗ {k}: {v}"
for k, v in self.failed.items()))
if self.open_questions:
parts.append("OPEN:\n" + "\n".join(f" ? {q}" for q in self.open_questions))
return "\n\n".join(parts)Why It Beats Both Other Strategies
─────────────────────────────────────────
BOUNDED it is a fixed-size summary of state,
not a growing transcript. It does not
need trimming at all.
CURATED only what matters is in it, because
your code decides what goes in — not
the accident of what happened recently.
ALWAYS AT render it at the END of the context
THE END every step, so the goal and state sit in
the highest-attention position (Module 2,
Chapter 1's fix for goal drift).
EXPLICIT "DO NOT RETRY" mechanically prevents the
FAILURES repeated-failure loop, rather than
hoping the model notices.
─────────────────────────────────────────
def update_scratchpad(pad, call, result):
"""Maintain state in CODE — deterministic, and cheaper than asking the model."""
if isinstance(result, dict) and "error" in result:
pad.failed[f"{call.name}({call.args})"] = result.get("message", result["error"])
else:
pad.done.append(f"{call.name} → {summarise_result(result)}")
pad.facts.update(extract_facts(result))
return pad5. Compressing Tool Results
Module 2, Chapter 1 measured tool results at ~80% of agent context. This is where the budget is won or lost.
def compress(result, max_chars=2_000):
"""Reduce a tool result to what the model can actually use."""
if isinstance(result, list):
if len(result) > 10:
head = json.dumps(result[:5], default=str)
return (f"{head}\n...[{len(result)} items total, showing first 5. "
f"Refine your query or request a specific item.]")
return json.dumps(result, default=str)
text = result if isinstance(result, str) else json.dumps(result, default=str)
if len(text) <= max_chars:
return text
return text[:max_chars] + f"\n...[truncated from {len(text)} chars]"
def compress_history(messages, current_step, keep_full_for=2):
"""OLD tool results get compressed harder than recent ones."""
out = []
for m in messages:
age = current_step - m.get("step", current_step)
if m["role"] == "tool" and age > keep_full_for:
out.append({**m, "content": compress(m["content"], max_chars=300)})
else:
out.append(m)
return outThe Age-Based Principle
─────────────────────────────────────────
A tool result from the last step may need full
detail — the agent is reasoning about it now.
A result from eight steps ago needs only its
CONCLUSION. Keeping 4,000 tokens of it is pure
waste.
Compressing by age recovers large amounts of
budget with almost no loss of useful signal.
─────────────────────────────────────────
6. Assembling the Context
All of it, in the order that matters.
def build_context(system, tools, scratchpad, history, budget, enc):
fixed = count(system, enc) + count(tools, enc)
pad_text = scratchpad.render()
pad_tokens = min(count(pad_text, enc), budget.scratchpad_max)
allowance = budget.history_allowance(fixed, pad_tokens)
kept, dropped = fit(compress_history(history, len(history)), allowance, enc)
if dropped:
summary = compact(client, dropped, scratchpad.summary)
scratchpad.summary = summary # rolled up, not stacked
kept = [{"role": "user", "content": f"[Earlier steps]\n{summary}"}] + kept
return [
{"role": "system", "content": system}, # 1. stable prefix — CACHEABLE
*kept, # 2. the elastic middle
{"role": "user", "content": pad_text}, # 3. state, LAST = most attended
]Why This Order
─────────────────────────────────────────
STATIC FIRST the system prompt and tool schemas
never change, so a stable prefix
gets prompt-cache hits (Gen AI
Notes, Module 2, Chapter 4).
ELASTIC MIDDLE the part that varies, positioned
where lower attention costs least.
STATE LAST goal, progress and failures in the
highest-attention position.
Cost and quality point the same way here — the
ordering that caches best is also the ordering
that reasons best.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Short-term memory is budget allocation: fixed prefix, curated scratchpad, elastic history, and a reserved output allowance that input must never consume.
- Trimming must pin the goal and constraints — positional trimming alone deletes the first message, which is the task itself.
- Summarisation must explicitly preserve failures and roll up rather than stack, or the agent retries tools that already failed.
- A structured scratchpad rendered at the end of the context is the strongest technique: bounded, curated, and it prevents retry loops mechanically.
Concept Check
- Why is naive "keep the last N messages" trimming actively dangerous for an agent?
- What regression appears when a summarisation prompt omits the instruction to preserve tool failures?
- Why do cost optimisation and reasoning quality agree on the static-first, state-last ordering?
Next Chapter
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index