Evaluating And Operating Agents
Debugging and Error Recovery
The instinct is to read from the end, where the error is. That is usually the wrong end.
JrCodex·8 min read
Jr Codex Agentic AI Notes
Level: Advanced Prerequisites: Chapter 3: Observability and Tracing Time to complete: ~25 minutes
Table of Contents
- Reading a Failed Trace
- The Diagnostic Table
- Bisecting With Forks
- Retry Strategy
- Fallbacks and Degradation
- Failing Usefully
- Summary & Next Steps
1. Reading a Failed Trace
The instinct is to read from the end, where the error is. That is usually the wrong end.
The Method
─────────────────────────────────────────
1. Find the FIRST step whose output is wrong,
not the step that raised.
2. Read what the model SAW at that step — the
full assembled context, not your prompt
template.
3. Ask: given exactly that input, was the
decision reasonable?
YES ──► the CONTEXT was wrong. Fix memory,
perception, or the tool result.
NO ──► the DECISION was wrong. Fix the prompt,
the tool descriptions, or the model.
Most agent bugs are the first case, and most
debugging effort goes to the second.
─────────────────────────────────────────
Why the Error Location Misleads
─────────────────────────────────────────
An agent that crashes at step 14 usually went
wrong at step 6, when a tool returned an empty
list and it drew the wrong conclusion.
Steps 7-13 were reasonable given that wrong
conclusion. The exception at 14 is a symptom
eight steps downstream of the cause.
─────────────────────────────────────────
2. The Diagnostic Table
Symptom Look At Likely Cause
─────────────────────────────────────────
Wrong tool chosen tool descriptions overlapping or missing
"DO NOT USE FOR"
(Module 2, Ch.3)
Same call repeated the trace at that earlier result scrolled
step out of attention
(Module 2, Ch.1)
Forgot the goal context assembly goal not re-anchored at
the end (Module 3, Ch.2)
Right answer, far too step count vs no plan; wandering
many steps min_steps (Module 4, Ch.1)
Confident but wrong tool results at that agent inferred rather
step than queried; grounding
rule missing
Fails only in fixture vs live diff an unhandled real-world
production failure mode
(Chapter 2, Section 4)
Gradual quality eval deltas over prompt patches and tool
decline releases accumulation
─────────────────────────────────────────
The Habit Worth Building
─────────────────────────────────────────
Before changing anything, state which row you are
in.
"The agent is dumb" is not a diagnosis, and the
fix that follows it is usually a longer prompt —
which is the least effective intervention
available.
─────────────────────────────────────────
3. Bisecting With Forks
Agent runs are non-deterministic, so you cannot reproduce a failure by re-running. Module 6, Chapter 2's checkpoints solve this.
def bisect(run_id, checkpointer, agent, is_healthy):
"""Binary search the checkpoints for the first bad state."""
lo, hi = 0, last_step(run_id, checkpointer)
while lo < hi:
mid = (lo + hi) // 2
state = checkpointer.at_step(run_id, mid)
if is_healthy(state):
lo = mid + 1 # still good — the problem is later
else:
hi = mid # already bad — look earlier
return lo # the first step where it went wrongdef test_fix(run_id, checkpointer, agent, bad_step, change):
"""Fork from just before the failure and try ONE change."""
state = checkpointer.at_step(run_id, bad_step - 1)
change(state) # new prompt, corrected tool result, ...
return agent.resume_from(state) # everything before is held constantWhy This Is the Only Reliable Method
─────────────────────────────────────────
Re-running the whole task changes the trajectory,
so you cannot tell whether your fix helped or the
sampling was kinder.
Forking holds steps 1 to N-1 EXACTLY constant and
varies one thing. That is a controlled experiment
rather than a re-roll.
─────────────────────────────────────────
4. Retry Strategy
Retrying the wrong thing burns budget without improving anything.
def classify_failure(error) -> str:
if error.get("error") in ("rate_limited", "timeout", "server_error"):
return "transient" # the SAME call may work
if error.get("error") in ("not_found", "empty", "forbidden"):
return "semantic" # the same call will NEVER work
if error.get("error") == "invalid_arguments":
return "correctable" # a DIFFERENT call may work
return "unknown"
def handle(error, call, state, tools):
kind = classify_failure(error)
if kind == "transient" and state.attempts[call.id] < 3:
sleep(backoff_with_jitter(state.attempts[call.id]))
return "retry_same" # the only case where retrying identically is right
if kind == "correctable":
return "retry_with_feedback" # give the error to the model; it fixes the args
if kind == "semantic":
return "inform_and_continue" # tell the agent, let it choose differently
return "escalate"The Distinction That Saves the Most Money
─────────────────────────────────────────
TRANSIENT retry identically, with backoff.
SEMANTIC NEVER retry. "not_found" will return
not_found forever.
An agent retrying a semantic failure is Module 4's
loop pathology, and blind retry logic in the
executor CAUSES it. Classify before retrying.
─────────────────────────────────────────
5. Fallbacks and Degradation
The Fallback Ladder
─────────────────────────────────────────
1. RETRY the same call transient only
2. SAME tool, simpler args narrower query,
smaller page
3. DIFFERENT tool another route to the
same fact
4. CHEAPER MODEL if the failure was
capacity or rate
5. PARTIAL RESULT return what is known,
state what is missing
6. HUMAN escalate with context
Descend one rung at a time. Never jump from 1 to 6.
─────────────────────────────────────────
async def with_fallbacks(primary, alternatives, budget):
errors = []
for attempt in [primary, *alternatives]:
if budget.exhausted():
break
try:
result = await attempt()
if result.ok:
return result
errors.append(result.error)
except Exception as e:
errors.append(str(e))
return PartialResult(errors=errors, # rung 5 — never a bare failure
partial=collect_completed(),
message="Could not complete; here is what was established.")Degrade, Do Not Collapse
─────────────────────────────────────────
An agent that answers three of five questions and
says which two it could not is useful.
An agent that returns "failed" after doing the
same work is not — and it cost the same.
This is the Gen AI Notes' fail-soft principle,
and it matters more for agents because the work
already done is more expensive.
─────────────────────────────────────────
6. Failing Usefully
@dataclass
class AgentFailure:
reason: str # what stopped it, in plain language
stage: str # which step or sub-task
established: list[str] # facts confirmed before failing
attempted: list[str] # what was tried, so nobody repeats it
blocked_by: str | None # the specific blocker
suggested_next: str | None # what a human could do
trace_url: str # the full trace, one click away
partial_output: str | None # whatever WAS produced
cost_cents: floatCompare the Two Failures
─────────────────────────────────────────
USELESS
"The agent was unable to complete the task."
USEFUL
"Could not retrieve Q3 revenue: the finance
database rejected the connection (permission
denied for role `agent_ro`).
Established: Q1 and Q2 figures, and the
regional split.
Tried: direct query, cached report, the
summary API.
Next: grant `agent_ro` read access to
`fin.revenue_q3`, or supply the figure
manually.
Partial report attached. Cost: 14c.
Trace: /traces/run-8841"
─────────────────────────────────────────
Why This Is Engineering, Not Politeness
─────────────────────────────────────────
The second version turns a failed run into a
resolvable ticket. Someone can act on it in two
minutes without opening the trace.
It also prevents the most expensive recovery
pattern there is: a human re-doing all the work
the agent already did, because the failure did not
say what had been established.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Debug from the first wrong step, not the step that raised; then ask whether the decision was reasonable given what the model actually saw, which separates context bugs from decision bugs.
- Non-determinism makes re-running useless for reproduction — bisect the checkpoints and fork to test one change against a constant history.
- Classify failures before retrying: transient failures justify an identical retry, semantic ones never will, and blind retry logic manufactures loop pathologies.
- Descend the fallback ladder one rung at a time and always return partial results with what was established, what was tried, and what a human should do next.
Concept Check
- An agent crashes at step 14. Why is step 14 usually the wrong place to start reading?
- Why does forking from a checkpoint give a more trustworthy signal about a fix than re-running the task?
- Which failure classification must never be retried, and what pathology does retrying it create?
Module 7 Complete — What's Next
You can now measure agents, trace them, and recover from their failures. Module 8 addresses the class of failure that measurement alone cannot fix: an agent taking a real, irreversible action that nobody sanctioned.
Next Module
→ Module 8: Safety, Control & Governance
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index