Evaluating And Operating Agents
Building an Agent Eval Set
An agent eval case needs more than an input and an expected output — it must describe the environment and the acceptable path.
JrCodex·7 min read
Jr Codex Agentic AI Notes
Level: Advanced Prerequisites: Chapter 1: Evaluating Agents Time to complete: ~25 minutes
Table of Contents
- The Case Structure
- Sourcing Cases
- Deterministic Environments
- Injecting Failures
- Assertions Over Judgement
- Running It in CI
- Summary & Next Steps
1. The Case Structure
An agent eval case needs more than an input and an expected output — it must describe the environment and the acceptable path.
from dataclasses import dataclass, field
@dataclass
class AgentCase:
id: str
prompt: str
fixtures: dict = field(default_factory=dict) # the world the agent will see
expected: str | None = None # if a ground truth exists
required_elements: list = field(default_factory=list) # must appear in the answer
min_steps: int = 1 # denominator for step efficiency
appropriate_tools: list = field(default_factory=list)
forbidden_tools: list = field(default_factory=list) # calling these FAILS the case
inject_failures: dict = field(default_factory=dict) # tool -> failure to simulate
max_cost_cents: float = 20.0
tags: list = field(default_factory=list) # "regression", "adversarial", ...The Two Fields That Do the Most Work
─────────────────────────────────────────
forbidden_tools
Turns a safety property into a mechanical
check. "Must never call issue_refund on a
read-only question" becomes a test that FAILS,
not a hope.
min_steps
Without it, step efficiency has no denominator
and you cannot tell a 3-step run from a 9-step
one on the same task. Set it by solving the case
by hand once.
─────────────────────────────────────────
2. Sourcing Cases
The Mix
─────────────────────────────────────────
40% TYPICAL real traffic, the ordinary
requests
25% EDGE empty results, ambiguous
phrasing, very large results,
multiple valid answers
20% ADVERSARIAL prompt injection in tool
results, out-of-scope requests,
requests that should be REFUSED
15% REGRESSION every production bug, preserved
forever
─────────────────────────────────────────
def case_from_trace(trace, expected_answer=None) -> AgentCase:
"""Turn a real production run into a permanent test case."""
return AgentCase(
id=f"trace-{trace.run_id}",
prompt=trace.initial_prompt,
fixtures={c.name + ":" + hash_args(c.args): c.result # FREEZE what it saw
for c in trace.tool_calls},
expected=expected_answer,
min_steps=count_necessary(trace), # after human review
appropriate_tools=list({c.name for c in trace.tool_calls}),
tags=["regression", trace.failure_kind] if trace.failed else ["typical"],
)The Regression Bucket Grows for Free
─────────────────────────────────────────
Every production failure becomes a case. The
fixtures freeze exactly the tool results that
triggered it, so the bug is reproducible even
though the agent is not deterministic.
This is the highest-value habit in this module,
and it costs one function call per incident.
─────────────────────────────────────────
3. Deterministic Environments
An agent that hits live systems cannot be evaluated: the world changes under it, and it may act on real data.
class FixtureTools:
"""Replay recorded tool results. Deterministic, free, and safe."""
def __init__(self, fixtures: dict, failures: dict | None = None):
self.fixtures, self.failures = fixtures, failures or {}
self.calls = []
def execute(self, name, args):
self.calls.append((name, args))
key = f"{name}:{hash_args(args)}"
if name in self.failures: # simulated failure (Section 4)
return self.failures[name]
if key in self.fixtures:
return self.fixtures[key]
return {"error": "no_fixture", # UNSEEN call — informative,
"message": f"No recorded result for {name}({args}).",
"hint": "This call was not made in the recorded trace."}Why the "no_fixture" Response Matters
─────────────────────────────────────────
It tells you the agent tried something the
original run did not — which is exactly the signal
you want when a prompt change alters behaviour.
Returning an empty result instead would hide the
divergence and let the agent proceed as if
nothing had changed.
─────────────────────────────────────────
The Three Environment Tiers
─────────────────────────────────────────
FIXTURES replayed results. Fast, free,
deterministic. Use for CI on every
commit.
SANDBOX a real but isolated system with
seeded data. Slower, catches
integration bugs. Nightly.
SHADOW production tools, read-only, real
traffic, output discarded. Weekly, or
before a release.
─────────────────────────────────────────
4. Injecting Failures
Chapter 1 identified recovery rate as the best predictor of production robustness. It cannot be measured unless things fail.
FAILURE_MODES = {
"timeout": {"error": "timeout", "message": "Request timed out after 30s."},
"not_found": {"error": "not_found", "message": "No record with that id.",
"hint": "Ids are case-sensitive."},
"rate_limit": {"error": "rate_limited", "message": "Try again in 60s."},
"permission": {"error": "forbidden", "message": "Your role cannot read this field."},
"empty": {"results": [], "total_matching": 0},
"malformed": "<<<not json at all>>>",
"huge": {"results": [{"id": i} for i in range(50_000)]}, # context bomb
}
def recovery_cases(base: AgentCase) -> list[AgentCase]:
"""One variant per failure mode, on the agent's FIRST tool."""
return [
AgentCase(**{**base.__dict__,
"id": f"{base.id}-{mode}",
"inject_failures": {base.appropriate_tools[0]: failure},
"tags": [*base.tags, "recovery"]})
for mode, failure in FAILURE_MODES.items()
]What Each Mode Tests
─────────────────────────────────────────
timeout / rate_limit does it retry sensibly, or
hammer the tool?
not_found does it try a different
lookup, or loop?
permission does it report the block, or
pretend it succeeded?
empty does it conclude "no data
exists", or keep searching
forever? (Module 2's
"negative conclusions"
weakness)
malformed does it crash the run?
huge does perception truncate, or
does the context explode?
─────────────────────────────────────────
5. Assertions Over Judgement
The Preference Order
─────────────────────────────────────────
1. MECHANICAL ASSERTIONS free, instant,
unambiguous
2. STRUCTURAL CHECKS did it call the right
tools, in a sane order?
3. LLM JUDGE only for the parts that
genuinely need judgement
Most agent eval can be assertions. Teams reach for
a judge first because it feels thorough, and pay
for it on every CI run.
─────────────────────────────────────────
def evaluate_case(agent, case) -> Result:
tools = FixtureTools(case.fixtures, case.inject_failures)
run = agent.run(case.prompt, tools=tools)
failures = []
# 1. MECHANICAL — no model involved
called = {name for name, _ in tools.calls}
if forbidden := called & set(case.forbidden_tools):
failures.append(f"called forbidden tool(s): {forbidden}")
if run.cost_cents > case.max_cost_cents:
failures.append(f"cost {run.cost_cents:.1f}c over budget {case.max_cost_cents}c")
for element in case.required_elements:
if element.lower() not in run.final_answer.lower():
failures.append(f"answer missing required element: {element}")
# 2. STRUCTURAL
if len(tools.calls) > case.min_steps * 3:
failures.append(f"{len(tools.calls)} steps vs minimum {case.min_steps}")
# 3. JUDGE — last resort, and only if the cheap checks passed
if not failures and case.expected:
verdict = judge_equivalent(run.final_answer, case.expected)
if not verdict.equivalent:
failures.append(f"judge: {verdict.reason}")
return Result(case.id, passed=not failures, failures=failures, run=run)6. Running It in CI
def run_suite(agent, cases, gates) -> bool:
results = [evaluate_case(agent, c) for c in cases]
scores = aggregate(results)
print(f"passed {sum(r.passed for r in results)}/{len(results)}")
for name, threshold in gates.items():
actual = scores[name]
ok = actual >= threshold if name not in LOWER_IS_BETTER else actual <= threshold
print(f" {'PASS' if ok else 'FAIL'} {name}: {actual:.2f} (gate {threshold})")
if not ok:
return False
return True
GATES = {
"unauthorised_actions": 0, # ABSOLUTE — Chapter 1
"task_completion": 0.85,
"recovery_rate": 0.70,
"redundant_call_rate": 0.15, # lower is better
"cost_per_success": 8.0, # cents, lower is better
}Gate on Deltas, Not Just Absolutes
─────────────────────────────────────────
An absolute gate catches a bad release. A DELTA
gate catches a slow slide.
if completion < baseline - 0.03: fail
Agents degrade gradually as prompts accumulate
patches and tools multiply. Comparing each run to
a stored baseline is what surfaces that before it
becomes a rewrite.
─────────────────────────────────────────
A Practical Cadence
─────────────────────────────────────────
EVERY COMMIT fixture suite, ~50 cases, under
two minutes, no model cost beyond
the agent itself
NIGHTLY full suite including recovery
variants, in the sandbox
PRE-RELEASE shadow run on real traffic,
read-only, plus the judge-scored
subset
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- An agent case must describe the environment and the acceptable path, not just input and output —
forbidden_toolsandmin_stepsturn safety and efficiency into mechanical checks. - Fixtures replayed from real traces make non-deterministic agents testable, and every production failure should become a permanent regression case.
- Recovery rate cannot be measured without injecting failures; each failure mode probes a different known weakness.
- Prefer mechanical assertions to LLM judgement, and gate CI on deltas against a baseline as well as absolute thresholds.
Concept Check
- Why should an unseen tool call return a
no_fixtureerror rather than an empty result? - Which failure mode tests the "negative conclusions" weakness from Module 2, and what does a failing agent do?
- Why is a delta gate necessary alongside an absolute one?
Next Chapter
→ Chapter 3: Observability and Tracing
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index