Agentic AI

Workflows And Multi Agent Systems

State, Checkpointing & Durability

A 40-step agent run takes four minutes and costs

JrCodex·7 min read

Jr Codex Agentic AI Notes

Level: Advanced Prerequisites: Chapter 1: Workflow Patterns Time to complete: ~25 minutes

Table of Contents

  1. Why Durability Is the Threshold
  2. Designing the State Object
  3. Checkpointing
  4. Resuming Safely
  5. Time Travel and Forking
  6. Rollback and Compensation
  7. Summary & Next Steps

1. Why Durability Is the Threshold

The Arithmetic
─────────────────────────────────────────
  A 40-step agent run takes four minutes and costs
  real money.

  Without durability, ANY of the following discards
  all of it:
    - a deploy
    - a pod restart
    - a transient network failure
    - the user closing the tab
    - a rate limit at step 38

  With durability, each of those costs one step.
─────────────────────────────────────────

The Second Reason, Which Matters More
─────────────────────────────────────────
  Human approval is not a pause of seconds. It is a
  pause of HOURS — someone reads a Slack message
  after lunch.

  You cannot hold a process open for that. Human-in-
  the-loop (Module 8, Chapter 2) is IMPOSSIBLE
  without persisted state.

  Durability is not an optimisation. It is what
  makes an entire category of agent viable.
─────────────────────────────────────────

2. Designing the State Object

Everything needed to resume must be in the state. Nothing that matters may live only in local variables.

from dataclasses import dataclass, field, asdict
from typing import Literal
import json, time
 
@dataclass
class RunState:
    run_id: str
    goal: str
    status: Literal["running", "waiting_approval", "done", "failed"] = "running"
 
    messages: list = field(default_factory=list)          # short-term memory (Module 3)
    plan: dict | None = None                              # Module 4, Chapter 1
    scratchpad: dict = field(default_factory=dict)
 
    step: int = 0
    cost_cents: float = 0.0
    completed_tool_calls: dict = field(default_factory=dict)   # call_id -> result
 
    pending_approval: dict | None = None
    updated_at: float = field(default_factory=time.time)
 
    def to_json(self) -> str:
        return json.dumps(asdict(self))
 
    @classmethod
    def from_json(cls, raw: str) -> "RunState":
        return cls(**json.loads(raw))
Three Rules for State Design
─────────────────────────────────────────
  SERIALISABLE ONLY. No open connections, no file
  handles, no client objects. If it cannot be JSON,
  it cannot be checkpointed — reconstruct it on
  resume instead.

  RECORD COMPLETED SIDE EFFECTS. `completed_tool_
  calls` is what prevents re-executing a refund
  after a crash. This field is the difference
  between safe and dangerous resumption.

  INCLUDE THE BUDGET. Cost and step count must
  survive a restart, or every crash silently resets
  the bounds from Module 4, Chapter 4.
─────────────────────────────────────────

3. Checkpointing

class Checkpointer:
    """Persist after every step. The cost is a write; the benefit is the whole run."""
 
    def __init__(self, db): self.db = db
 
    def save(self, state: RunState) -> None:
        state.updated_at = time.time()
        self.db.execute(
            "INSERT INTO checkpoints (run_id, step, state, at) VALUES (?,?,?,?)",
            (state.run_id, state.step, state.to_json(), state.updated_at),
        )                                            # APPEND, never overwrite — Section 5
 
    def latest(self, run_id: str) -> RunState | None:
        row = self.db.query_one(
            "SELECT state FROM checkpoints WHERE run_id=? ORDER BY step DESC LIMIT 1",
            (run_id,))
        return RunState.from_json(row["state"]) if row else None
 
    def at_step(self, run_id: str, step: int) -> RunState | None:
        row = self.db.query_one(
            "SELECT state FROM checkpoints WHERE run_id=? AND step=?", (run_id, step))
        return RunState.from_json(row["state"]) if row else None
Where to Checkpoint
─────────────────────────────────────────
  AFTER every model call and BEFORE every tool
  execution.

  That ordering matters. The dangerous window is
  between deciding to act and recording the result —
  a crash there is where duplicate side effects come
  from.

  Checkpointing the DECISION before executing it
  means resumption knows what was in flight.
─────────────────────────────────────────

4. Resuming Safely

Naive resumption re-runs the step that was in flight. For a read that is waste; for a refund it is a second refund.

def resume(run_id, checkpointer, agent, tools):
    state = checkpointer.latest(run_id)
    if state is None or state.status in ("done", "failed"):
        return state
 
    while state.step < MAX_STEPS:
        reply = state.messages[-1] if pending(state) else agent.plan(state)
 
        if not reply.tool_calls:
            state.status = "done"
            checkpointer.save(state)
            return state
 
        for call in reply.tool_calls:
            if call.id in state.completed_tool_calls:        # ALREADY DONE before the crash
                result = state.completed_tool_calls[call.id] # replay the RECORDED result
            else:
                result = tools.execute(call)
                state.completed_tool_calls[call.id] = result  # record BEFORE continuing
                checkpointer.save(state)                      # and checkpoint immediately
 
            state.messages.append(tool_message(call.id, result))
 
        state.step += 1
        checkpointer.save(state)
    return state
The Two-Layer Protection
─────────────────────────────────────────
  LAYER 1   completed_tool_calls, checked before
            every execution. Handles the common case
            cleanly.

  LAYER 2   idempotency keys inside the tools
            themselves (Module 2, Chapter 3).
            Handles the crash that happens BETWEEN
            executing and recording — the gap layer
            1 cannot close.

  You need both. Layer 1 alone still has a window;
  layer 2 alone re-runs everything on every resume.
─────────────────────────────────────────

5. Time Travel and Forking

Because checkpoints are appended rather than overwritten, every prior state is still there.

def replay(run_id, checkpointer, upto=None):
    """Debug a non-deterministic run by walking its actual history."""
    for step in range(0, upto or MAX_STEPS):
        state = checkpointer.at_step(run_id, step)
        if state is None:
            break
        print(f"step {state.step}: cost={state.cost_cents:.2f} "
              f"last={summarise(state.messages[-1])}")
 
def fork(run_id, checkpointer, from_step, modification):
    """Branch from an earlier state with a change — A/B a fix without re-running."""
    state = checkpointer.at_step(run_id, from_step)
    state.run_id = f"{run_id}-fork-{from_step}"
    modification(state)                        # e.g. edit the prompt, swap a tool result
    checkpointer.save(state)
    return state.run_id
Why This Is the Only Real Debugging Tool
─────────────────────────────────────────
  An agent run is non-deterministic. Re-running it
  produces a DIFFERENT trajectory, so you cannot
  reproduce the bug you are chasing.

  Checkpoint history is the actual sequence that
  actually happened. Forking lets you change ONE
  thing at step 12 and see what would have followed.

  Without this you are reading logs and guessing.
  Module 7, Chapter 4 builds on it.
─────────────────────────────────────────

6. Rollback and Compensation

State can be rolled back. The world cannot.

The Asymmetry
─────────────────────────────────────────
  Restoring a checkpoint from step 8 undoes the
  agent's memory of steps 9-12.

  It does NOT un-send the email sent at step 10.

  Anything with an external effect needs a
  COMPENSATING ACTION, not a rollback.
─────────────────────────────────────────

@dataclass
class Effect:
    tool: str
    args: dict
    result: dict
    compensate: str | None          # the tool that UNDOES this, if one exists
 
COMPENSATIONS = {
    "issue_refund":  "reverse_refund",
    "create_ticket": "close_ticket",
    "send_email":    None,          # NO compensation exists — this is the point
}
 
def rollback_to(state, checkpointer, step, tools):
    """Restore state, and compensate for effects that cannot be un-done by restoring."""
    target = checkpointer.at_step(state.run_id, step)
 
    for effect in reversed(effects_after(state, step)):     # REVERSE order, like a stack
        if effect.compensate:
            tools.execute(effect.compensate, undo_args(effect))
        else:
            log.warning("uncompensatable effect: %s — human notification required",
                        effect.tool)
            notify_operator(effect)                          # a person must handle it
    return target
The Design Consequence
─────────────────────────────────────────
  Classify every tool by whether it is reversible
  BEFORE you build the system.

  REVERSIBLE      compensate automatically
  IRREVERSIBLE    require human approval BEFORE
                  execution (Module 8, Chapter 2) —
                  because there is no "after"

  An irreversible action taken without approval is
  a permanent consequence of a probabilistic
  decision. That is the situation the whole of
  Module 8 exists to prevent.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Durability turns any restart, deploy or rate limit from a lost run into a lost step, and it is what makes hours-long human approval possible at all.
  • State must be fully serialisable, must record completed side effects, and must carry the cost and step budget across restarts.
  • Checkpoint after the model call and before tool execution, and protect resumption with both a completed-calls record and idempotency keys inside the tools.
  • Append-only checkpoints give replay and forking, which is the only practical way to debug a non-deterministic run — and rollback restores state but never un-does external effects.

Concept Check

  1. Why does the checkpoint need to happen before tool execution rather than only after?
  2. Why are completed_tool_calls and tool-level idempotency keys both required rather than either alone?
  3. An agent sent an email at step 10 and you roll back to step 8. What actually happens, and what must the system do?

Next Chapter

→ Chapter 3: Multi-Agent Architectures

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index