Agentic AI

Safety Control And Governance

Human-in-the-Loop

Choosing a Level Per ACTION, Not Per Agent

JrCodex·8 min read

Jr Codex Agentic AI Notes

Level: Advanced Prerequisites: Chapter 1: Agentic Risks & Failure Modes Time to complete: ~25 minutes

Table of Contents

  1. The Oversight Spectrum
  2. What Requires Approval
  3. Implementing an Approval Gate
  4. Designing the Approval Request
  5. Rubber-Stamping
  6. Escalation and Timeouts
  7. Summary & Next Steps

1. The Oversight Spectrum

Five Levels
─────────────────────────────────────────
  1. HUMAN IN CONTROL     the agent suggests; the
                          human does everything

  2. APPROVE EACH ACTION  the agent proposes; the
                          human approves each step

  3. APPROVE SOME         read freely; mutations
                          need approval
                          ── the practical default

  4. NOTIFY AFTER         acts freely; the human is
                          told, and can undo

  5. FULLY AUTONOMOUS     acts; nobody watches in
                          real time
─────────────────────────────────────────

Choosing a Level Per ACTION, Not Per Agent
─────────────────────────────────────────
  The common mistake is picking one level for the
  whole agent.

  A single agent should sit at level 5 for reads,
  level 3 for ordinary writes, and level 2 for
  anything irreversible or above a value threshold.

  One agent, three levels, chosen by what the action
  does.
─────────────────────────────────────────

2. What Requires Approval

The Four Triggers
─────────────────────────────────────────
  IRREVERSIBILITY   can it be undone? Sending,
                    deleting, publishing, paying —
                    no. This is the primary trigger
                    (Module 6, Chapter 2).

  VALUE             above a monetary or impact
                    threshold, even if reversible.

  SCOPE             affects many records or many
                    people at once. A refund is one
                    thing; 4,000 refunds is another.

  CONFIDENCE        the agent is uncertain, or a
                    self-consistency check
                    disagreed (Module 2, Chapter 2).
─────────────────────────────────────────

@dataclass
class ApprovalPolicy:
    always: set = field(default_factory=lambda: {"delete_account", "issue_payment"})
    value_threshold_cents: int = 10_000
    scope_threshold_records: int = 50
 
    def requires_approval(self, call, tool, confidence) -> tuple[bool, str]:
        if tool.name in self.always:
            return True, "always requires approval"
        if tool.irreversible:
            return True, "irreversible action"
        if (amount := call.args.get("amount_cents", 0)) > self.value_threshold_cents:
            return True, f"value {amount/100:.2f} over threshold"
        if call.estimated_records > self.scope_threshold_records:
            return True, f"affects {call.estimated_records} records"
        if confidence == "low":
            return True, "agent confidence is low"
        return False, ""
The Rule Behind the Triggers
─────────────────────────────────────────
  Approve where you cannot RECOVER, not where you
  are merely nervous.

  A reversible action with a good audit trail and a
  compensating action (Module 6, Chapter 2) can run
  unattended, because a mistake is fixable.

  An irreversible action cannot, at any confidence
  level — because the model has no calibrated
  confidence to threshold on (Module 2, Chapter 1).
─────────────────────────────────────────

3. Implementing an Approval Gate

Approval takes hours, so it requires the durable state from Module 6, Chapter 2.

async def step_with_approval(state, call, tool, policy, checkpointer, notifier):
    needed, reason = policy.requires_approval(call, tool, state.confidence)
 
    if not needed:
        return tool.execute(call)
 
    request = ApprovalRequest(
        run_id=state.run_id, call=call, reason=reason,
        rendered=render_for_human(call, tool, state),          # Section 4
        expires_at=time.time() + 24 * 3600,
    )
 
    state.status = "waiting_approval"
    state.pending_approval = asdict(request)
    checkpointer.save(state)                    # the run now survives ANY downtime
    await notifier.send(request)
 
    return APPROVAL_PENDING                     # the process EXITS here — nothing blocks
 
 
async def on_approval_decision(run_id, decision, actor, checkpointer, agent):
    """Resumed later by an event (Module 6, Chapter 1) — hours or days later."""
    state = checkpointer.latest(run_id)
    request = state.pending_approval
 
    audit.record(run_id=run_id, call=request["call"], decision=decision,
                 actor=actor, at=time.time())              # ALWAYS audit both outcomes
 
    if decision == "approved":
        result = agent.tools.execute(rebuild_call(request["call"]))
    else:
        result = {"error": "denied_by_operator",           # an OBSERVATION, not a crash
                  "message": f"{actor} denied this: {decision.note}",
                  "hint": "Do not retry. Choose a different approach or stop."}
 
    state.messages.append(tool_message(request["call"]["id"], result))
    state.status, state.pending_approval = "running", None
    checkpointer.save(state)
    return await agent.resume(state)
Denial Is an Observation
─────────────────────────────────────────
  Feeding the denial back as a tool result — with
  the reason and "do not retry" — lets the agent
  adapt: try a smaller refund, ask the user, or stop
  and explain.

  Treating denial as an exception throws away the
  work already done and tells the agent nothing.
─────────────────────────────────────────

4. Designing the Approval Request

The approval is only as good as what the reviewer can see in ten seconds.

What a Reviewer Needs
─────────────────────────────────────────
  WHAT       the action, in plain language, with
             the actual values
  WHY        the agent's reasoning, and the evidence
             it used
  IMPACT     what changes, how many records, how
             much money
  REVERSIBLE can this be undone? Say so explicitly
  CONTEXT    the original request and the key
             findings
  ALTERNATIVE what happens if denied
  TRACE      a link, for the reviewer who wants more
─────────────────────────────────────────

Rendered
─────────────────────────────────────────
  APPROVAL NEEDED — run 8841

  ACTION     Refund $340.00 to cust_4821
             (order ord_9912)
  REVERSIBLE No. Funds leave immediately.

  WHY        Customer reported the item never
             arrived. Carrier tracking shows
             "delivery exception" on 14 Aug.
             No prior refunds on this account.

  IMPACT     1 payment, $340.00, to the original
             card.

  IF DENIED  The agent will draft a request for
             more evidence instead.

  [Approve]  [Deny]  [Deny + note]   trace ↗
─────────────────────────────────────────

The Design Rule
─────────────────────────────────────────
  The reviewer must be able to decide WITHOUT
  opening the trace.

  If they have to read the transcript to judge, the
  gate is theatre — it will be approved unread, and
  you have added latency without adding safety.
─────────────────────────────────────────

5. Rubber-Stamping

The most likely way an approval system fails.

The Mechanism
─────────────────────────────────────────
  A reviewer approves 200 requests a day.
  199 are routine.
  They approve in two seconds each, by reflex.
  Number 200 is the one that mattered.

  The gate EXISTS. It is not providing oversight.
─────────────────────────────────────────

Four Countermeasures
─────────────────────────────────────────
  RAISE THE BAR      if approval rate is 99%+, the
                     threshold is too low. Approve
                     less; automate the routine
                     cases properly.

  MAKE UNUSUAL       flag requests that differ from
  REQUESTS LOOK      the norm — an unusual amount, a
  UNUSUAL            new tool, a first-time action
                     for this account.

  REQUIRE A REASON   for high-value actions, make
                     the approver type why. It is
                     friction, and friction is the
                     point.

  MEASURE THE        median approval time under five
  BEHAVIOUR          seconds means nobody is reading.
                     Track it as a safety metric.
─────────────────────────────────────────

def flag_anomalies(request, history) -> list[str]:
    flags = []
    amounts = [h.amount for h in history if h.tool == request.tool]
    if amounts and request.amount > percentile(amounts, 95):
        flags.append(f"amount is above the 95th percentile for {request.tool}")
    if request.tool not in {h.tool for h in history[-500:]}:
        flags.append("this tool has not been used recently")
    if request.account in {h.account for h in history[-20:] if h.denied}:
        flags.append("a recent request for this account was DENIED")
    return flags

6. Escalation and Timeouts

async def handle_expiry(state, request, checkpointer, notifier):
    """A pending approval must resolve. Waiting forever is a failure mode too."""
    if time.time() < request.expires_at:
        return
 
    if request.escalation_level == 0:                    # nudge the primary reviewer
        request.escalation_level, request.expires_at = 1, time.time() + 4 * 3600
        await notifier.remind(request)
 
    elif request.escalation_level == 1:                  # escalate to a second approver
        request.escalation_level, request.expires_at = 2, time.time() + 4 * 3600
        await notifier.escalate(request, to=request.backup_approver)
 
    else:                                                # DENY BY DEFAULT
        await on_approval_decision(state.run_id, "denied_timeout", actor="system",
                                   checkpointer=checkpointer, agent=agent)
Deny by Default, Never Approve by Default
─────────────────────────────────────────
  A timeout that auto-approves converts every
  unavailable reviewer into an unattended
  irreversible action — precisely the situation the
  gate exists to prevent.

  A timeout that denies is safe: the work is
  preserved in the checkpoint, the agent reports
  what it could not complete, and a human can
  resume it later.
─────────────────────────────────────────

Report the Queue
─────────────────────────────────────────
  Approval queue depth and wait time belong on the
  safety dashboard (Module 7, Chapter 3).

  A growing queue means the agent is blocked and
  nobody has noticed — a silent outage that looks
  like normal operation.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Choose an oversight level per action rather than per agent: autonomous for reads, gated for writes, always-approve for irreversible or high-value actions.
  • Approval takes hours, so it is impossible without durable state — the process must exit at the gate and resume from a checkpoint on an event.
  • Feed denials back as tool observations with a reason, so the agent can adapt rather than lose all prior work.
  • Rubber-stamping is the likeliest failure: if the approval rate is near 100% or median decision time is seconds, the gate is theatre — and timeouts must deny, never approve.

Concept Check

  1. Why is irreversibility a better approval trigger than the agent's own confidence?
  2. What makes an approval gate "theatre," and which two metrics detect it?
  3. Why must an approval timeout deny by default, and what happens to the work in progress?

Next Chapter

→ Chapter 3: Control and Containment

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index