Agentic AI

Safety Control And Governance

Control and Containment

Every control in this chapter assumes the agent

JrCodex·7 min read

Jr Codex Agentic AI Notes

Level: Advanced Prerequisites: Chapter 2: Human-in-the-Loop Time to complete: ~25 minutes

Table of Contents

  1. Assume the Agent Will Be Wrong
  2. Least Privilege
  3. Enforcement in the Executor
  4. Sandboxing
  5. Kill Switches
  6. Audit Trails
  7. Summary & Next Steps

1. Assume the Agent Will Be Wrong

The Design Premise
─────────────────────────────────────────
  Every control in this chapter assumes the agent
  will, at some point, try to do the worst thing its
  tools permit — through injection, misalignment,
  or plain error.

  The question is never "will it behave?" It is
  "what happens when it does not?"

  That is the same premise as security engineering
  generally, and it produces the same answer:
  containment, not trust.
─────────────────────────────────────────

Where Controls Must Live
─────────────────────────────────────────
  ✗ IN THE PROMPT     the model may ignore it, and
                      injection can override it

  ✓ IN THE EXECUTOR   deterministic code the model
                      cannot influence

  ✓ IN THE           credentials that simply do not
    CREDENTIALS       grant the capability

  ✓ IN THE           the tool cannot reach what it
    ENVIRONMENT       cannot see

  Prompts express intent. Only the last three
  ENFORCE it. Use prompts as well — never instead.
─────────────────────────────────────────

2. Least Privilege

Four Dimensions to Narrow
─────────────────────────────────────────
  CAPABILITY   read vs write vs delete
  SCOPE        one record, one table, one tenant —
               not the database
  TIME         credentials that expire with the run
  RATE         how many calls, how fast
─────────────────────────────────────────

@dataclass(frozen=True)
class Grant:
    tool: str
    capability: Literal["read", "write", "delete"]
    scope: dict                            # e.g. {"tenant": "acme", "table": "orders"}
    max_calls: int
    expires_at: float
 
def grants_for(task) -> list[Grant]:
    """Issue the NARROWEST grants this specific task needs — per run, not per agent."""
    base = [Grant("get_customer", "read", {"tenant": task.tenant,
                                           "customer_id": task.customer_id},
                  max_calls=5, expires_at=time.time() + 600)]
    if task.kind == "refund":
        base.append(Grant("issue_refund", "write",
                          {"tenant": task.tenant, "max_amount_cents": 50_000},
                          max_calls=1, expires_at=time.time() + 600))
    return base
Per-Run Grants, Not Per-Agent Roles
─────────────────────────────────────────
  A static "support agent" role must be the union of
  everything any support task could need — which is
  far more than any single task needs.

  Issuing grants per RUN means a run handling
  cust_4821 physically cannot read cust_9000, even
  if injected to try.

  This turns a whole class of injection attacks into
  a permission error the agent reports.
─────────────────────────────────────────

3. Enforcement in the Executor

class GuardedExecutor:
    """Every control is checked HERE, where the model has no influence."""
 
    def __init__(self, tools, grants, policy, audit, bounds):
        self.tools, self.grants = tools, {g.tool: g for g in grants}
        self.policy, self.audit, self.bounds = policy, audit, bounds
        self.counts = {}
 
    def execute(self, call, state):
        tool = self.tools.get(call.name)
        if tool is None:
            return {"error": "unknown_tool", "message": f"No tool named {call.name}."}
 
        grant = self.grants.get(call.name)
        if grant is None:                                  # NOT GRANTED for this run
            self.audit.deny(call, "no grant")
            return {"error": "forbidden",
                    "message": f"{call.name} is not available for this task.",
                    "hint": "Use a different approach or report that you cannot proceed."}
 
        if time.time() > grant.expires_at:
            return {"error": "expired", "message": "This grant has expired."}
 
        used = self.counts.get(call.name, 0)
        if used >= grant.max_calls:
            return {"error": "rate_limited",
                    "message": f"{call.name} may be called {grant.max_calls} times per run."}
 
        if violation := scope_violation(call.args, grant.scope):    # THE key check
            self.audit.deny(call, violation)
            return {"error": "out_of_scope", "message": violation}
 
        needed, reason = self.policy.requires_approval(call, tool, state.confidence)
        if needed:
            return request_approval(call, reason, state)            # Chapter 2
 
        self.counts[call.name] = used + 1
        self.audit.allow(call)
        return tool.run(**call.args)
Denials Are Observations
─────────────────────────────────────────
  Every rejection returns a structured error the
  agent can act on, not an exception.

  The agent learns "I cannot do that here" and
  adapts — which is what you want, and is also how
  an injection attempt shows up in your audit log
  rather than in your data.
─────────────────────────────────────────

4. Sandboxing

When You Need One
─────────────────────────────────────────
  Any tool that executes agent-generated CODE:
  a Python interpreter, a shell, a SQL runner
  against a live database, a browser.

  These are the tools where "worst plausible
  arguments" (Chapter 1) is unbounded.
─────────────────────────────────────────

def run_sandboxed(code: str, timeout=10) -> dict:
    """Container isolation with everything unnecessary removed."""
    return docker.run(
        image="python:3.12-slim",
        command=["python", "-c", code],
        network_disabled=True,           # NO network — blocks exfiltration entirely
        read_only=True,                  # immutable root filesystem
        tmpfs={"/tmp": "size=64m"},      # scratch space, capped
        mem_limit="256m",
        pids_limit=64,                   # no fork bombs
        cap_drop=["ALL"],                # drop every Linux capability
        security_opt=["no-new-privileges"],
        user="nobody",
        timeout=timeout,
        remove=True,                     # destroyed after the call — no state carries over
    )
The Layers, in Order of Importance
─────────────────────────────────────────
  NETWORK OFF    the highest-value control. Without
                 network, code cannot exfiltrate
                 anything regardless of what it
                 does.

  EPHEMERAL      destroyed after each call, so
                 nothing persists between runs.

  RESOURCE CAPS  memory, CPU, processes, time — a
                 runaway becomes a timeout rather
                 than an outage.

  NO PRIVILEGES  dropped capabilities and a
                 non-root user, so a container
                 escape has nothing to escape with.
─────────────────────────────────────────

Do Not Build Your Own Sandbox
─────────────────────────────────────────
  Restricted `exec`, AST filtering and blocklists of
  dangerous builtins have all been bypassed
  repeatedly and generically.

  Use OS-level or VM-level isolation, or a hosted
  code-execution service. Process isolation is a
  solved problem; language-level sandboxing is not.
─────────────────────────────────────────

5. Kill Switches

Four Scopes
─────────────────────────────────────────
  RUN       stop one agent run
  TOOL      disable one tool fleet-wide (a tool
            started returning wrong data)
  AGENT     stop one agent type
  GLOBAL    stop everything

  You need all four. Fleet-wide problems cannot be
  fixed by stopping one run, and a single bad tool
  should not require stopping the fleet.
─────────────────────────────────────────

class KillSwitch:
    """Checked before EVERY tool call. Must be fast, and must not need a deploy."""
 
    def __init__(self, store): self.store = store        # Redis, or a config service
 
    def check(self, run_id, agent_type, tool_name):
        if self.store.get("halt:global"):
            raise Halted("global halt is active")
        if self.store.get(f"halt:agent:{agent_type}"):
            raise Halted(f"agent type {agent_type} is halted")
        if self.store.get(f"halt:tool:{tool_name}"):
            raise Halted(f"tool {tool_name} is disabled")
        if self.store.get(f"halt:run:{run_id}"):
            raise Halted("this run was stopped")
Three Requirements
─────────────────────────────────────────
  NO DEPLOY NEEDED   if stopping requires shipping
                     code, it is not a kill switch.
                     A config flag, effective in
                     seconds.

  CHECKED OFTEN      before every tool call, not
                     once at startup. A run already
                     in flight must stop.

  STATE PRESERVED    halting checkpoints and exits.
                     The run can be inspected and
                     resumed after the fix — halting
                     should not destroy the evidence.
─────────────────────────────────────────

6. Audit Trails

What Every Entry Records
─────────────────────────────────────────
  WHO      run id, agent type, the human on whose
           behalf it acted
  WHAT     tool, full arguments, result summary
  WHEN     timestamp
  WHY      the agent's stated reason, and the goal
  DECISION allowed, denied, or approved — and by
           whom
  RESULT   success, failure, or error
─────────────────────────────────────────

def record(audit, run, call, decision, actor=None, result=None):
    audit.append({
        "at": time.time(),
        "run_id": run.id, "agent": run.agent_type,
        "on_behalf_of": run.user_id,                 # ATTRIBUTION to a person
        "goal": run.goal,
        "tool": call.name, "args": redact(call.args),
        "reason": run.last_reasoning,
        "decision": decision,                        # allowed | denied | approved
        "approver": actor,
        "result": summarise(result),
        "trace_url": run.trace_url,
    })
Why Audit Is Not Just Tracing
─────────────────────────────────────────
  Tracing (Module 7, Chapter 3) is for ENGINEERS
  debugging behaviour. It is sampled, retained
  briefly, and may be redacted heavily.

  An audit log is for ACCOUNTABILITY. It is
  complete, immutable, retained per policy, and
  must answer "who authorised this action, and
  why" — including for actions that were DENIED.

  Denials matter most: a log full of denied
  out-of-scope calls is how you discover an
  injection attempt that your controls stopped.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Controls belong in the executor, the credentials and the environment — prompts express intent but cannot enforce it, and injection overrides them.
  • Issue grants per run rather than per agent, so a run physically cannot touch data outside its task even when injected to try.
  • Network-off, ephemeral, resource-capped OS isolation is the sandbox; language-level sandboxing has been bypassed generically and should not be built yourself.
  • Kill switches need four scopes, must work without a deploy, and must preserve state — and audit logs must record denials, which is how stopped attacks become visible.

Concept Check

  1. Why is a per-run grant meaningfully safer than a per-agent role, in injection terms?
  2. Which single sandbox control does the most work, and why?
  3. What does an audit log capture that a trace does not, and why do denied calls matter most?

Next Chapter

→ Chapter 4: Governance and Trust

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index