Safety Control And Governance
Governance and Trust
The AI Notes cover AI regulation generally. Agents raise three questions that prediction systems do not.
JrCodex·7 min read
Jr Codex Agentic AI Notes
Level: Advanced Prerequisites: Chapter 3: Control and Containment; AI Notes, Module 6, Chapter 5 Time to complete: ~20 minutes
Table of Contents
- Why Agents Attract Regulation
- The EU AI Act, Practically
- The NIST AI Risk Management Framework
- Accountability and Disclosure
- An Agent System Card
- The Deployment Checklist
- Summary & Next Steps
1. Why Agents Attract Regulation
The AI Notes cover AI regulation generally. Agents raise three questions that prediction systems do not.
The Three Agent-Specific Questions
─────────────────────────────────────────
ATTRIBUTION a model produced an output and a
human acted on it — the human
decided. An agent ACTED. Who
decided?
FORESEEABILITY a classifier's outputs are
enumerable. An agent's action
sequences are not, so "we did not
anticipate that" is both more true
and less defensible.
MEANINGFUL regulation frequently requires human
OVERSIGHT oversight of consequential
decisions. An agent acting for hours
unattended has to demonstrate what
that oversight consisted of.
─────────────────────────────────────────
The Practical Consequence
─────────────────────────────────────────
You will be asked to EVIDENCE your controls, not
merely to have them.
Everything in Chapters 2 and 3 — approval records,
audit logs, grants, kill switches — is also the
documentation that answers a regulator or an
auditor. Build them once, use them twice.
─────────────────────────────────────────
2. The EU AI Act, Practically
The Risk Tiers
─────────────────────────────────────────
UNACCEPTABLE prohibited outright — social
scoring, certain biometric uses,
manipulative systems
HIGH RISK permitted with substantial
obligations — employment, credit,
education, essential services, law
enforcement, critical
infrastructure
LIMITED RISK transparency obligations — users
must know they are dealing with AI
MINIMAL RISK most business automation; no
specific obligations
─────────────────────────────────────────
What HIGH RISK Requires
─────────────────────────────────────────
- a documented risk management system
- data governance for training and inputs
- technical documentation and record-keeping
(your audit log, Chapter 3)
- automatic logging of operation
- transparency to users about capabilities and
limits
- HUMAN OVERSIGHT that can intervene and stop the
system (Chapter 2's gates, Chapter 3's kill
switch)
- accuracy, robustness and cybersecurity measures
─────────────────────────────────────────
The Determining Question
─────────────────────────────────────────
It is the USE CASE, not the technology, that sets
the tier.
A support agent that answers questions is minimal
risk. The SAME agent, deciding who receives a
refund in a regulated financial product, may not
be.
Classify by what the agent DECIDES and whom it
AFFECTS — before you build, because high-risk
obligations shape the architecture.
─────────────────────────────────────────
A necessary note: this is a practitioner's orientation, not legal advice, and obligations phase in over time. Confirm current requirements for your jurisdiction and use case with someone qualified.
3. The NIST AI Risk Management Framework
Voluntary, widely adopted, and genuinely useful as an organising structure.
The Four Functions
─────────────────────────────────────────
GOVERN policies, roles, accountability.
WHO owns this agent? Who can stop it?
MAP context and risk identification.
Chapter 1's risk register.
MEASURE analysis and tracking.
Module 7's evaluation and metrics.
MANAGE prioritise, respond, recover.
Chapters 2 and 3's controls, plus
incident response.
─────────────────────────────────────────
Why It Maps Cleanly to This Curriculum
─────────────────────────────────────────
GOVERN ── this chapter
MAP ── Module 8, Chapter 1
MEASURE ── Module 7
MANAGE ── Module 8, Chapters 2-3
If you have followed the curriculum, you have most
of an AI RMF profile already. What is usually
missing is GOVERN — named ownership and written
policy, which is organisational rather than
technical.
─────────────────────────────────────────
4. Accountability and Disclosure
Naming an Owner
─────────────────────────────────────────
Every deployed agent needs a named human who:
- authorised the deployment
- can stop it (and knows how)
- is accountable for its actions
- reviews its incidents
"The platform team" is not a name. An agent
without a named owner is an agent nobody will stop
at 2am.
─────────────────────────────────────────
What to Disclose, and to Whom
─────────────────────────────────────────
TO USERS INTERACTING WITH IT
that it is an AI system; what it can and cannot
do; how to reach a human
TO PEOPLE AFFECTED BY ITS ACTIONS
that an automated system took the action; how to
contest it; what recourse exists
TO OPERATORS
the system card (Section 5), the runbook, the
kill-switch procedure
The second is the one most often missed, and it is
the one regulation cares most about.
─────────────────────────────────────────
ACTION_NOTICE = """This action was taken by an automated system.
What: {action}
When: {when}
Why: {reason}
Reviewed: {approver} # or "not reviewed by a human"
Contest: {contest_url}
Reference: {run_id}"""5. An Agent System Card
One document per deployed agent, kept current. It is what someone reads at 2am and what an auditor asks for.
The Template
─────────────────────────────────────────
IDENTITY
name, version, owner, deployment date
PURPOSE
what it does; what it explicitly does NOT do
CAPABILITIES
tools, with capability and scope per tool
data it can read; systems it can write to
AUTONOMY
which actions are autonomous
which require approval, and by whom
which are prohibited
RISK CLASSIFICATION
EU AI Act tier and the reasoning
lethal trifecta status and which leg is broken
CONTROLS
grants model, sandboxing, rate limits
kill switch scopes and how to use them
per-run and per-day budgets
EVALUATION
eval set size and composition
current scorecard (Module 7, Chapter 1)
known failure modes and limitations
OPERATIONS
dashboards, alerts, on-call owner
incident history and resolutions
─────────────────────────────────────────
Two Sections That Earn Their Place
─────────────────────────────────────────
"WHAT IT DOES NOT DO"
Prevents scope creep. When someone asks the
agent to do something new, the card is the
record of what was actually assessed.
"KNOWN FAILURE MODES"
Writing these down is what turns a surprise into
an anticipated event. It is also the most
honest thing in the document, and the section
reviewers trust the card by.
─────────────────────────────────────────
6. The Deployment Checklist
BEFORE FIRST DEPLOYMENT
─────────────────────────────────────────
□ Risk register complete (Chapter 1)
□ Lethal trifecta assessed; a leg broken
□ Every tool scoped to least privilege
□ Irreversible actions gated on approval
□ Grants issued per run, not per agent
□ Code execution sandboxed, network off
□ Kill switch tested at all four scopes
□ Audit log capturing allows AND denies
□ Named owner, reachable, knows the runbook
□ System card written
BEFORE SCALING UP
─────────────────────────────────────────
□ Eval set from real traces (Module 7, Ch.2)
□ Scorecard baseline recorded; CI gates set
□ Tracing wired; four dashboards live
□ Cost per user per day alert configured
□ Circuit breaker for fleet-wide failure
□ Approval queue depth monitored
□ Rubber-stamp metrics tracked
□ Incident response runbook written and
rehearsed once
ONGOING
─────────────────────────────────────────
□ Every incident becomes a regression case
□ Scorecard reviewed each release, deltas gated
□ Approval thresholds revisited quarterly
□ System card updated when tools change
□ Grants re-audited when scope changes
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Agents raise attribution, foreseeability and meaningful-oversight questions that prediction systems do not, and you will be asked to evidence controls rather than merely have them.
- The EU AI Act's tier is set by use case, not technology — the same agent can be minimal or high risk depending on what it decides and whom it affects.
- The NIST functions map onto this curriculum: govern is this chapter, map is Chapter 1, measure is Module 7, manage is Chapters 2–3; govern is what teams usually lack.
- Every agent needs a named human owner and a system card whose "what it does not do" and "known failure modes" sections are the ones that matter most.
Concept Check
- Why does an agent raise an attribution question that a recommendation model does not?
- The same support agent is minimal risk in one deployment and potentially high risk in another. What determines which?
- Which NIST function does a team following this curriculum most often lack, and why is it not a technical gap?
Module 8 Complete — What's Next
You now have the full picture: reasoning, memory, planning, frameworks, orchestration, evaluation and control. Module 9 assembles all of it into one working system, then closes the curriculum.
Next Module
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index