Agentic AI

Capstone And Beyond

Designing the System

A RESEARCH-AND-REPORT AGENT for an internal team.

JrCodex·7 min read

Jr Codex Agentic AI Notes

Level: Advanced Prerequisites: All previous modules Time to complete: ~25 minutes

Table of Contents

  1. The Brief
  2. Choosing the Agentic Level
  3. The Tool Set
  4. The Risk Assessment
  5. The Architecture
  6. The Decisions, Summarised
  7. Summary & Next Steps

1. The Brief

The System
─────────────────────────────────────────
  A RESEARCH-AND-REPORT AGENT for an internal team.

  IN    a research question, and a depth setting
  OUT   a sourced report, every claim citing a
        retrieved document or a query result

  It may search an internal document store, query a
  metrics database, and fetch approved external
  pages. It may publish the finished report to the
  team wiki — with approval.
─────────────────────────────────────────

Why This Task
─────────────────────────────────────────
  It genuinely requires an agent (Section 2), it
  touches every module, and it has the property
  that makes agents hard: the necessary steps depend
  on what earlier steps find.

  It also has a real irreversible action — publishing
  — which forces the Module 8 machinery to be more
  than decorative.
─────────────────────────────────────────

2. Choosing the Agentic Level

Apply Module 1, Chapter 2's test before anything else.

The Test
─────────────────────────────────────────
  "Could I write the sequence of steps before
   seeing the input?"

  For "summarise this document" ── YES. Not an
  agent. Write a chain.

  For "why did enterprise churn rise in Q3?" ── NO.
  Whether to query the metrics DB depends on what
  the documents say. Whether to search externally
  depends on whether internal sources answered it.

  ──► level 5: a PLANNING agent (Module 4).
─────────────────────────────────────────

Why Not Multi-Agent
─────────────────────────────────────────
  Check Module 6, Chapter 5's three legitimate
  reasons:

  CONTEXT ISOLATION   marginal — sources are
                      summarised before entering
                      context anyway
  TOOL SET SIZE       6 tools. Far below the ~20
                      threshold.
  PARALLELISM         yes, sub-questions are
                      independent

  Only one of three, and parallelism is obtainable
  from a PLAN with independent steps (Module 4,
  Chapter 1) without paying for coordination.

  ──► ONE agent. Revisit only if measurement says
      otherwise.
─────────────────────────────────────────

3. The Tool Set

TOOLS = [
    # ---- read-only, unrestricted ----
    Tool("search_docs",
         "Semantic search over the internal document store. "
         "USE FOR: policy, product, and historical context. "
         "DO NOT USE FOR: numeric metrics — use query_metrics. "
         "Returns up to 5 excerpts with doc ids. Read-only, ~300ms.",
         capability="read", irreversible=False),
 
    Tool("get_document",
         "Fetch one full document by id, from a search result. "
         "USE FOR: reading a promising excerpt in full. "
         "Returns up to 8,000 characters, truncated with a marker.",
         capability="read", irreversible=False),
 
    Tool("query_metrics",
         "Run a PARAMETERISED metrics query. "
         "USE FOR: counts, rates, time series. "
         "DO NOT USE FOR: free-form SQL — only the named queries below. "
         "Returns rows plus the query that ran.",
         capability="read", irreversible=False),
 
    Tool("fetch_url",
         "Fetch an approved external page. "
         "USE FOR: public documentation and vendor pages on the allowlist. "
         "Returns extracted text, wrapped as untrusted content.",
         capability="read", irreversible=False, returns_untrusted=True),
 
    # ---- state-changing ----
    Tool("save_draft",
         "Save the report draft. Reversible; versions are kept.",
         capability="write", irreversible=False),
 
    Tool("publish_report",
         "Publish to the team wiki. Notifies the channel.",
         capability="write", irreversible=True, requires_approval=True),
]
Three Deliberate Choices
─────────────────────────────────────────
  NAMED QUERIES, NOT SQL
    `query_metrics` exposes parameterised queries,
    not `run_sql`. Module 8, Chapter 1's excessive
    functionality, avoided by design.

  fetch_url IS ALLOWLISTED
    Arbitrary fetch is an open door for untrusted
    content from anywhere.

  publish IS THE ONLY IRREVERSIBLE TOOL
    Exactly one gate to defend, and drafting is free.
    Keeping the irreversible surface to one tool is
    the single best structural safety decision here.
─────────────────────────────────────────

4. The Risk Assessment

The Lethal Trifecta (Module 8, Chapter 1)
─────────────────────────────────────────
  PRIVATE DATA         yes — internal docs, metrics
  UNTRUSTED CONTENT    yes — fetch_url
  EXTERNAL COMMS       publish_report notifies a
                       channel

  All three present. A leg MUST be broken.
─────────────────────────────────────────

How We Break It
─────────────────────────────────────────
  1. publish_report requires HUMAN APPROVAL, and the
     approval renders the full report for review. An
     injected instruction has to survive a human
     reading the output.

  2. fetch_url results are wrapped as untrusted and
     may only be CITED, never executed as
     instructions. The report format requires every
     claim to carry a source id, so injected text
     cannot become an unsourced assertion.

  3. fetch_url is ALLOWLISTED, so an injection must
     first live on a domain we approved.

  Three independent barriers on one leg. Any one may
  fail; all three failing silently is unlikely.
─────────────────────────────────────────

Goal Specification (Module 8, Chapter 1)
─────────────────────────────────────────
  SUCCESS
    - the question is answered
    - every claim cites a retrieved source
    - contradictions between sources are surfaced,
      not resolved silently
    - gaps are stated explicitly

  NOT SUCCESS, even if it looks complete
    - a fluent report with unsourced claims
    - answering an easier adjacent question
    - omitting a sub-question because it was hard
    - presenting one source as consensus
─────────────────────────────────────────

5. The Architecture

The Flow
─────────────────────────────────────────
  question
     │
     ▼
  [1] LOAD MEMORY          prior findings, lessons
     │                     (Module 3, Chapter 3)
     ▼
  [2] PLAN                 3-6 sub-questions, with
     │                     dependencies
     │                     (Module 4, Chapter 1)
     ▼
  [3] EXECUTE ──────────►  per sub-question, a SHORT
     │   (parallel where   ReAct loop, own context
     │    independent)     (Module 4, Chapter 2)
     ▼
  [4] CHECK COVERAGE       every sub-question
     │                     answered? gaps named?
     │                     (Module 4, Chapter 4)
     │        └─ gaps ──► RE-PLAN (max 2)
     ▼
  [5] SYNTHESISE           report, every claim sourced
     │
     ▼
  [6] VERIFY               do all cited ids resolve?
     │                     (Module 4, Chapter 3 —
     │                      a LEVEL 1 check)
     ▼
  [7] SAVE DRAFT           reversible, always
     │
     ▼
  [8] APPROVAL ──────────► human reviews, publishes
     │                     (Module 8, Chapter 2)
     ▼
  [9] WRITE MEMORY         lessons, if any
                           (Module 3, Chapter 3)
─────────────────────────────────────────

Where Each Module Lands
─────────────────────────────────────────
  Module 2   tool descriptions, ReAct in step 3
  Module 3   memory at steps 1 and 9; scratchpad
             throughout
  Module 4   plan at 2, execute at 3, coverage at 4,
             verification at 6
  Module 5   LangGraph, for step 8's durable pause
  Module 6   parallel execution at 3, checkpointing
             throughout
  Module 7   tracing on every step; the eval set
  Module 8   approval at 8, grants and audit
             throughout
─────────────────────────────────────────

6. The Decisions, Summarised

The Design Record
─────────────────────────────────────────
  Level 5 planning agent, not a chain
      because required steps depend on findings

  ONE agent, not multi-agent
      only 1 of 3 legitimate reasons applies, and
      parallelism comes from the plan instead

  LangGraph, not a bare loop
      because approval takes hours and needs durable
      state — the one requirement that forces a
      framework (Module 5, Chapter 5)

  Named queries, not free SQL
      blast radius of the worst tool

  One irreversible tool, gated
      one gate to defend

  Citation-required output format
      turns verification into a LEVEL 1 mechanical
      check instead of an LLM judgement
─────────────────────────────────────────

The Choice Worth Copying
─────────────────────────────────────────
  Requiring every claim to carry a source id is the
  highest-leverage decision in this design.

  It makes verification mechanical, makes injection
  hard to weaponise, and makes the report
  reviewable — three problems solved by one
  constraint on the output format.

  This is the Gen AI Notes' structured-output
  argument, applied at the system level.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Apply the level test first: this task needs an agent because the necessary steps depend on what earlier steps find, and needs a plan because it decomposes into independent sub-questions.
  • Only one of the three legitimate multi-agent reasons applied, and the parallelism it offered was obtainable from a dependency graph without coordination cost.
  • The trifecta was broken with three independent barriers on one leg — approval, citation-required output, and an allowlist — rather than one control trusted absolutely.
  • Constraining the output format so every claim cites a source solves verification, injection resistance and reviewability at once.

Concept Check

  1. Why is a single agent the right choice here despite the task having clearly separable sub-questions?
  2. Which single requirement forced the use of a framework, and why can it not be avoided?
  3. Explain how requiring source ids on every claim contributes to injection resistance.

Next Chapter

→ Chapter 2: Building the Agent

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index