Agentic AI

Evaluating And Operating Agents

Observability and Tracing

interleaved with every other concurrent run.

JrCodex·7 min read

Jr Codex Agentic AI Notes

Level: Advanced Prerequisites: Chapter 2: Building an Agent Eval Set Time to complete: ~20 minutes

Table of Contents

  1. Why Logs Are Not Enough
  2. The Trace Model
  3. Instrumenting an Agent
  4. LangSmith and OpenTelemetry
  5. Cost Attribution
  6. Dashboards and Alerts
  7. Summary & Next Steps

1. Why Logs Are Not Enough

The Problem With Log Lines
─────────────────────────────────────────
  A 20-step agent produces ~200 log lines,
  interleaved with every other concurrent run.

  Even filtered to one run, a flat log cannot show:
    - which model call PRODUCED a given tool call
    - how long each step took RELATIVE to the whole
    - what the model actually SAW at step 12
    - where the cost went
    - which sub-agent a call belonged to

  Agent execution is a TREE. A log is a list.
─────────────────────────────────────────

What Tracing Adds
─────────────────────────────────────────
  A trace preserves the STRUCTURE: nested spans with
  timing, inputs, outputs and metadata.

  It answers "what happened, in what order, nested
  how, costing what" — which is the only question
  worth asking about a failed agent run.
─────────────────────────────────────────

2. The Trace Model

The Hierarchy
─────────────────────────────────────────
  TRACE  (one run)
   └─ SPAN agent_loop
       ├─ SPAN llm_call step=1        1.2s  $0.004
       │    in:  messages[3]
       │    out: tool_calls[1]
       ├─ SPAN tool get_customer      0.1s
       │    in:  {"id": "cust_4821"}
       │    out: {...}
       ├─ SPAN llm_call step=2        1.8s  $0.006
       ├─ SPAN tool list_orders       0.3s
       │    └─ SPAN retrieval         0.2s   ← nested
       └─ SPAN llm_call step=3        2.1s  $0.009
            out: final answer
─────────────────────────────────────────

The Four Things Every Span Needs
─────────────────────────────────────────
  IDENTITY    trace id, span id, parent span id
  TIMING      start, end, duration
  PAYLOAD     input and output (redacted as needed)
  METADATA    model, tokens, cost, error, step
              number, agent name

  Parent span id is what turns a list into a tree.
  Omit it and you have logs again.
─────────────────────────────────────────

3. Instrumenting an Agent

import time, uuid, contextlib
 
class Tracer:
    def __init__(self, sink): self.sink, self.stack = sink, []
 
    @contextlib.contextmanager
    def span(self, name, **metadata):
        span = {
            "trace_id": self.stack[0]["trace_id"] if self.stack else str(uuid.uuid4()),
            "span_id": str(uuid.uuid4()),
            "parent_id": self.stack[-1]["span_id"] if self.stack else None,
            "name": name, "start": time.time(), "metadata": metadata,
        }
        self.stack.append(span)
        try:
            yield span
        except Exception as e:
            span["error"] = f"{type(e).__name__}: {e}"
            raise
        finally:
            span["end"] = time.time()
            span["duration"] = span["end"] - span["start"]
            self.stack.pop()
            self.sink.write(span)                    # emit on close, so timing is real
def traced_agent(client, goal, tools, tracer):
    with tracer.span("agent_run", goal=goal) as run:
        messages = [{"role": "user", "content": goal}]
 
        for step in range(MAX_STEPS):
            with tracer.span("llm_call", step=step) as s:
                reply = client.chat.completions.create(
                    model="gpt-4o", messages=messages, tools=tools.schemas)
                s["metadata"].update(
                    model="gpt-4o",
                    input_tokens=reply.usage.prompt_tokens,
                    output_tokens=reply.usage.completion_tokens,
                    cost_cents=price(reply.usage),
                    prompt_hash=hash_messages(messages),   # HASH, not the raw prompt
                )
 
            if not reply.choices[0].message.tool_calls:
                run["metadata"]["outcome"] = "completed"
                return reply.choices[0].message.content
 
            messages.append(reply.choices[0].message)
            for call in reply.choices[0].message.tool_calls:
                with tracer.span("tool", tool=call.function.name) as t:
                    result = tools.execute(call)
                    t["metadata"]["error"] = result.get("error") if isinstance(result, dict) else None
                messages.append(tool_message(call.id, result))
 
        run["metadata"]["outcome"] = "step_limit"
Redaction Is Not Optional
─────────────────────────────────────────
  Traces capture prompts and tool results, which
  means they capture customer data.

  Store a prompt HASH by default and the full text
  only where policy allows. Redact known PII fields
  in tool outputs before they reach the sink.

  A trace store is a data store, with all the
  obligations that implies.
─────────────────────────────────────────

4. LangSmith and OpenTelemetry

# LangSmith — automatic for LangChain/LangGraph, decorator for anything else.
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "support-agent"
 
from langsmith import traceable
 
@traceable(run_type="tool", name="get_customer")
def get_customer(customer_id: str) -> dict:
    ...
 
@traceable(run_type="chain", name="investigate")
def investigate(ticket):
    ...                                     # nesting is captured automatically
# OpenTelemetry — vendor-neutral, lands in your existing APM alongside service traces.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
 
with tracer.start_as_current_span("agent_run") as span:
    span.set_attribute("gen_ai.system", "openai")
    span.set_attribute("gen_ai.request.model", "gpt-4o")
    span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
Choosing Between Them
─────────────────────────────────────────
  LANGSMITH        purpose-built for LLM work.
                   Prompt diffing, run comparison,
                   dataset creation from traces, and
                   evaluation wired in. Best
                   developer experience by some
                   margin.

  OPENTELEMETRY    vendor-neutral, and agent spans
                   sit in the SAME trace as your HTTP
                   and database spans. Best when the
                   agent is one component of a larger
                   system.

  Doing both is common and reasonable: OTel for
  operations, LangSmith for agent development.
─────────────────────────────────────────

5. Cost Attribution

def attribute(trace) -> dict:
    """Answer 'where did the money go' along every axis that matters."""
    llm = [s for s in trace.spans if s["name"] == "llm_call"]
 
    by_step  = {s["metadata"]["step"]: s["metadata"]["cost_cents"] for s in llm}
    by_agent = {}
    for s in llm:
        agent = s["metadata"].get("agent", "main")
        by_agent[agent] = by_agent.get(agent, 0) + s["metadata"]["cost_cents"]
 
    input_tokens = sum(s["metadata"]["input_tokens"] for s in llm)
    tool_bytes = sum(len(str(s.get("output", ""))) for s in trace.spans
                     if s["name"] == "tool")
 
    return {"total_cents": sum(by_step.values()),
            "by_step": by_step, "by_agent": by_agent,
            "input_tokens": input_tokens,
            "tool_output_bytes": tool_bytes,        # the usual culprit — Module 2, Ch.1
            "growth": cost_growth_per_step(by_step)}
The Two Diagnostics This Gives You
─────────────────────────────────────────
  COST GROWTH PER STEP
    Rising sharply means the context is
    accumulating — a memory management problem
    (Module 3), not a model problem.

  TOOL OUTPUT BYTES vs INPUT TOKENS
    If tool bytes dominate, your perception layer is
    not truncating enough. This is the single most
    common cause of an expensive agent, and it is
    invisible without this measurement.
─────────────────────────────────────────

6. Dashboards and Alerts

Four Dashboards
─────────────────────────────────────────
  RELIABILITY   completion rate, recovery rate,
                error rate BY TYPE (rate limit,
                timeout, tool error, budget) —
                aggregate error counts hide which
                problem you have

  EFFICIENCY    p50/p95 steps per run, cost per
                SUCCESS, tokens per run, cache hit
                rate

  QUALITY       eval scores per release, drift
                against the stored baseline,
                thumbs/regeneration from users

  SAFETY        unauthorised action COUNT, approval
                queue depth and wait time, budget
                breach count
─────────────────────────────────────────

Alerts That Actually Fire Usefully
─────────────────────────────────────────
  COST PER USER PER DAY over threshold
    The silent failure. Nothing is down, latency is
    fine, errors are zero — and the bill is 40x.
    An injection loop or a retry bug looks exactly
    like this.

  STEP COUNT p95 rising
    Agents are wandering more than they used to.
    An early warning that a prompt or tool change
    degraded selection.

  RECOVERY RATE falling
    A tool's error messages changed and the agent no
    longer understands them.

  ANY unauthorised action
    Page someone. This is not a metric to watch.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Agent execution is a tree and logs are a list; the parent span id is what makes a trace answer questions a log cannot.
  • Every span needs identity, timing, payload and metadata — and traces capture customer data, so redaction and prompt hashing are part of the design.
  • Cost attribution by step and by agent gives two diagnostics: rising per-step cost means a memory problem, and dominant tool-output bytes mean perception is not truncating.
  • Alert on cost per user per day, because a runaway agent looks perfectly healthy on every conventional signal.

Concept Check

  1. What can a trace answer that a filtered log cannot, and why?
  2. Per-step cost is rising steeply through a run. Which module's problem is that, and which is it not?
  3. Why is "cost per user per day" the alert most likely to catch a prompt-injection loop?

Next Chapter

→ Chapter 4: Debugging and Error Recovery

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Agentic AI Index