Skip to content
← All guides

Evaluation & Operations

Flagship field guide

MDX

Agent Tracing and Observability

What to record across prompts, model decisions, tool calls, state changes, approvals, and outcomes.

4 min readAgentic Systems Editorial Team

Editorial review: clarity, operational relevance, safety boundaries, and source quality.

Capture the trajectory

A useful trace connects the user goal to model calls, tool requests, tool results, state transitions, and the final outcome. Correlation identifiers make a distributed task possible to reconstruct.

A trace should form one causal timeline: user request, normalized goal, model calls, tool proposals, validation decisions, tool results, state transitions, approvals, and final verification. Record model, prompt, tool, and schema versions. Without them, two visually similar runs may be impossible to compare.

Protect sensitive data

Prompts and tool results may contain secrets or personal information. Redact at collection time, restrict trace access, and set retention rules that match the data's sensitivity.

Observability can create a second sensitive-data store. Classify fields before collection, redact at the boundary, hash or tokenize identifiers where possible, and apply role-based views. Never rely on operators to notice secrets later. Retention should reflect debugging value and legal requirements, not an indefinite default.

Observe product outcomes

Token counts and tool latency explain system behavior, but business metrics show whether the agent helped. Track completion, correction, escalation, abandonment, and downstream error rates.

Operational dashboards should connect technical signals to outcomes. Track completion and escalation by task type alongside tool errors, loop length, cost, and latency. Alert on changes in distributions—such as repeated calls or a new denied-tool pattern—because agent failures often appear as unusual sequences before totals breach a threshold.

Practical example

Tracing a failed refund investigation

The trace shows that retrieval returned an obsolete policy, the model proposed a valid-looking refund, and the policy service denied it. Because source version, denial code, and subsequent action are recorded, the team can fix indexing and confirm the guard contained the error. A final-answer log alone would show only an apology and hide both the cause and the successful control.

Field checklist

Apply it in practice

  • Use one correlation ID across the full trajectory.
  • Record component versions and structured stop reasons.
  • Redact sensitive data before traces are stored.
  • Monitor sequence anomalies and user outcomes together.

Decision framework

Questions to answer before you build

A useful trace reconstructs the causal path from goal to verified outcome while protecting the sensitive data that prompts and tools naturally collect.

Can one task be reconstructed?

Connect model calls, tool proposals, validation, results, state changes, approvals, and final verification with one correlation identifier.

Which versions influenced behavior?

Record model, prompt, tool, schema, retrieval-index, and policy versions so changes can be compared rather than guessed.

What should trigger an alert?

Watch distributions and sequences such as repeated calls, new denial patterns, rising corrections, or longer paths—not only aggregate error rates.

Common failure signals

Watch for these warning signs

  • Logging final answers without the actions and observations that produced them.
  • Creating an unrestricted secondary store of personal data and secrets.
  • Monitoring tokens and latency without connecting them to user completion and downstream errors.

Field manual

Implementation blueprint

  1. 01

    Define the run spine

    Give the task, model turns, tool proposals, approvals, state changes, and verification events one correlation identifier.

    Deliverable: A causal timeline rather than disconnected model logs.

  2. 02

    Record decision versions

    Attach model, prompt, tool schema, policy, retrieval index, and evaluator versions to every run.

    Deliverable: The ability to compare behavior changes without guessing what changed.

  3. 03

    Capture outcome semantics

    Store terminal state, evidence, downstream effect, human correction, and reason codes.

    Deliverable: Operational metrics tied to user outcomes rather than token counts alone.

  4. 04

    Minimize sensitive data

    Classify, redact, tokenize, restrict, and expire trace fields at collection time.

    Deliverable: Useful observability without a shadow copy of every secret and personal record.

Reusable working artifact

Minimum viable trace event

Events should be small, typed, ordered, and linked to the state transition they caused.

{
  "run_id": "run_7f2",
  "sequence": 6,
  "event": "tool_result",
  "timestamp": "2026-07-21T10:30:12Z",
  "task_type": "order_investigation",
  "component_versions": {
    "model": "routing-policy-v3",
    "prompt": "support-18",
    "tool_schema": "order-lookup-4"
  },
  "action": { "tool": "find_order", "argument_fingerprint": "sha256:…" },
  "result": { "code": "found", "latency_ms": 184, "evidence_ids": ["evt_91"] },
  "state_transition": { "from": "gathering", "to": "policy_check" },
  "data_class": "restricted",
  "retention_class": "debug_30d"
}

Measurement

Operational scorecard

ReconstructabilitySampled runs an operator can explain end-to-endMissing proposals, validation, or postconditions break the causal chain.
Behavior driftChanges in tool sequences, denials, corrections, and path lengthDistribution shifts often appear before aggregate failure rates.
Trace exposureSensitive fields collected without a documented debugging needRemove or tokenize at ingestion rather than relying on later cleanup.

Failure drills

Rehearse before the system has real authority

  • Select a failed run and ask an operator to identify the first incorrect event without reading raw private content.
  • Change a prompt version and confirm dashboards can compare path length, tool choice, and corrections before and after.
  • Exercise deletion and retention policies across traces, evaluation exports, caches, and review queues.

Selected primary references

Continue with the source material

These sources inform the wider editorial perspective for this topic. They are not presented as line-by-line citations for every statement.

↑ Back to top