AI agent evaluation and observability

Direct answer

Evaluate an AI agent at two levels: did it achieve the correct user outcome, and did it take an acceptable path to get there? Score the final result, the trajectory of decisions and tool calls, tool selection and argument precision, safety-policy compliance, latency, and cost. Run a versioned test set before release, block regressions on explicit gates, and monitor the same signals in production through redacted traces and user outcomes.

An agent is not adequately tested because its final sentence looks good. It can reach a plausible answer after querying the wrong customer, leaking sensitive context to a tool, retrying five times, or spending more than the task is worth. Conversely, a strict exact-path assertion can reject a safe, efficient alternative. Good evaluation separates outcome requirements from path constraints.

What you will build

This tutorial gives you a practical evaluation plan for an agent that reads support cases, searches approved documentation, and drafts a response. The same structure applies to research, coding, operations, and workflow agents.

You will define:

• an outcome rubric for correctness and usefulness
• trajectory checks for required, forbidden, and wasteful behavior
• tool-call precision and safety tests
• latency and cost budgets
• a versioned offline test set
• release gates for agent, prompt, model, or tool changes
• production traces, monitors, and a feedback loop

If you are still defining basic prompts, start with prompt engineering for code. Agent evaluation begins once a model can choose actions or tools rather than only produce one response.

Why agent evaluation is different

A single model call usually maps one input to one output. An agent can observe, plan, call a tool, inspect the result, revise its plan, call another tool, and stop. The number and order of intermediate states may vary.

That creates three kinds of correctness:

1. Outcome correctness: the user received the right result.
2. Process correctness: the agent used allowed evidence and actions.
3. Operational correctness: it finished within reliability, latency, and cost limits.

Some tasks care mostly about the outcome. A brainstorming assistant may take several valid routes. Other tasks impose hard path requirements. A refund agent must verify authorization before issuing a refund even if the final amount is correct. Mark that distinction in each test instead of applying one scoring rule to everything.

Start with an agent contract

Write the contract before collecting metrics. For the support agent, it might say:

> Given a support case and tenant identity, retrieve only that tenant's records, use approved documentation, draft a cited answer, ask for human review when evidence is insufficient, and never modify customer data. Finish within 12 seconds and a defined per-run cost budget.

Turn the contract into observable dimensions:

A gate must be tied to product risk. Do not pick “90% accuracy” because it sounds respectable. Define what constitutes a pass, which failures are release blockers, and which can be reviewed manually.

Outcome evaluation

Outcome evals judge what the user ultimately receives or what changed in the system.

Use deterministic assertions when the answer is objectively checkable:

• exact ticket, account, or document identifiers
• a JSON schema and business validation
• arithmetic, dates, or status transitions
• citations that resolve to the supplied evidence
• whether an approved side effect occurred exactly once

Use a rubric when quality has legitimate variation:

• factual support: every material claim follows from retrieved evidence
• completeness: all parts of the request are addressed
• relevance: no unrelated procedure is included
• actionability: the user can take the next step
• calibrated uncertainty: missing evidence is stated rather than invented
• style: clear, respectful, and consistent with product policy

Human review remains the reference for subjective, high-risk judgments. A model grader can increase coverage, but it is another model with its own errors. Calibrate it against a human-labeled sample, require a short evidence-based rationale, and regularly inspect disagreements.

Trajectory evaluation

A trajectory is the sequence of observations, decisions, tool calls, tool results, retries, and the final response. Evaluate it with constraints at three strengths.

Required steps

These are invariants, not preferred reasoning styles:

• authenticate and resolve tenant scope before data retrieval
• read the policy version used for a regulated decision
• request confirmation before an irreversible side effect
• validate tool output before using it

Forbidden steps

Examples include:

• calling an unapproved tool
• sending secrets or unnecessary personal data to a model or tool
• querying another tenant
• writing when the task is read-only
• repeating a non-idempotent action

Efficiency signals

These reveal waste but should not overconstrain valid paths:

• duplicate searches with equivalent arguments
• repeated calls after a decisive result
• avoidable transfers to a larger model
• loops that make no state progress
• retrieving documents never used in the answer

Avoid asserting an exact hidden chain of thought or one perfect tool sequence. Record externally visible decisions and actions. Two paths can both be valid; a fragile exact-match trajectory test rewards imitation instead of correct behavior.

Measure tool precision

Tool quality has several components:

Selection precision asks whether calls were necessary and appropriate. Of all tool calls made, how many were useful and allowed?

Selection recall asks whether the agent made every required call. Did it omit the authorization check or evidence lookup?

Argument accuracy validates IDs, filters, dates, scopes, and enum values against the test fixture.

Result use checks whether the final answer reflects the tool result without changing or inventing facts.

Side-effect integrity checks that write actions are authorized, idempotent where possible, and executed the expected number of times.

A tool call should have a stable schema, a short description of when to use it, server-side authorization, input validation, timeout behavior, and an idempotency strategy for writes. Evaluation cannot compensate for a tool that lets the model choose arbitrary tenant IDs without enforcement.

Safety evaluation

Build adversarial cases around the actual capabilities of the agent:

• prompt injection inside retrieved documents
• malicious instructions in a support ticket
• requests for another user's data
• tool output containing executable-looking text
• attempts to reveal system prompts, keys, or hidden records
• pressure to skip approval because the request is “urgent”
• repeated or ambiguous requests that could duplicate a write

Test both refusal and safe completion. “Refuse everything unusual” is not a useful safety policy. The agent should continue with allowed parts, explain blocked actions briefly, and offer a safe escalation path.

Defense belongs outside the model too: least-privilege credentials, per-tool authorization, allowlists, data minimization, output encoding, sandboxing, rate limits, approval for consequential actions, and audit logs. Treat model instructions as one control, not the security boundary.

Latency and cost evaluation

Measure full-run latency and every material span:

• model response time
• queue time
• retrieval and tool duration
• approval wait time, reported separately
• retries and backoff
• final rendering or streaming completion

Report distributions such as p50 and p95, not only an average. Segment by task class because a two-tool lookup and a ten-source investigation should not share one unexplained number.

For cost, record model input, cached input where supported, output, tool fees, search or storage charges, and failed or retried calls. The useful product metric is cost per accepted outcome, not cost per invocation:

cost per accepted outcome =
  total model and tool cost / number of outcomes that passed

Set two controls: an evaluation gate on expected cost and a runtime ceiling on calls, tokens, elapsed time, or spend. A dashboard after an infinite loop is evidence, not containment.

Design the test set

Start with 30 to 50 cases if the product is new. Coverage matters more than an arbitrary large count.

Include:

• ordinary successful cases across major intents
• boundary values, missing fields, and malformed tool results
• ambiguous cases where escalation is correct
• authorization and cross-tenant attacks
• prompt injection in user input and retrieved content
• tool timeout, rate limit, partial failure, and stale data
• cases that previously failed in production
• expensive cases likely to trigger loops or oversized context

Store each case with a stable ID, sanitized input fixture, mocked or recorded tool responses, expected outcome assertions, trajectory constraints, risk level, and provenance. Separate the immutable test definition from run results.

type AgentEvalCase = {
  id: string;
  risk: "low" | "medium" | "high";
  input: unknown;
  toolFixtures: Record<string, unknown>;
  expected: {
    requiredFacts?: string[];
    requiredTools?: string[];
    forbiddenTools?: string[];
    requiresHumanReview?: boolean;
  };
  budgets: { maxToolCalls: number; maxDurationMs: number };
};

Keep a hidden holdout set for major changes if contributors routinely tune against the main suite. Otherwise the agent can overfit familiar wording while general capability stalls.

Build a repeatable evaluation harness

For every run, pin or record:

• agent and prompt version
• model identifier and relevant inference settings
• tool schema versions
• retrieval corpus or index version
• test-set version
• evaluator and rubric version
• dependency or environment version when it changes behavior

Run tools against deterministic fixtures for most offline tests. Add a smaller integration suite against staging to detect authentication, schema, network, and provider behavior that mocks cannot reproduce. Never point a routine eval at production write tools.

The harness should return per-case assertions plus aggregates by intent and risk. A global score can hide a complete failure on a small but critical category.

Set regression gates

Use hard gates for non-negotiable behavior:

• zero unauthorized data access
• zero unapproved side effects
• all required confirmations observed
• no known secret disclosure
• no regression on critical cases

Use bounded gates for variable dimensions:

• outcome pass rate does not fall beyond an agreed tolerance
• tool precision stays above its baseline
• p95 latency and expected cost remain within budgets
• escalation rate remains within a reviewed range

Compare candidate and baseline on the same cases. Review changed failures, not only aggregate deltas. A one-point overall improvement is not a win if the new version breaks the only account-deletion test.

Run the suite when prompts, models, tool descriptions, schemas, routing, retrieval, policies, or context assembly change. These are code changes even when no application source file changed.

Trace agents in production

A trace should reconstruct a run without requiring raw private content. Use a root span for the agent run and child spans for model requests, retrieval, tool calls, policy checks, approvals, and handoffs.

Useful fields include:

• trace and run ID
• agent, prompt, model, tool, and policy versions
• task class and risk class
• span status, duration, retry count, and stop reason
• token and cost data when available
• tool name and a redacted argument summary
• outcome class, escalation, and user feedback

Do not log secrets, authorization headers, raw credentials, or unrestricted prompts and tool results. Decide retention, access, regional storage, and deletion rules before enabling detailed traces. Use hashes, structured categories, field-level redaction, and sampled encrypted payloads only where approved.

OpenTelemetry has generative-AI semantic conventions, but parts remain under active development. Pin the convention version or use a small internal attribute layer so a telemetry naming change does not break every dashboard.

Production monitoring

Offline evals cover known cases. Production monitoring finds distribution shift, integration failures, and new abuse.

Monitor:

• run success, error, timeout, and cancellation rates
• outcome proxies such as acceptance, correction, retry, and escalation
• tool-call errors, denied calls, duplicate writes, and no-progress loops
• latency and cost by task, agent version, and model
• safety-policy triggers and confirmed incidents
• retrieval failures, empty evidence, and citation defects
• differences between staged rollout cohorts

Alerts should point to an action. A sudden cross-tenant denial spike may trigger investigation and temporary tool restriction. A cost spike may lower call limits or disable an expensive route. Keep a kill switch and a known safe fallback.

Sample production failures into a review queue, sanitize them, label root cause, and add representative cases to the regression set. Do not dump every thumbs-down example into tests without checking whether the user request itself was valid.

A practical four-week eval plan

Week 1: contract and instrumentation

Define outcomes, forbidden actions, risk classes, versions, and trace fields. Instrument model and tool spans with redaction. Establish current latency and cost baselines.

Week 2: first test set

Create 30–50 cases across happy paths, ambiguity, attacks, tool failures, and historical defects. Add deterministic graders first and a human rubric for subjective quality.

Week 3: release gates

Run the baseline, inspect every critical failure, choose hard and bounded gates, and put the suite in CI or the release workflow. Test a deliberate bad prompt to prove the gate can fail.

Week 4: production loop

Roll out by cohort, build operational dashboards and actionable alerts, review sampled traces, and convert verified failures into sanitized regression cases. Assign an owner and review cadence.

Common mistakes

Judging only the final answer. This misses unauthorized access, waste, and unsafe side effects.

Scoring only the trajectory. Agents can take multiple valid paths. Reserve strict order checks for true business invariants.

Using one aggregate score. Break results down by risk and intent so common easy cases do not hide rare critical failures.

Letting a model grader define truth. Calibrate graders against humans and use deterministic checks wherever possible.

Testing live writes. Mock side effects offline and use isolated staging accounts for integration tests.

Logging everything for debugging. Raw traces can become a second sensitive database. Minimize, redact, restrict, and expire them.

Monitoring without limits. Add runtime call, time, and cost ceilings plus a kill switch.

Freezing the test set. Add new validated failures and remove obsolete cases through review, while preserving version history.

Release checklist

• [ ] Agent contract names outcomes, forbidden behavior, and budgets
• [ ] Test cases cover normal, edge, adversarial, and failure paths
• [ ] Outcome and trajectory criteria are separated
• [ ] Tool selection, arguments, result use, and side effects are checked
• [ ] Critical safety failures are hard gates
• [ ] Candidate is compared with a recorded baseline
• [ ] Traces include versions, spans, latency, usage, and outcome
• [ ] Sensitive fields are redacted with tested rules
• [ ] Production monitors have owners and response actions
• [ ] Kill switch and fallback have been exercised

Continue learning

• AI agents explained ? define the behavior being evaluated
• AI agent security ? convert threats into release gates
• Context engineering ? test context changes instead of trusting intuition

Official sources

• OpenAI: Agent evals, trace grading, and Agents SDK tracing
• Anthropic: Building effective agents and Develop tests and evaluations
• OpenTelemetry: Generative AI semantic conventions
• NIST: AI Risk Management Framework

Takeaways

Agent quality is a system property. Test outcomes, observable trajectories, tools, safety, latency, and cost together. Gate releases on versioned cases, then use privacy-aware production traces to discover what the offline set missed.

The durable loop is: define the contract, evaluate a representative set, block meaningful regressions, observe production, and turn verified failures into new tests.