An AI agent is reliable only when it completes the intended task, uses the right tools within its permissions, handles failure safely and leaves the system in the expected state. A convincing final message is not enough: an agent can report success after a tool failed, repeat an action during a retry or spend far more time and money than the task justifies.

Evaluate the full trajectory. The AI Testing & Evaluation hub explains how agent scenarios fit alongside deterministic application tests, model evaluations and human review.

Define success as observable state

For each task, specify:

• the starting state and acting identity;
• allowed tools and maximum permissions;
• required intermediate approvals;
• expected final state;
• forbidden side effects;
• acceptable cost, steps and duration;
• safe behavior when completion is impossible.

“Agent says the issue is fixed” is not a success criterion. “The approved file changed, tests pass, no unrelated files changed and the deployment was not triggered” is testable.

Evaluate four layers

Layer	Questions
Outcome	Did the task reach the correct final state?
Trajectory	Were planning, tool selection and arguments appropriate?
Control	Were identity, permissions and approvals enforced?
Operations	Were latency, cost, retries and recovery acceptable?

Keep hard control failures separate from quality averages. One unauthorized action cannot be cancelled out by several fluent responses.

Build representative agent scenarios

Include:

• ordinary successful tasks;
• ambiguous requests that require clarification;
• tool timeouts and malformed results;
• unavailable dependencies;
• conflicting instructions from untrusted content;
• permission-denied actions;
• repeated or reordered events;
• tasks that cannot be completed safely;
• confirmed production incidents.

Use sandboxed tools and disposable state in automated tests. Never evaluate destructive behavior against production merely to obtain a realistic trace.

Measure task success correctly

Task success should come from objective evidence where possible:

• tests and changed-file scope for coding agents;
• database or API state for business workflows;
• citations and source support for research agents;
• delivered artifact plus recipient and approval record for communication agents;
• browser-visible state for browser agents.

Repeat variable scenarios enough to understand their failure distribution, but do not impose one universal run count. Increase repetitions for unstable or high-impact cases and report uncertainty rather than treating a small sample as exact reliability.

The Coding Agents hub provides repository governance and review boundaries. Use Playwright browser testing when success depends on observable UI behavior.

Grade the trajectory

Capture a structured trace containing:

• model and prompt version;
• tool inventory presented to the agent;
• tool chosen at each step;
• arguments before and after validation;
• result, error and retry decision;
• approval request and approver identity;
• token, cost and timing metadata;
• final state and user-facing response.

Then test concrete properties:

• Was the selected tool permitted and relevant?
• Did arguments match the schema and business rules?
• Did the agent use untrusted tool output as instructions?
• Did it retry only safe operations?
• Did it stop after success or a defined limit?
• Did it preserve enough evidence for review?

Model judges can help label planning quality, but tool permission, argument validity and final state should be checked deterministically whenever possible. See LLM-as-a-Judge.

Detect loops and budget failures

Track more than total steps. Useful loop signals include:

• repeated identical tool call and arguments;
• alternating between two states without progress;
• repeated retrieval of the same evidence;
• retrying a permanent error;
• growing context without new information;
• approaching a time, token or cost budget.

Define stop conditions in the application, not only in the evaluation harness. Test that the agent exits with a safe, useful status and does not claim completion after the limit is reached.

Measure cost and latency per successful task, plus tail values and wasted work from failed runs. A cheaper individual model call does not imply a cheaper agent if it requires more steps and retries.

Test permissions and approvals

The application must authorize the action independently of the model. Evaluate:

• least-privilege tool exposure;
• tenant and resource boundaries;
• approval before consequential side effects;
• expiration and revocation of delegated access;
• argument-level authorization;
• denial behavior that does not leak sensitive state;
• audit records linking action, identity and approval.

Include negative cases where the agent asks for a tool it should not have. A secure result is refusal or escalation—not a hidden attempt through another tool. Connect this layer to AI Security.

Test failure recovery

Inject controlled failures:

Failure	Reliable response
Transient tool timeout	bounded retry with idempotency protection
Permanent validation error	revise arguments or stop; no retry loop
Partial side effect	inspect state before retrying
Permission denied	explain or escalate; do not bypass
Missing dependency	request input or stop safely
Interrupted workflow	resume only from verified state

Recovery tests should assert state, not merely that the agent produced an apology. AI Application Architecture covers idempotency and reliable workflow boundaries.

Production metrics that matter

Aggregate metrics should be segmentable by task, tool, model and risk level:

• verified task success and abandonment;
• human correction or takeover rate;
• tool error and invalid-argument rate;
• approval acceptance and rejection;
• loop or budget-limit termination;
• retry and fallback use;
• median and tail completion time;
• cost per successful task;
• repeated side effects and security incidents;
• user-reported failures linked to traces.

Avoid inferring success solely from the absence of an error or from user silence. Sample completed tasks for reviewed outcome verification, with appropriate privacy controls.

Gate agent changes before release

Rerun relevant scenarios when models, prompts, tools, schemas, permissions or orchestration change. Store dataset, grader, trace schema and configuration versions with every result.

A useful release sequence is:

1. deterministic tool, schema and permission tests;
2. offline agent scenarios with sandboxed tools;
3. reviewed traces for changed high-risk behavior;
4. regression comparison against the approved baseline;
5. limited canary with rollback signals;
6. production sampling and incident-to-regression feedback.

Use LLM regression testing for comparison and GitHub Actions for protected checks. AI Operations owns the monitoring, canary and rollback layer.

Agent reliability is not a single benchmark score. It is evidence that the system reaches the right outcome repeatedly, respects control boundaries, recovers from realistic failures and remains operable when production conditions differ from the demo.