Testing an AI application requires two complementary systems. Deterministic tests prove that authentication, APIs, schemas, tools and user workflows behave correctly. Evaluations measure whether variable model output is useful, grounded and safe enough for the task.

Neither replaces the other. A fluent answer can arrive through a broken workflow, while a perfectly valid JSON response can still contain a bad answer. The AI Testing & Evaluation hub connects these layers across application, model and agent testing.

Start with the system, not the model

Map the application before choosing metrics:

1. Inputs: user messages, documents, images, retrieved context and tool results.
2. Model behavior: instructions, model choice, sampling settings and structured-output contract.
3. Application logic: authentication, routing, retries, queues, storage and fallbacks.
4. Agent actions: tool selection, arguments, permissions, approvals and side effects.
5. User outcome: correctness, usefulness, latency, cost and recovery when something fails.

This map prevents a common mistake: assigning every failure to “the model.” Retrieval can return the wrong document, an API can truncate a stream, a tool can receive invalid arguments, or the interface can render an earlier response after a retry.

The AI testing stack

Layer	What it proves	Typical method
Contract	Requests and responses have the required shape	Schema and type validation
Application	Auth, queues, streaming and fallbacks work	Unit, integration and API tests
Workflow	A user or agent completes the intended path	E2E tests with controlled model responses
Model quality	Output meets task-specific expectations	Evaluation dataset and graders
Agent behavior	Tools, permissions and recovery are correct	Scenario tests, traces and side-effect assertions
Production	Real traffic remains reliable	Monitoring, sampling and reviewed incidents

The AI Application Architecture foundation explains the contracts and failure boundaries that make these layers testable.

1. Test deterministic behavior first

Do not spend model calls on behavior ordinary tests can verify. Stub or record the provider response and test that the application:

• rejects missing or invalid authentication;
• handles rate limits and provider timeouts safely;
• does not duplicate a tool action after a retry;
• renders streaming, cancelled and partial states correctly;
• validates structured output before using it;
• redacts secrets and private context from logs;
• preserves tenant and permission boundaries;
• offers a recoverable fallback when generation fails.

For API-specific coverage, use testing AI APIs with Postman and automated validation. For browser workflows, compare Playwright and Cypress for AI applications.

2. Build an evaluation dataset

An evaluation dataset is a versioned collection of inputs plus the evidence needed to judge their outputs. Begin with a small representative set, then expand it from real failures.

Include:

• ordinary production tasks;
• difficult but valid inputs;
• multilingual or long-context cases where relevant;
• previous incidents and user complaints;
• retrieval cases with known supporting documents;
• unsafe, out-of-scope or prompt-injection attempts;
• tool scenarios with allowed and forbidden actions.

Do not copy sensitive production data into an eval file without a lawful purpose, minimization and access controls. Redact or synthesize cases when the original input is not required.

Each case should have stable metadata:

{
  "id": "support-refund-policy-017",
  "input": "Can I receive a refund after 45 days?",
  "expected_facts": ["policy window is 30 days"],
  "required_behavior": ["cite policy", "do not promise refund"],
  "risk": "high",
  "source_version": "refund-policy-2026-07"
}

Version datasets independently from prompts. Otherwise a changed test set can disguise a model regression.

3. Choose graders by failure mode

Use the most objective signal available.

• Exact or programmatic checks: schema validity, citations present, tool name, argument range, code execution and database state.
• Reference-based checks: required facts, supported claims and retrieval attribution.
• Model graders: tone, completeness or task-specific quality when a calibrated rubric exists.
• Human review: high-impact decisions, ambiguous output and periodic calibration of automated graders.

An LLM judge is another model, not ground truth. Validate its rubric against reviewed examples, measure disagreement and keep deterministic checks separate. Never let one aggregate score hide a severe safety or permission failure.

4. Test agents as stateful workflows

An agent test must inspect more than its final answer. Capture the trace and assert:

• which tools were available;
• which tool was selected;
• whether arguments matched the schema;
• whether the acting identity had permission;
• whether approval occurred before a consequential action;
• whether retries repeated a side effect;
• how the agent recovered from a tool failure;
• what state changed after the run.

Use fake or sandboxed tools in CI. Production credentials should not be available to an untrusted test or pull request. The AI Security foundation covers agent permissions and credential boundaries; the Coding Agents hub covers review gates for agent-generated changes.

5. Run regressions on every meaningful change

Prompt edits are only one source of drift. Rerun the relevant suite when you change:

• the model or provider;
• system instructions or tool descriptions;
• retrieval documents, chunking or ranking;
• embedding models or vector indexes;
• structured-output schemas;
• safety policies and permissions;
• application orchestration or fallback logic.

Compare the candidate with a pinned baseline on the same dataset. Report per-slice results—not only a global average—so a gain on easy cases cannot conceal a loss on a high-risk segment. See LLM regression testing for the complete pipeline.

6. Define production quality gates

A release gate should reflect product risk. A useful sequence is:

1. deterministic unit, contract and security tests;
2. targeted offline evaluations;
3. cost and latency budgets;
4. reviewed trace samples for changed agent behavior;
5. staged or canary deployment;
6. production monitoring with a rollback condition.

Record the model snapshot, prompt version, dataset version, code commit and grader configuration with every result. Without that provenance, a passing report cannot be reproduced.

GitHub Actions can run the deterministic suite and selected evaluations, but keep secrets scoped and avoid granting deployment credentials to untrusted pull requests. Connect the gate to CI/CD pipelines for AI applications and the operational response to AI Operations.

Human review is a control, not a metric

Use humans where judgment or impact requires accountability:

• calibrating model graders;
• reviewing disagreements and borderline cases;
• approving high-risk agent actions;
• labeling production failures;
• deciding whether a quality tradeoff is acceptable.

Measure reviewer agreement and provide a concrete rubric. “Looks good” is not a reproducible evaluation standard.

A practical first implementation

Start with one critical workflow:

1. Map its contracts, model calls, tools and side effects.
2. Write deterministic tests for every failure boundary.
3. Create a small, representative eval dataset from real tasks and known failures.
4. Add objective graders before model-based graders.
5. Run the suite against the current production configuration to establish a baseline.
6. Require the suite for changes to prompts, models, retrieval or tools.
7. Sample production traces and turn confirmed incidents into regression cases.

The goal is not to prove that an AI system is perfect. It is to make its acceptable behavior explicit, detect meaningful regressions before release and preserve evidence for the decisions that still require human judgment.