An AI API can return HTTP 200 and still fail: malformed structured output, a truncated stream, an unsupported citation, a duplicated tool action or a response that violates the product’s safety policy. Reliable API testing therefore combines ordinary protocol checks with AI-specific contracts and evaluations.

This guide uses Postman collections as an executable interface contract. The same design can run in another API client or test runner. The important part is separating deterministic API behavior from variable model quality. See the AI Testing & Evaluation hub for the complete testing stack.

Define the API contract first

For each endpoint, document:

• authentication and required scopes;
• request schema and size limits;
• synchronous, streaming or asynchronous response mode;
• structured-output or tool-call schema;
• rate-limit behavior;
• timeout, cancellation and retry semantics;
• error envelope and request identifier;
• data retention and logging boundaries.

The AI Application Architecture foundation shows how these contracts fit into reliable model workflows. Tests cannot compensate for an undefined retry or idempotency policy.

Build a layered collection

Organize the collection by behavior rather than one happy-path request:

AI API
├── Authentication
│   ├── valid service token
│   ├── missing token
│   ├── expired token
│   └── insufficient scope
├── Generate
│   ├── valid request
│   ├── invalid model or parameter
│   ├── structured output
│   └── provider failure
├── Streaming
│   ├── complete stream
│   ├── cancelled client
│   └── interrupted stream
└── Limits
    ├── request too large
    ├── quota exhausted
    └── retry response

Use environments for base URLs and non-secret identifiers. Keep tokens in an approved secret store for automated runs; do not commit them to collection exports.

Test authentication and authorization

Cover more than “token present”:

• missing or malformed authorization;
• expired or revoked credentials;
• correct identity with the wrong scope;
• tenant isolation;
• tool permission boundaries;
• safe error messages that do not reveal secrets or account state.

An agent endpoint must not gain permission merely because the model requested a tool. Authorization belongs to the application and acting identity. Link those controls to the AI Security foundation.

Validate ordinary JSON responses

Postman test scripts can assert the deterministic envelope:

pm.test('returns a valid completed response', function () {
  pm.response.to.have.status(200);
  pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json');

  const body = pm.response.json();
  pm.expect(body).to.have.property('request_id').that.is.a('string');
  pm.expect(body).to.have.property('status', 'completed');
  pm.expect(body.output).to.be.an('object');
});

Validate the full response against a versioned schema when the contract matters. A few property assertions do not prove that nested tool arguments or result objects are safe.

Test structured outputs and tool calls

Model-produced JSON crosses two validation boundaries:

1. the text or stream must form valid JSON;
2. the parsed value must satisfy the application schema and semantic rules.

Test:

• missing required fields;
• extra fields when the schema forbids them;
• wrong enum values and types;
• out-of-range numbers;
• tool names outside the allowlist;
• valid structure with unauthorized or unsafe arguments;
• schema-version mismatches.

Structured generation reduces formatting failures but does not validate business meaning or authorization. Use Structured Outputs for provider constraints and the JSON parsing foundation for malformed or truncated responses.

Test streaming responses

A streaming endpoint needs more than a final status assertion. Verify:

• headers and content type are sent correctly;
• events can be parsed incrementally;
• the stream terminates with the documented completion signal;
• partial text is not mistaken for a completed structured value;
• cancellation releases work and does not trigger a side effect;
• an interrupted stream leaves the UI or client in a recoverable state;
• error events preserve a request identifier without leaking provider details.

Postman can be useful for manually inspecting a stream, but CI validation may be clearer in a small purpose-built runner that parses every event and asserts the sequence. Choose the runner based on what must be proven, not on forcing every protocol into one interface.

Simulate model and provider failures

Exercise the application boundary with a fake provider or controlled test double:

Failure	Expected application behavior
Provider timeout	bounded wait, safe error or approved fallback
Rate limit	honor retry policy; no retry storm
Invalid provider JSON	reject before state change
Partial stream	mark incomplete and allow recovery
Tool schema mismatch	do not execute tool
Provider unavailable	stable public error; internal request ID retained

Do not make CI depend on provoking failures from a live provider. Controlled responses are faster, repeatable and safer. The AI API failure guide covers retry and fallback design.

Test rate limits, quotas and retries

Verify both sides of the limit:

• the API rejects excess work with the documented response;
• the response communicates retry timing where applicable;
• the client backs off rather than retrying immediately;
• concurrent retries do not duplicate jobs or tool calls;
• user, tenant and global limits remain isolated;
• quota usage and error reporting remain consistent.

Use an idempotency key for retryable operations that create state, then assert that repeated requests resolve to one logical operation. Never load-test a production endpoint without explicit authorization and controls.

Keep answer quality out of brittle API assertions

Do not assert an open-ended response string word for word. In the API collection, verify the transport, schema, provenance fields and safety-critical invariants. Send the output plus case metadata to an evaluation pipeline for:

• factual correctness;
• groundedness in supplied context;
• task completion;
• policy compliance;
• style or usefulness.

Run candidate and baseline configurations on the same versioned dataset. LLM regression testing explains how to prevent prompt, model, RAG and embedding changes from silently degrading quality.

Use LLMs to propose tests safely

An LLM can inspect an OpenAPI fragment and suggest missing cases, but treat the result as untrusted draft material:

1. provide a limited, non-sensitive contract;
2. require structured candidate cases;
3. validate the generated JSON;
4. review assumptions against product rules;
5. convert approved cases into deterministic scripts;
6. execute them only in an isolated environment.

Do not give a model production tokens or let newly generated requests run automatically against a privileged endpoint. The model may invent fields, expected codes or destructive cases.

Run the collection in CI

A production gate can run the reviewed collection against an ephemeral or staging environment:

1. provision isolated test data;
2. inject least-privilege credentials from CI secrets;
3. run contract, auth, failure and limit tests;
4. export machine-readable results;
5. run relevant offline evaluations;
6. publish artifacts for review;
7. block deployment on hard contract or security failures.

Keep privileged deployment credentials away from untrusted pull-request jobs. Connect the workflow to CI/CD pipelines for AI applications and use AI Operations for post-deployment canaries and monitoring.

A minimum useful AI API suite

Before shipping a new model endpoint, cover at least:

• valid, missing and insufficient authentication;
• valid and invalid request schemas;
• structured-output validation;
• one controlled provider timeout and rate-limit response;
• streaming completion, cancellation and interruption if supported;
• idempotency for retryable state changes;
• redaction and stable public errors;
• a representative offline evaluation set for output quality.

That suite tests the system developers operate—not merely whether a model can return an answer once.