AI API failures are not all the same. A provider can reject a key, throttle a tenant, time out after generating tokens, return malformed structured output, interrupt a stream, or answer successfully with unusable content. Treating every failure as “retry three times” can multiply latency, cost, and duplicate side effects.

A reliable application first classifies the failure, then chooses whether to retry, fall back, ask the user, queue work, or stop.

Failure policy at a glance

Failure	Default action	Retry?	Fallback?
Invalid credentials	Stop and alert owner	No	No
Invalid request or unsupported parameter	Fix request	No	Only after translation
Context window exceeded	Reduce or re-chunk input	No identical retry	Possibly
Rate limit	Respect provider headers and queue	Yes, bounded	Possibly
Provider 5xx/network failure	Backoff within a retry budget	Yes	Yes, if task-compatible
Application timeout	Cancel upstream when possible	Carefully	Possibly
Safety or policy rejection	Surface a safe explanation	No bypass retry	No
Invalid structured output	Validate and perform one bounded repair	Sometimes	Possibly
Interrupted stream	Mark incomplete; resume only if supported	Not blindly	Possibly

Normalize errors at your application boundary

Clients should not need to understand every provider’s error format. Translate upstream failures into a stable internal envelope while preserving diagnostic metadata server-side.

{
  "error": {
    "code": "model_rate_limited",
    "message": "The model is temporarily busy. Try again shortly.",
    "retryable": true,
    "requestId": "req_01J..."
  }
}

The public response should not include provider credentials, raw stack traces, internal URLs, full prompts, or sensitive tool arguments. Logs may record provider, model, status, latency, retry count, and request ID, but content logging needs an explicit privacy and retention policy.

See AI Application Architecture and API authentication for the surrounding boundary.

Set timeouts by operation

Use separate budgets for:

• connection establishment;
• first token;
• total generation;
• tool execution;
• background job completion.

A single global timeout hides which dependency failed. Propagate cancellation to the provider and tools when the user disconnects or cancels. Otherwise the UI stops waiting while billable work continues.

Retry only transient failures

Use exponential backoff with jitter so many workers do not retry simultaneously:

async function retryTransient(run, maxAttempts = 3) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await run();
    } catch (error) {
      if (!error.retryable || attempt === maxAttempts) throw error;
      const base = 250 * 2 ** (attempt - 1);
      const jitter = Math.random() * base * 0.25;
      await new Promise(resolve => setTimeout(resolve, base + jitter));
    }
  }
}

The caller must set retryable from a deliberate classification. Do not infer it from “an exception happened.” Respect provider retry headers when present.

Use a retry budget

Limit retries per request, tenant, and time window. Without a budget, an upstream incident can create a retry storm that consumes remaining capacity and multiplies model spend.

A useful policy might allow one fast retry for a transient connection failure, then queue the job or use an approved fallback. Interactive requests need a tighter latency budget than offline document processing.

Coordinate retry and quota logic through rate limiting for AI APIs.

Fallbacks are product decisions

A cheaper or smaller model may be acceptable for classification but unsafe for code migration or a regulated workflow. Define fallback eligibility per task—not globally.

Record:

• requested and responding model;
• why fallback occurred;
• whether capabilities changed;
• token usage and cost;
• quality or validation result.

Do not silently drop tools, structured-output guarantees, context, or safety requirements. A fallback that changes behavior should be visible to the application and, where material, the user.

An AI gateway is a natural place to enforce this policy.

Circuit breakers protect a failing provider

After repeated qualifying failures, temporarily stop sending new requests to that route. Use a small number of probe requests to determine whether it recovered. Circuit breakers should be scoped by provider, region, or model so one incident does not disable unrelated capacity.

Streaming failures need an explicit state

Once output reaches the user, an automatic retry can produce a second, conflicting answer. Mark interrupted output as incomplete and offer a deliberate retry. For persistent workflows, store checkpoints outside the stream.

Never execute tool calls merely because partial text suggested them. Validate the complete structured tool request and authorization before execution.

Asynchronous jobs and webhooks

Long-running generation should often become a job:

1. accept and validate the request;
2. create a job with an idempotency key;
3. process it in a worker;
4. persist progress and outcome;
5. notify through polling, events, or a signed webhook.

Connect this with webhook architecture and idempotency for AI workflows.

Structured output is not automatically valid

Schema-constrained output reduces parsing failures but does not prove factual or business validity. Validate types, allowed values, references, permissions, and domain rules. Keep repair attempts bounded and retain the original failure for debugging.

Production checklist

• Classify errors before retrying.
• Set operation-specific timeout and cancellation budgets.
• Use exponential backoff, jitter, and retry limits.
• Make consequential operations idempotent.
• Define fallbacks per task and capability.
• Redact public errors and logs.
• Track retries, fallback rate, latency, token use, and cost.
• Give interrupted streams and jobs an explicit state.
• Test provider outages and malformed responses before launch.

Reliable AI error handling is not about hiding failure. It is about containing it, explaining it safely, and preserving a trustworthy system state.