An AI request can time out before a connection exists, while waiting for the first model token, during a stream or after an agent has already executed tools. “Increase the timeout” is therefore not a diagnosis.

Identify the timeout phase

Phase	What it measures	Common cause
DNS	hostname resolution	resolver, service discovery or record failure
Connect	TCP/TLS establishment	route, firewall, listener or saturation
First byte/token	time before response begins	queueing, cold model load or provider latency
Read/idle	gap during response	stalled stream, proxy idle limit or upstream reset
Total workflow	end-to-end deadline	tools, retries, retrieval and model work combined

Log these phases separately with one request ID. A single “request took 60 seconds” metric cannot tell whether the model or infrastructure was slow.

Diagnose the path

Check DNS, destination, port and TLS from the same network environment as the application. Then inspect gateway and upstream saturation, queue depth, model loading, context length and concurrency.

In Kubernetes, verify Service endpoints and pod readiness. For local inference, confirm the model server is listening on the interface and port the container or client can actually reach.

Set a deadline hierarchy

The user-facing deadline should contain smaller budgets for connection, provider response and internal tools. Propagate cancellation when the overall request expires so upstream generation and tools do not continue consuming capacity.

Streaming does not eliminate deadlines. Define idle and total limits separately; a stream can begin quickly and then stall.

Retry and fallback

Retry only errors classified as transient and only within a bounded budget. Add jittered backoff, cap attempts and prevent retry storms. Do not automatically replay:

• non-idempotent tool actions;
• uploads or jobs with unknown state;
• expensive model requests after a client has left;
• authentication and validation failures.

Before falling back to another model or provider, verify schema, safety, tool and quality compatibility. Handling AI API failures covers this decision flow.

Test and monitor

Inject controlled DNS, connect, provider and mid-stream delays. Assert cancellation, cleanup, retry limits and user-visible recovery. Track tail latency, timeout phase, provider, model, route, fallback and cost per successful task.

Connect timeouts to AI Application Architecture, AI Deployment & Hosting, AI Operations and AI Testing & Evaluation. Related diagnostics: connection refused and connection reset.