A 502 Bad Gateway means a proxy, load balancer or API gateway could not obtain a usable response from its upstream. In an AI application, that upstream may be a model server, agent service, retrieval API or streaming worker.

The gateway is where the error surfaced—not necessarily where it began. Use the AI Operations hub to connect this incident to monitoring and rollback.

Trace one failed request end to end

Capture a request ID at every hop:

client → edge/CDN → API gateway → application → model server/tool

Compare gateway logs, application logs and provider or inference telemetry for the same request. Do not start by increasing every timeout; that can convert a visible failure into resource exhaustion.

Classify the upstream failure

Evidence	Likely cause
connection refused	upstream process is down, wrong port or wrong network address
connect timeout	routing, firewall, DNS or overloaded listener
upstream closed early	crash, restart, OOM or protocol mismatch
invalid response headers	wrong protocol or application emitted malformed response
failure after stream began	interrupted model stream; gateway may no longer replace the response cleanly

Check the service directly from the same network context as the gateway:

curl -v http://UPSTREAM_HOST:PORT/health
ss -lntp
docker ps
docker logs MODEL_CONTAINER --tail 200

For Kubernetes, inspect endpoints and pods rather than assuming the Service has a ready backend:

kubectl get pods,svc,endpoints -n AI_NAMESPACE
kubectl describe pod MODEL_POD -n AI_NAMESPACE
kubectl logs MODEL_POD -n AI_NAMESPACE --previous

AI-specific causes

Model workloads often fail upstream because:

• model loading exceeds readiness or startup limits;
• RAM, VRAM or container memory is exhausted;
• long context or high concurrency saturates the inference server;
• a deployment points at a model revision that is not ready;
• streaming is buffered or terminated by an incompatible intermediary;
• an agent tool or provider closes the connection mid-workflow.

Check resource signals and recent deployments before blaming the proxy. The AI memory troubleshooting foundation and Kubernetes OOMKilled guide cover memory failures.

Align timeouts deliberately

Define separate budgets for connection, first byte, idle stream and total workflow time. The client, gateway and upstream should agree about cancellation. A gateway timeout shorter than expected model startup produces false 502/504 failures; an unbounded upstream timeout ties up capacity.

Do not automatically retry every request. Retry only transient failures, with bounded backoff and idempotency protection. A repeated agent action or paid model request can create duplicate side effects and cost. See handling AI API failures.

Fix and verify

After correcting the process, route, resource limit or timeout:

1. confirm upstream readiness from the gateway network;
2. run a small non-streaming request;
3. test a representative streaming request;
4. verify cancellation and interrupted-stream behavior;
5. monitor 502 rate, latency, restarts and saturation;
6. preserve the incident as a regression scenario.

The AI Application Architecture foundation covers gateways, retries and idempotency. A durable fix removes the failed boundary; it does not merely suppress the status code.