OpenAI went down three times last month. Anthropic had a rate-limit storm that throttled half its users for two hours. Google’s Gemini API returned 503s during peak traffic on a Tuesday afternoon.

If your app depends on a single AI provider with no fallback, your app went down too. Your users saw spinners, error screens, or worse — silent failures that corrupted data.

This is the reality of building on AI APIs in 2026. The models are incredible. The infrastructure behind them is not invincible. Here’s how to build systems that stay up when your providers don’t.

Why AI Providers Go Down

AI API failures come in three flavors:

Hard outages. The provider is completely unreachable. Status page goes red. Nothing you can do except wait — or route elsewhere.

Rate limits. You hit your tokens-per-minute or requests-per-minute ceiling. The API returns 429s. Your app stalls even though the provider is technically healthy.

Silent degradation. The API responds, but latency spikes from 500ms to 8 seconds. Or responses come back truncated, lower quality, or with increased error rates. These are the hardest to detect and the most dangerous.

All three require different handling. A good fallback system addresses each one. Let’s look at four patterns that, layered together, make your AI-powered app resilient.

Pattern 1: Multi-Provider Fallback

The most straightforward pattern. If Provider A fails, try Provider B. If B fails, try C.

Claude → GPT-4o → Gemini.

This works because the major providers rarely go down simultaneously. You define a priority list of models across providers and cascade through them on failure. The tradeoff is that each provider has slightly different behavior, formatting, and capabilities — so your prompts need to be provider-agnostic, or you need provider-specific prompt templates.

This is the foundation. If you implement only one pattern from this post, make it this one. We covered the routing layer for this in our multi-model architecture guide and the AI gateway pattern.

Pattern 2: Model Tier Fallback

Sometimes the provider is fine, but the specific model you want is overloaded. Or you’re burning through expensive tokens on a task that doesn’t need the top-tier model.

Opus → Sonnet → Haiku.

Within a single provider, fall back from the most capable (and expensive) model to a lighter one. A summarization task that ideally uses Claude Opus can gracefully degrade to Sonnet or Haiku if Opus is slow or unavailable. The output quality drops slightly, but the user gets a response instead of an error.

This pairs well with multi-provider fallback. First try your preferred model, then a cheaper model on the same provider, then jump to another provider entirely. OpenRouter makes this trivial with its built-in fallback routing.

Pattern 3: Circuit Breaker

Fallback chains solve the “what do I call instead” problem. Circuit breakers solve the “stop wasting time on something that’s broken” problem.

Without a circuit breaker, every request to a failing provider burns timeout seconds before falling back. If your timeout is 10 seconds and you’re getting 50 requests per second, that’s 500 wasted seconds of user-facing latency every second.

A circuit breaker tracks failure rates. When failures cross a threshold (say, 5 failures in 30 seconds), the circuit “opens” — requests skip the failing provider entirely and go straight to the fallback. After a cooldown period, the circuit “half-opens” and sends a test request. If it succeeds, traffic resumes. If not, the circuit stays open.

Three states: closed (normal), open (skip this provider), half-open (testing recovery).

This is critical for production systems. Without it, a failing provider drags down your entire response time even when you have perfectly good fallbacks available.

Pattern 4: Graceful Degradation

What happens when all your providers are down? Or when you’ve exhausted every fallback and nothing responds?

You degrade gracefully instead of failing hard:

• Cached responses. Return a recent cached response for identical or similar queries. Stale data beats no data for many use cases.
• Simplified output. Fall back to rule-based logic, templates, or a simpler non-AI code path. A chatbot can say “I’m having trouble right now, but here are some relevant help articles” using keyword matching.
• Human handoff. Route the request to a human agent or queue it for processing when services recover.

The key insight: not every request needs a fresh LLM response. Identify which do and which don’t, and build your degradation tiers accordingly. We covered more strategies for this in our error handling guide for AI agents.

Implementation with LiteLLM

LiteLLM gives you multi-provider fallback, model tiering, and basic circuit breaking out of the box. Here’s a production-ready setup combining all four patterns:

import litellm
from litellm import completion
import hashlib, json

# Simple response cache
cache = {}

def cached_key(messages):
    return hashlib.md5(json.dumps(messages).encode()).hexdigest()

def ai_completion(messages, **kwargs):
    """Multi-provider fallback with circuit breaker and graceful degradation."""
    
    # Fallback chain: provider fallback + model tier fallback
    model_chain = [
        "anthropic/claude-sonnet-4-20250514",    # Primary
        "anthropic/claude-3-5-haiku-20241022",  # Tier fallback
        "openai/gpt-4o",                # Provider fallback
        "openai/gpt-4o-mini",           # Tier + provider fallback
        "gemini/gemini-2.0-flash",      # Last resort
    ]
    
    last_error = None
    for model in model_chain:
        try:
            response = completion(
                model=model,
                messages=messages,
                timeout=15,
                num_retries=1,
                **kwargs
            )
            # Cache successful responses
            cache[cached_key(messages)] = response
            return response
        except Exception as e:
            last_error = e
            continue
    
    # Graceful degradation: return cached response
    key = cached_key(messages)
    if key in cache:
        return cache[key]
    
    # Final fallback: raise with context
    raise RuntimeError(f"All providers failed. Last error: {last_error}")

LiteLLM handles the unified API across providers — same interface for Anthropic, OpenAI, and Google. The num_retries and timeout parameters give you basic retry logic per provider. The fallback chain cascades through providers and model tiers. The cache provides graceful degradation when everything is down.

For circuit breaking at scale, LiteLLM’s proxy server (litellm --config config.yaml) supports allowed_fails and cooldown_time per model, which implements the circuit breaker pattern at the infrastructure level rather than in application code.

# litellm config.yaml
model_list:
  - model_name: default
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      allowed_fails: 3
      cooldown_time: 60
  - model_name: default
    litellm_params:
      model: openai/gpt-4o
      allowed_fails: 3
      cooldown_time: 60

This gives you automatic circuit breaking — after 3 failures, the model is skipped for 60 seconds.

Monitoring: Know Before Your Users Do

Fallback patterns are useless if you don’t know they’re activating. Track these metrics:

• Fallback activation rate. How often are requests hitting non-primary models? If it’s climbing, something is degrading.
• Latency per provider. Spot silent degradation before it becomes an outage.
• Error rate by provider and model. Know which provider is causing problems.
• Circuit breaker state changes. Alert when a circuit opens. Alert again when it closes.
• Cache hit rate during degradation. Understand how many users are getting stale responses.

Wire these into your existing observability stack — Datadog, Grafana, whatever you use. The goal is to know your primary provider is struggling before your users notice.

Putting It All Together

The four patterns layer on top of each other:

1. Circuit breaker detects a failing provider and opens the circuit
2. Multi-provider fallback routes to the next healthy provider
3. Model tier fallback tries a cheaper model if the preferred one is unavailable
4. Graceful degradation catches the case where everything is down

This is the same resilience thinking that backend engineers have applied to databases and microservices for years. AI APIs are just another external dependency — treat them that way.

For the broader architecture around this, see our AI app architecture patterns guide. And if you want a managed layer that handles routing and fallback for you, check out the AI gateway pattern.

Your AI provider will go down. The question is whether your app goes down with it.