Most AI-powered apps start the same way: a direct API call to an LLM, buried somewhere in a route handler. It works. Then traffic grows, costs spike, latency becomes unpredictable, and you realize you’ve built a house on sand.

The fix isn’t more API calls — it’s architecture. After building and reviewing dozens of production AI systems, I’ve identified five patterns that keep showing up in the ones that actually scale. This guide covers each one with diagrams, use cases, and practical examples so you can pick the right foundation before you start building.

Pattern 1: AI Gateway

The AI Gateway centralizes every LLM interaction behind a single internal service. Instead of scattering API calls across your codebase, every request flows through one layer that handles authentication, caching, rate limiting, logging, and fallbacks.

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Frontend    │     │  Backend     │     │  Worker      │
│  Service     │     │  Service     │     │  Service     │
└──────┬───────┘     └──────┬───────┘     └──────┬───────┘
       │                    │                    │
       └────────────┬───────┘────────────────────┘
                    │
             ┌──────▼───────┐
             │  AI Gateway   │
             │  ┌──────────┐ │
             │  │ Cache     │ │
             │  │ Auth      │ │
             │  │ Logging   │ │
             │  │ Fallback  │ │
             │  └──────────┘ │
             └──────┬───────┘
                    │
        ┌───────────┼───────────┐
        ▼           ▼           ▼
   ┌─────────┐ ┌─────────┐ ┌─────────┐
   │ OpenAI  │ │ Claude  │ │ Gemini  │
   └─────────┘ └─────────┘ └─────────┘

When to use it: As soon as you have more than one service calling an LLM, or more than one model provider. It’s the single most impactful pattern for reducing LLM API costs because it gives you one place to add caching and request deduplication.

Example: An e-commerce platform uses AI for product descriptions, customer support chat, and search ranking. Without a gateway, each team manages their own API keys, retry logic, and error handling. With a gateway, you get unified billing visibility, a shared semantic cache that prevents duplicate calls, and the ability to swap providers without touching application code.

The gateway pattern is foundational — most of the other patterns in this guide work better when layered on top of it. It also becomes the natural place to implement fallback logic when a provider goes down: the gateway detects the failure and reroutes to an alternative model without any upstream service knowing. For a deep dive into implementation, see the full AI Gateway pattern guide.

Pattern 2: RAG (Retrieval-Augmented Generation)

RAG splits the AI workflow into two stages: first retrieve relevant context from your own data, then pass that context to the LLM alongside the user’s question. This keeps the model grounded in facts instead of hallucinating.

┌──────────────┐
│  User Query   │
└──────┬───────┘
       │
       ▼
┌──────────────┐     ┌──────────────────┐
│  Embedding    │────▶│  Vector Database  │
│  Model        │     │  (Pinecone, PG,   │
└──────────────┘     │   Qdrant, etc.)   │
                      └────────┬─────────┘
                               │ Top-K results
                               ▼
                      ┌──────────────────┐
                      │  Prompt Assembly   │
                      │  Query + Context   │
                      └────────┬─────────┘
                               │
                               ▼
                      ┌──────────────────┐
                      │  LLM Generation   │
                      └────────┬─────────┘
                               │
                               ▼
                      ┌──────────────────┐
                      │  Response         │
                      └──────────────────┘

When to use it: Whenever the LLM needs to answer questions about your data — internal docs, product catalogs, support tickets, legal documents. If you’re building anything where accuracy matters more than creativity, RAG is non-negotiable. It’s also the pattern that lets you avoid fine-tuning in most cases: instead of retraining a model on your data, you retrieve the relevant slice at query time.

Example: A legal tech startup indexes thousands of case documents into a vector database. When a lawyer asks “What precedents exist for data breach liability in healthcare?”, the system retrieves the 10 most relevant case summaries, injects them into the prompt, and the LLM synthesizes an answer with citations. No fine-tuning required, and the knowledge base updates in real time as new cases are added.

The retrieval quality makes or breaks RAG. Chunking strategy, embedding model choice, and re-ranking all matter enormously. For a hands-on walkthrough, see how to build a local RAG pipeline with Ollama, or read the scaling guide for production RAG systems.

Pattern 3: Multi-Model Routing

Not every request needs GPT-4. A classification question can go to a small, fast model. A complex reasoning task goes to a large model. Image analysis goes to a vision model. Multi-model routing matches each request to the best model for the job based on task type, complexity, cost constraints, or latency requirements.

┌──────────────┐
│  Incoming     │
│  Request      │
└──────┬───────┘
       │
       ▼
┌──────────────────┐
│  Router / Classifier │
│  (rules or ML)       │
└──────┬───────────┘
       │
       ├─── Simple query ──────▶ Small model (fast, cheap)
       │
       ├─── Complex reasoning ─▶ Large model (capable)
       │
       ├─── Code generation ───▶ Code-specialized model
       │
       └─── Vision task ───────▶ Multimodal model

When to use it: When you’re processing diverse request types and your LLM bill is growing. Routing 70% of requests to a smaller model can cut costs by 50-80% with negligible quality loss on those tasks. It also reduces latency for simple requests — a small model responds in 100ms where a large model takes 2 seconds.

Example: A developer tools company routes inline code completions to a small, fast model (sub-200ms responses), sends code review requests to a mid-tier reasoning model, and escalates complex architectural questions to the most capable model available. A lightweight classifier at the gateway level examines the prompt length, task tag, and user tier to make the routing decision.

For a comparison of models to inform your routing rules, see the AI model comparison guide. The full multi-model architecture guide covers router implementation patterns in detail.

Pattern 4: Queue-Based Processing

Some AI tasks don’t need an immediate response. Document summarization, batch embeddings, report generation, content moderation at scale — these belong in a queue, not in a request-response cycle. Queue-based processing decouples the request from the AI work, giving you retry logic, backpressure handling, and cost smoothing for free.

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  API Server   │────▶│  Message      │────▶│  AI Worker    │
│  (accepts     │     │  Queue        │     │  Pool         │
│   request,    │     │  (SQS, Redis, │     │  (processes   │
│   returns     │     │   RabbitMQ)   │     │   AI tasks)   │
│   job ID)     │     └──────────────┘     └──────┬───────┘
└──────────────┘                                  │
       ▲                                          │
       │              ┌──────────────┐            │
       └──────────────│  Results      │◀───────────┘
         (poll or     │  Store        │
          webhook)    │  (DB / S3)    │
                      └──────────────┘

When to use it: For any AI task where the user doesn’t need the result within the same HTTP request. Batch processing, background enrichment, scheduled reports, or any workflow where you’re processing hundreds or thousands of items. It also protects you from provider rate limits — the queue naturally throttles throughput. And when a provider has an outage, messages stay in the queue until the service recovers instead of failing loudly.

Example: A content platform lets users upload PDFs for AI-powered summarization. The upload endpoint accepts the file, drops a message on the queue, and returns a job ID immediately. A pool of workers picks up jobs, calls the LLM, stores the summary, and notifies the user via webhook or email. During peak hours, the queue absorbs the load instead of hammering the API and hitting rate limits.

Queue-based processing pairs naturally with the gateway pattern — workers route their calls through the gateway to get caching and fallback benefits. See the queue-based AI processing guide for implementation details.

Pattern 5: Streaming Responses

Streaming sends tokens to the client as the LLM generates them instead of waiting for the full response. This transforms perceived latency from “10 seconds of nothing” to “instant start, progressive rendering.” For chat interfaces, it’s table stakes.

┌────────┐         ┌──────────┐         ┌─────────┐
│ Client  │◀──SSE──│  Server   │◀─stream─│  LLM    │
│         │  or WS │          │         │  API    │
└────────┘         └──────────┘         └─────────┘

Timeline:
  t=0ms    Request sent
  t=200ms  First token arrives → client renders
  t=300ms  More tokens stream in...
  t=2500ms Final token → stream closes

vs. non-streaming:
  t=0ms    Request sent
  t=2500ms Full response arrives → client renders

When to use it: Any user-facing interface where the AI generates more than a sentence or two. Chat applications, writing assistants, code generation tools, and search with AI summaries all benefit. The UX improvement is dramatic — users perceive streaming responses as 3-5x faster even though total generation time is identical.

Example: A customer support chatbot streams responses via Server-Sent Events. The server opens a streaming connection to the LLM API, and each chunk is forwarded to the browser as it arrives. The frontend appends tokens to the message bubble in real time. If the user navigates away, the server aborts the upstream stream to avoid wasting tokens.

Streaming adds complexity around error handling (what if the stream fails mid-response?) and token counting (you can’t know the final cost until the stream completes). For a Node.js implementation walkthrough, see streaming AI responses in Node.js.

Combining Patterns

These patterns aren’t mutually exclusive — production systems typically combine three or more. Here’s how they layer together:

┌─────────────────────────────────────────────────┐
│                  AI Gateway                      │
│  (auth, cache, logging, fallback, cost tracking) │
├─────────────────────────────────────────────────┤
│                                                  │
│  ┌─────────────┐  ┌──────────┐  ┌────────────┐ │
│  │ Multi-Model  │  │  RAG     │  │  Queue     │ │
│  │ Router       │  │  Pipeline│  │  Workers   │ │
│  └──────┬──────┘  └────┬─────┘  └─────┬──────┘ │
│         │              │              │         │
│         └──────────────┼──────────────┘         │
│                        │                         │
│              ┌─────────▼──────────┐              │
│              │  Streaming Layer    │              │
│              │  (for sync paths)   │              │
│              └────────────────────┘              │
└─────────────────────────────────────────────────┘

A typical production stack looks like this:

1. Gateway first — every AI call flows through it. This gives you caching, fallbacks, and observability from day one.
2. Router at the gateway level — classify incoming requests and route to the appropriate model. Simple tasks go to cheap models, complex tasks to capable ones.
3. RAG for knowledge-grounded features — plug the retrieval pipeline into the gateway so cached retrievals benefit all consumers.
4. Queues for background work — anything that doesn’t need a synchronous response gets queued. Workers pull from the queue and call through the gateway.
5. Streaming for user-facing paths — synchronous requests that reach the user get streamed for better perceived performance.

Where to Start

If you’re early in your build, start with the gateway. It’s the lowest-effort, highest-leverage pattern because it gives you a single control point for everything that follows. Add RAG when you need knowledge grounding, routing when costs matter, queues when you hit scale, and streaming when UX matters.

The key insight is that these are infrastructure patterns, not AI patterns. They’re the same ideas — gateways, queues, caching, routing — that backend engineers have used for decades, adapted for the specific constraints of LLM APIs: high latency, token-based pricing, rate limits, and non-deterministic outputs.

Build the architecture first. The AI part is the easy part.

Choosing the Right Starting Point

Your starting pattern depends on your primary constraint:

Constraint	Start with
Multiple teams calling LLMs	AI Gateway
Users need answers from your data	RAG
LLM costs are too high	Multi-model routing
High volume, not time-sensitive	Queue-based processing
Chat or interactive UI	Streaming

Most teams should start with the gateway regardless, then layer on the pattern that addresses their biggest pain point. You don’t need all five on day one — but knowing they exist means you can design your system to accommodate them later without a rewrite.

Further Reading

• The AI Gateway Pattern — Full Guide
• Building a RAG System That Scales
• Multi-Model Architecture in Practice
• Queue-Based AI Processing
• AI Fallback Patterns for Resilient Apps
• Caching Strategies for LLM APIs
• How to Reduce LLM API Costs
• AI Model Comparison Guide