“Design an AI chatbot like ChatGPT.” You’ll hear this in system design interviews at every major tech company now. It touches on real-time streaming, stateful conversations, retrieval-augmented generation, and scaling GPU-bound workloads — all in one question.

Here’s how to walk through it step by step, the same way you would in a 45-minute interview.

1. Requirements

Start by clarifying scope. Don’t jump into boxes and arrows.

Functional requirements:

• Users send text messages and receive AI-generated responses
• Multi-turn conversations with memory of prior messages
• Streaming token-by-token responses (not waiting for full completion)
• Optional retrieval-augmented generation (RAG) over uploaded documents
• Conversation history — list, continue, and delete past chats

Non-functional requirements:

• Low latency to first token (< 500ms excluding model inference)
• High availability (99.9%+)
• Horizontal scalability — handle millions of concurrent conversations
• Rate limiting and usage-based billing per user/org
• Data privacy — conversations encrypted at rest and in transit

Scope out for the interview: voice, image generation, plugins, fine-tuning. Mention them, then move on.

2. High-Level Architecture

Here’s the bird’s-eye view. If you’ve read through common AI app architecture patterns, this will look familiar.

┌──────────┐       ┌──────────────┐       ┌───────────────┐
│  Client   │──────▶│  API Gateway │──────▶│  Chat Service │
│ (Web/App) │◀──SSE─│  + Auth/Rate │       │  (Stateless)  │
└──────────┘       └──────────────┘       └───────┬───────┘
                                                  │
                          ┌───────────────────────┼───────────────────┐
                          │                       │                   │
                   ┌──────▼──────┐   ┌────────────▼───┐   ┌──────────▼──────┐
                   │ Conversation │   │  LLM Inference  │   │  RAG / Retrieval│
                   │   Store      │   │  Workers (GPU)  │   │  Service        │
                   │  (Postgres)  │   │                 │   │                 │
                   └─────────────┘   └────────┬────────┘   └────────┬────────┘
                                              │                     │
                                     ┌────────▼────────┐   ┌───────▼────────┐
                                     │  Model Registry  │   │  Vector DB     │
                                     │  + Prompt Cache  │   │  (Pinecone/    │
                                     │                  │   │   pgvector)    │
                                     └─────────────────┘   └────────────────┘

Key decisions:

• Chat Service is stateless. Conversation state lives in the database, not in memory. This lets us scale horizontally without sticky sessions — similar to the approach in a standard chat app design.
• LLM Inference Workers are separate. GPU resources are expensive and scale differently from CPU-bound API servers.
• RAG is a sidecar service. Not every request needs retrieval, so we keep it decoupled.

3. API Design

Keep it REST for CRUD, SSE for streaming.

POST   /v1/conversations                    → Create conversation
GET    /v1/conversations                    → List conversations
DELETE /v1/conversations/:id                → Delete conversation

POST   /v1/conversations/:id/messages       → Send message (returns SSE stream)
GET    /v1/conversations/:id/messages       → Get message history

The POST /messages endpoint is the interesting one. The request body:

{
  "role": "user",
  "content": "Explain how load balancers work",
  "stream": true,
  "model": "gpt-5",
  "rag_context": { "collection_id": "abc123" }
}

The response is a Server-Sent Events stream. Each event carries a delta token, and the final event includes usage metadata (token counts, latency). This is the same pattern covered in depth in streaming AI responses with Node.js.

4. Conversation Storage

Conversations are relational data with a clear parent-child structure. Postgres is the right default.

conversations
├── id (UUID, PK)
├── user_id (FK)
├── title
├── model
├── created_at
└── updated_at

messages
├── id (UUID, PK)
├── conversation_id (FK)
├── role (enum: system | user | assistant)
├── content (text)
├── token_count (int)
├── created_at
└── metadata (jsonb) — latency, model version, finish_reason

Why not a document store? You could use MongoDB or DynamoDB here, but conversations have predictable structure, and you’ll want to run analytics queries (tokens per user, avg conversation length). Postgres with jsonb for flexible metadata gives you both.

Context window management: When a conversation exceeds the model’s context window, you need a strategy. Options:

1. Sliding window — send only the last N messages
2. Summarization — periodically compress older messages into a summary message
3. Hybrid — summarize old context, keep recent messages verbatim

Most production systems use option 3. Store the summary as a special system message at the start of the context.

5. Streaming Responses

The user expects to see tokens appear as the model generates them. The flow:

Client ──POST──▶ Chat Service ──gRPC stream──▶ Inference Worker
  ◀──SSE──────────────◀──────── token deltas ◀────────────────

1. Client opens an SSE connection via POST /messages
2. Chat Service persists the user message, assembles the prompt (conversation history + system prompt + optional RAG context)
3. Chat Service opens a gRPC stream to an Inference Worker
4. Tokens flow back through the chain — each delta is flushed to the client immediately
5. On stream completion, Chat Service persists the full assistant message and closes the SSE connection

Backpressure: If the client disconnects mid-stream, the Chat Service must detect this and cancel the inference request. Wasting GPU cycles on abandoned requests is expensive.

6. RAG Integration

When a user uploads documents or enables knowledge base search, the request path adds a retrieval step before inference. For a deeper dive, see building a local RAG pipeline.

Chat Service ──query──▶ RAG Service ──embed──▶ Embedding Model
                                     ──search─▶ Vector DB
                        ◀── top-k chunks ◀─────────────────

1. RAG Service embeds the user’s query using the same embedding model used at ingestion time
2. Vector DB returns the top-k most similar chunks (typically k=5–10)
3. Chunks are injected into the system prompt as grounding context
4. The LLM generates a response grounded in the retrieved documents

Chunk size matters. Too small and you lose context. Too large and you waste tokens. 512–1024 tokens per chunk with 10–20% overlap is a solid starting point.

7. Rate Limiting and Billing

AI inference is expensive. You need rate limiting at multiple layers.

Layer	What it limits	Implementation
API Gateway	Requests per minute per user	Token bucket (Redis)
Chat Service	Tokens per minute per org	Sliding window counter
Inference	Concurrent requests per model	Semaphore / queue

Billing is token-based. Log input tokens and output tokens per request to a usage table. Aggregate asynchronously for billing. Don’t compute bills in the hot path.

The API Gateway handles auth and coarse rate limiting. For details on how that layer works under the hood, see how load balancers actually work.

8. Scaling

Each component scales differently:

Chat Service (CPU-bound, stateless): Horizontal auto-scaling behind a load balancer. Scale on request count and CPU. This is the easy part.

Inference Workers (GPU-bound): The bottleneck. Strategies:

• Request batching — batch multiple prompts into a single GPU forward pass (continuous batching)
• Model parallelism — split large models across multiple GPUs
• Prompt caching — cache common prompt prefixes to skip redundant KV-cache computation. This alone can cut latency 30–50% for repeated system prompts. See how prompt caching works for the mechanics.
• Queue-based dispatch — use a job queue (Redis Streams, SQS) to buffer requests when all workers are busy, with priority lanes for paid tiers

Caching layers:

• Conversation cache (Redis): Cache recent conversation history to avoid DB reads on every message. TTL of 30 minutes after last activity.
• Embedding cache: Cache embeddings for frequently asked queries to skip the embedding model call.
• CDN: Serve the web client, static assets, and documentation from a CDN. Not relevant to the inference path, but it reduces load on your origin servers.

Database scaling: Partition the messages table by conversation_id. Most queries are scoped to a single conversation, so partition pruning keeps things fast. Read replicas for analytics workloads.

9. Monitoring

You can’t improve what you don’t measure. Key metrics:

Latency:

• Time to first token (TTFT) — the most user-visible metric
• Total generation time
• RAG retrieval latency (p50, p99)

Throughput:

• Tokens per second per worker
• Concurrent active streams
• Queue depth (inference backlog)

Quality:

• Hallucination rate (sampled, human-reviewed)
• RAG retrieval relevance scores
• User feedback signals (thumbs up/down)

Cost:

• GPU utilization per worker
• Tokens served per dollar
• Cache hit rates (prompt cache, conversation cache, embedding cache)

Set alerts on TTFT p99 > 2s, queue depth > 100, and GPU utilization < 30% (you’re over-provisioned) or > 90% (you’re about to drop requests).

---

Wrapping Up

The “design an AI chatbot” question tests whether you can reason about GPU-bound workloads, streaming protocols, and stateful-but-scalable storage — all things that don’t come up in a typical web app design.

Hit these points in your interview and you’ll stand out:

1. Separate inference workers from stateless API servers
2. Stream tokens via SSE, with backpressure handling
3. Store conversations in Postgres, manage context windows explicitly
4. Decouple RAG as an optional retrieval step
5. Rate limit at multiple layers — gateway, service, and inference
6. Cache aggressively — prompt prefixes, conversations, embeddings

The architecture patterns here apply well beyond chatbots. If you’re building any AI-powered product, the same separation of concerns holds. Start with the AI app architecture patterns overview for the broader picture.