You ship an AI feature. The demo looks incredible. Then real users hit it and stare at a spinner for eight seconds. Half of them leave.

This is the core tension of building with LLMs in 2026: the models are powerful, but they’re slow. A typical LLM call takes 2–30 seconds depending on the model, prompt length, and output size. Users trained on sub-second web apps won’t wait that long — at least not without the right UX to carry them through.

The good news is that perceived latency and actual latency are different things. You can’t make the model think faster, but you can make the experience feel fast. Here are five battle-tested patterns that production AI apps use to bridge the gap.

1. Streaming: Show Tokens as They Arrive

Instead of waiting for the full response and dumping it on screen, stream tokens to the user as the model generates them. The first token typically arrives in 200–500ms, which feels nearly instant. The user starts reading while the model is still thinking.

This is the single highest-impact latency pattern. If you’re building anything conversational — chatbots, writing assistants, code generators — streaming should be your default.

When to use it: Any interactive feature where the user is watching and waiting for text output.

// Server: pipe the stream straight through
app.post('/api/chat', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  const stream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: req.body.messages,
    stream: true,
  });
  for await (const chunk of stream) {
    const text = chunk.choices[0]?.delta?.content || '';
    if (text) res.write(`data: ${JSON.stringify({ text })}\n\n`);
  }
  res.end();
});

// Client: render tokens as they land
const source = new EventSource('/api/chat');
source.onmessage = (e) => {
  const { text } = JSON.parse(e.data);
  outputEl.textContent += text;
};

The key detail: don’t buffer. Every token that sits in a buffer is wasted perceived-speed. For a deeper dive on the server side, see our guide on streaming AI responses in Node.js.

2. Optimistic UI: Show a Placeholder, Fill It In

Sometimes you know the shape of the response before the model returns it. A product description will be a paragraph. A summary will be three bullet points. A classification will be one of five labels.

In these cases, render the UI skeleton immediately and fill in the AI-generated content when it arrives. The user sees progress, not a void.

When to use it: Structured outputs where the format is predictable — summaries, classifications, form auto-fills, generated cards.

function ProductDescription({ productId }) {
  const { data, isLoading } = useAIGenerate(`/api/describe/${productId}`);

  return (
    <div className="description">
      {isLoading ? (
        <>
          <SkeletonLine width="100%" />
          <SkeletonLine width="90%" />
          <SkeletonLine width="75%" />
        </>
      ) : (
        <p>{data.description}</p>
      )}
    </div>
  );
}

Pair this with a subtle animation on the skeleton and users perceive the wait as 30–40% shorter than a static spinner. This pattern fits naturally into the broader AI app architecture patterns we’ve covered previously.

3. Background Processing: Queue and Notify

Not every AI task needs a synchronous response. Document analysis, batch summarization, image generation, long-form content creation — these can take 30 seconds to several minutes. Forcing the user to wait on-screen is a losing strategy.

Instead, accept the request, drop it on a queue, and notify the user when it’s done. This decouples the user experience from model latency entirely.

When to use it: Any task where the result isn’t needed in the next 5 seconds — reports, bulk operations, content pipelines, heavy multi-step agent workflows.

// Accept and queue
app.post('/api/analyze-document', async (req, res) => {
  const jobId = crypto.randomUUID();
  await queue.send({
    jobId,
    userId: req.user.id,
    documentUrl: req.body.url,
  });
  res.json({ jobId, status: 'processing' });
});

// Worker processes async, notifies on completion
async function processJob(job) {
  const result = await analyzeDocument(job.documentUrl);
  await db.results.save({ jobId: job.jobId, result });
  await notify(job.userId, {
    title: 'Analysis complete',
    link: `/results/${job.jobId}`,
  });
}

The user gets an instant acknowledgment, goes about their work, and comes back when the result is ready. For implementation details on the queue infrastructure, check out queue-based AI processing.

4. Caching: Same Question, Cached Answer

A surprising amount of AI traffic is repetitive. Multiple users ask the same question. The same product needs the same description. The same document gets summarized by different team members.

Caching LLM responses — whether exact-match or semantic — can eliminate latency entirely for repeated queries. Response times drop from seconds to milliseconds.

When to use it: Any feature with repeated or similar inputs — FAQ bots, product descriptions, classification tasks, search-triggered summaries.

async function cachedCompletion(prompt: string) {
  const key = `llm:${hashPrompt(prompt)}`;
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const result = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }],
  });
  const output = result.choices[0].message.content;
  await redis.set(key, JSON.stringify(output), 'EX', 3600);
  return output;
}

For high-traffic apps, layer this with semantic caching — embedding the prompt and matching against similar past queries within a cosine-similarity threshold. Our breakdown of how prompt caching works covers both exact and semantic approaches.

5. Model Selection: Use a Faster Model for Simple Tasks

Not every request needs your most capable (and slowest) model. A typo correction doesn’t need GPT-4o. A sentiment classification doesn’t need Claude Opus. Routing simple tasks to smaller, faster models cuts latency dramatically — often from 5+ seconds down to under 500ms.

When to use it: When your app handles a mix of simple and complex tasks. Route based on task complexity, user tier, or required quality level.

function selectModel(task: string): string {
  const fastTasks = ['classify', 'extract-entity', 'fix-grammar'];
  return fastTasks.includes(task) ? 'gpt-4o-mini' : 'gpt-4o';
}

async function handleRequest(task: string, input: string) {
  const model = selectModel(task);
  return openai.chat.completions.create({
    model,
    messages: [{ role: 'user', content: input }],
  });
}

An AI gateway is the natural place to implement this routing logic — it gives you a single control plane for model selection, fallbacks, rate limiting, and cost tracking across your entire app.

Combining Patterns

These patterns aren’t mutually exclusive. The best production apps layer them:

• Chat interface: Streaming + caching (stream on cache miss, instant replay on cache hit)
• Content generation dashboard: Optimistic UI + background processing (show skeleton, queue the heavy work, notify on completion)
• API product: Model selection + caching (route to fast models first, cache everything)

The goal is always the same: minimize the time between the user’s action and visible feedback. Whether that feedback is a streaming token, a skeleton placeholder, or a “we’re working on it” confirmation — anything beats a blank screen and a spinner.

The Bottom Line

LLM latency isn’t going away. Models are getting faster, but they’ll always be slower than a database lookup. The apps that win are the ones that treat latency as a UX problem, not just an infrastructure problem.

Start with streaming — it’s the highest-impact, lowest-effort change. Add caching for repeated queries. Move heavy tasks to background queues. Use optimistic UI for structured outputs. Route simple tasks to fast models.

Your users don’t need the response to be instant. They need the experience to feel responsive. That’s a solvable problem.