A single LLM call takes anywhere from 2 to 30 seconds. Multiply that by a hundred concurrent users and your API server is toast — threads exhausted, timeouts everywhere, requests dropped on the floor. The fix isn’t more servers. It’s a queue.

A message queue decouples the moment a user submits a request from the moment the LLM finishes processing it. Your API stays fast, your workers chew through jobs at their own pace, and nobody loses data when traffic spikes. This is the same pattern that powers every serious AI product in production today.

Why You Can’t Just Call the LLM Inline

Most developers start with the obvious approach: user hits your endpoint, you call OpenAI (or Anthropic, or your self-hosted model), wait for the response, and send it back. This works fine in development. It falls apart in production for three reasons:

1. Latency kills connections. A 15-second LLM call ties up a connection the entire time. Load balancers, reverse proxies, and browsers all have timeout thresholds that you’ll hit regularly.
2. Concurrency is limited. Node.js can handle thousands of lightweight requests per second, but if each one blocks on a 10-second LLM call, you’re down to a handful of concurrent users before things degrade.
3. Provider rate limits bite hard. LLM APIs enforce tokens-per-minute and requests-per-minute limits. Without a queue, a traffic spike means you blow past those limits and start getting 429 errors. A queue lets you control the flow deliberately.

The queue pattern solves all three. Your API responds in milliseconds with a job ID, and the heavy lifting happens asynchronously.

The Pattern: Accept → Enqueue → Process → Deliver

Here’s the flow that works for AI workloads:

1. API accepts the request — validates input, creates a job ID, returns it immediately (HTTP 202 Accepted).
2. Job goes into the queue — the message contains the prompt, model config, user ID, and any context needed.
3. Worker picks up the job — calls the LLM, handles retries, stores the result.
4. Client gets the result — either via webhook callback, polling the job status endpoint, or a WebSocket push.

This is the same architecture pattern used by image generation APIs, document processing pipelines, and batch inference systems. The queue is the shock absorber between unpredictable user traffic and expensive, slow AI calls.

Choosing a Queue: BullMQ, SQS, or RabbitMQ

Three options cover the vast majority of AI queue use cases:

BullMQ + Redis — The go-to for Node.js. BullMQ gives you priority queues, delayed jobs, rate limiting, retries with backoff, and a built-in dashboard (Bull Board). Redis is the broker. If you’re already running Redis for caching, this is the lowest-friction option.

AWS SQS — Fully managed, zero infrastructure to maintain. SQS scales to virtually unlimited throughput and pairs naturally with Lambda for serverless worker functions. The trade-off is less fine-grained control over job priority and scheduling compared to BullMQ. Works well if your AI gateway is already on AWS.

RabbitMQ — The most flexible option for complex routing. If you need to fan out a single request to multiple models, route different job types to specialized workers, or implement sophisticated dead-letter handling, RabbitMQ gives you the most control. Heavier to operate than the other two.

For most teams building AI features into a web app, BullMQ + Redis hits the sweet spot of simplicity and power.

Implementation: Node.js + BullMQ

Here’s a working implementation. Install dependencies first:

npm install bullmq ioredis openai express

Producer — the API endpoint that enqueues jobs:

import express from "express";
import { Queue } from "bullmq";
import { randomUUID } from "crypto";

const aiQueue = new Queue("ai-jobs", {
  connection: { host: "127.0.0.1", port: 6379 },
});

const app = express();
app.use(express.json());

app.post("/api/generate", async (req, res) => {
  const jobId = randomUUID();
  await aiQueue.add("llm-call", { prompt: req.body.prompt, jobId }, {
    jobId,
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
  });
  res.status(202).json({ jobId, status: "queued" });
});

app.get("/api/jobs/:id", async (req, res) => {
  const job = await aiQueue.getJob(req.params.id);
  if (!job) return res.status(404).json({ error: "not found" });
  const state = await job.getState();
  res.json({ jobId: job.id, state, result: job.returnvalue });
});

app.listen(3000);

Worker — processes jobs from the queue:

import { Worker } from "bullmq";
import OpenAI from "openai";

const openai = new OpenAI();

const worker = new Worker("ai-jobs", async (job) => {
  const completion = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: job.data.prompt }],
  });
  return { text: completion.choices[0].message.content };
}, {
  connection: { host: "127.0.0.1", port: 6379 },
  concurrency: 5,
});

worker.on("completed", (job) => console.log(`Done: ${job.id}`));
worker.on("failed", (job, err) => console.error(`Failed: ${job?.id}`, err.message));

The API responds instantly. The worker processes jobs at a controlled concurrency of 5. The client polls /api/jobs/:id until the state flips to completed and reads the result.

Retries and Dead Letter Queues

LLM APIs fail. Rate limits, network blips, model overload — you need a retry strategy. The BullMQ config above already handles this: 3 attempts with exponential backoff starting at 5 seconds (so retries at ~5s, ~10s, ~20s).

But what about jobs that fail all retries? That’s where dead letter queues (DLQs) come in. A DLQ captures permanently failed jobs so you can inspect them, alert on them, and replay them later:

const aiQueue = new Queue("ai-jobs", {
  connection: { host: "127.0.0.1", port: 6379 },
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
    removeOnComplete: 1000,
    removeOnFail: false, // keep failed jobs for inspection
  },
});

Set up an alert when the failed job count crosses a threshold. This is your early warning system for provider outages or broken prompts. Tracking failures also helps you reduce costs by identifying prompts that consistently fail and waste tokens on retries.

Scaling Workers

The beauty of the queue pattern is that scaling is trivial. Need more throughput? Add more workers. Each worker is an independent process that can run on a separate machine, container, or serverless function.

Practical scaling strategies:

• Adjust concurrency per worker. Start with concurrency: 5 and tune based on your LLM provider’s rate limits. If you’re allowed 60 requests per minute, running 5 concurrent workers with 2-second average response times keeps you right at the limit.
• Horizontal scaling. Run multiple worker processes behind a process manager (PM2) or as separate containers in ECS/Kubernetes. BullMQ handles job distribution automatically — no coordination needed.
• Auto-scale on queue depth. Monitor the number of waiting jobs. When the queue grows beyond a threshold, spin up additional workers. Scale back down when it drains. This is where SQS + Lambda shines — AWS handles the scaling for you.
• Priority queues. Not all jobs are equal. Paid users get priority 1, free users get priority 10. BullMQ supports this natively, so your paying customers never wait behind a flood of free-tier requests.

Monitoring Queue Depth

A queue you can’t see is a queue that will hurt you. Monitor these metrics:

• Waiting count — jobs in the queue not yet picked up. If this grows steadily, you need more workers.
• Active count — jobs currently being processed. This should stay close to your total concurrency.
• Failed count — jobs that exhausted all retries. Spike here means something is wrong upstream.
• Job latency — time from enqueue to completion. This is the number your users actually feel.

BullMQ exposes all of these through its API. Bull Board gives you a web dashboard out of the box. For production, push these metrics to your observability stack (Datadog, Grafana, CloudWatch) and set alerts on queue depth and failure rate.

When to Use This Pattern

Use a queue for AI processing when:

• LLM calls take more than 1-2 seconds (most do)
• You need to survive traffic spikes without dropping requests
• You want to enforce rate limits against your LLM provider
• You need retry logic for unreliable API calls
• You’re running batch processing or async generation

Skip the queue if you’re building a real-time chat interface where streaming responses directly to the user is the whole point. Even then, you might still want a queue for the non-streaming parts of your pipeline (embeddings, summarization, background tasks).

The queue is the most underrated piece of AI application architecture. It’s not glamorous, but it’s the difference between a demo that works and a product that stays up under load.