Serverless means you deploy code without managing a long-lived server. The provider allocates execution capacity, scales instances, and charges according to its plan and usage model.

For AI applications, serverless is often a strong orchestration layer around hosted models. It is usually a poor place to load a large model into memory on every cold start.

Where it fits

Good serverless AI workloads include:

• authenticated API endpoints that call hosted model providers;
• retrieval and prompt-assembly endpoints with bounded work;
• webhook ingestion for background model jobs;
• lightweight tool endpoints for agents;
• scheduled evaluations or cleanup;
• streaming a provider response to a browser when the platform supports it.

The function should validate input, enforce policy and budgets, call the model/tool, and persist necessary state. It should not rely on one process staying alive forever.

Where it breaks

Be cautious when the workload needs:

• a large model or GPU runtime loaded in local memory;
• predictable low latency with expensive cold initialization;
• an agent loop that may run beyond function-duration limits;
• high-volume steady compute that is cheaper on reserved capacity;
• durable in-memory state or a long-lived connection unsupported by the chosen platform;
• local disk state that must survive instance replacement.

Use a dedicated or managed inference service for the model and keep the serverless application layer thin. The serverless versus dedicated GPU comparison examines that split.

A minimal model proxy

export async function POST(request: Request) {
  const { prompt } = await request.json();
  if (typeof prompt !== 'string' || prompt.length > 8_000) {
    return Response.json({ error: 'Invalid prompt' }, { status: 400 });
  }

  // Call a configured model provider from the server.
  // Add authentication, timeout, spend limits, and abuse controls here.
  return Response.json({ status: 'accepted' }, { status: 202 });
}

The important part is the boundary: keys stay server-side, requests are bounded, and long work can move to a queue or durable workflow.

Streaming versus background work

Use streaming when a user is actively waiting and partial output improves the experience. Streaming AI responses in Node.js covers the HTTP pattern.

Use background work when execution must survive a disconnect, retry, approval delay, or deployment. Persist a job ID and deliver status through polling, a webhook, or a durable workflow. See long-running AI agents for the state model.

Serverless can be stateful—with another primitive

“Stateless functions” is a useful default, not a universal law. Providers now offer durable workflows, queues, databases, and stateful coordination. Cloudflare Durable Objects combine compute and durable storage and explicitly support agents and real-time applications. Vercel’s Fluid compute reuses function instances and separates active CPU from waiting time.

Design against the documented product, not a generic assumption about all serverless platforms.

Limits change, so verify them

Old rules such as “functions always time out after 10–30 seconds” are no longer reliable. Vercel’s current function limits, for example, vary by runtime, plan, and Fluid compute configuration.

Before choosing a provider, verify:

• maximum duration and streaming behavior;
• request/response and bundle sizes;
• concurrency and regional execution;
• background-task and queue semantics;
• connection support;
• pricing for CPU, memory, invocations, and network transfer;
• data residency, retention, and logging controls.

Cold starts and cost

Cold-start impact depends on runtime, bundle size, dependencies, region, and provider reuse. Measure it with your application. Avoid loading large SDK trees or model assets on every invocation.

Serverless can be inexpensive at low or bursty traffic, but “zero requests means zero cost” is too broad once databases, queues, observability, storage, and minimum plan fees are included. Model-provider tokens may dominate the bill anyway. Apply per-user budgets and hard spend controls.

Production checklist

• Authenticate before model or tool work.
• Bound prompt size, runtime, concurrency, retries, and spend.
• Use idempotency keys for retryable mutations.
• Store durable job state outside process memory.
• Stream only when the client benefits; queue work that must survive.
• Capture request IDs, model, latency, tokens, failures, and cost signals.
• Redact sensitive prompts and outputs from default logs.
• Test provider timeout, disconnect, duplicate event, and deployment during work.

Serverless is excellent glue for AI applications. It becomes a trap when a thin orchestration function quietly turns into a model host, an unbounded agent runtime, or the only place important state exists.