An AI gateway gives your application one stable endpoint while model providers, credentials, prices, and model names change behind it. It can enforce authentication, budgets, timeouts, logging policy, and routing once instead of duplicating those controls in every feature.

This tutorial builds a deliberately small gateway. It is suitable for learning and an internal prototype—not a claim that 150 lines replace a mature production gateway.

What the gateway owns

The gateway should own cross-provider concerns:

• application authentication and tenant identity;
• a stable internal request format;
• provider selection;
• model allowlists;
• timeouts and cancellation;
• rate and spending limits;
• request IDs and redacted logs;
• usage normalization;
• fallback policy.

Business prompts and feature-specific tool definitions should stay in the application layer.

Provider routing and cost policy

A gateway should route by application policy, not by whichever model name a client submits. Define aliases such as fast, reasoning, or private, then map each alias to an approved provider and model. A route can consider capability, regional availability, latency targets, data policy, and an explicit cost ceiling.

Direct OpenAI or Anthropic integrations give you provider-specific features and support paths. A multi-provider service such as OpenRouter can simplify access and fallback, but introduces another operational dependency. Compare the trade-offs rather than assuming one path is universally cheaper or more reliable.

Store input, output, cached-token, and other billable usage separately. Provider price sheets and model identifiers change, so keep a versioned pricing table and calculate estimated cost outside the request handler. The AI pricing directory is useful for comparison, but your production controls should use prices you have verified for the exact provider and account.

Install the example

mkdir ai-gateway && cd ai-gateway
npm init -y
npm install express

Use Node.js 20 or newer so the example can use the built-in fetch API.

Define a stable request contract

Clients should not send arbitrary provider URLs or provider credentials.

{
  "task": "support-summary",
  "model": "fast",
  "messages": [
    { "role": "user", "content": "Summarize this ticket" }
  ]
}

model: "fast" is an application alias. The gateway maps it to an approved provider model. That separation lets you migrate without changing every client.

Minimal gateway

import express from 'express';
import crypto from 'node:crypto';

const app = express();
app.use(express.json({ limit: '1mb' }));

const models = {
  fast: {
    url: 'https://api.provider.example/v1/responses',
    key: process.env.FAST_MODEL_API_KEY,
    upstreamModel: process.env.FAST_MODEL_ID
  },
  capable: {
    url: 'https://api.second-provider.example/v1/responses',
    key: process.env.CAPABLE_MODEL_API_KEY,
    upstreamModel: process.env.CAPABLE_MODEL_ID
  }
};

function authenticate(req, res, next) {
  const key = req.get('x-app-key');
  if (!key || key !== process.env.INTERNAL_APP_KEY) {
    return res.status(401).json({ error: 'unauthorized' });
  }
  next();
}

app.post('/v1/generate', authenticate, async (req, res) => {
  const requestId = crypto.randomUUID();
  const target = models[req.body.model];

  if (!target?.key || !target.upstreamModel) {
    return res.status(400).json({ error: 'unsupported_model', requestId });
  }

  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 30_000);

  try {
    const upstream = await fetch(target.url, {
      method: 'POST',
      headers: {
        authorization: `Bearer ${target.key}`,
        'content-type': 'application/json',
        'x-request-id': requestId
      },
      body: JSON.stringify({
        model: target.upstreamModel,
        messages: req.body.messages
      }),
      signal: controller.signal
    });

    const payload = await upstream.json();
    if (!upstream.ok) {
      console.error(JSON.stringify({ requestId, status: upstream.status }));
      return res.status(502).json({ error: 'model_provider_failed', requestId });
    }

    res.json({ requestId, output: payload });
  } catch (error) {
    const code = error.name === 'AbortError' ? 'model_timeout' : 'gateway_error';
    res.status(503).json({ error: code, requestId });
  } finally {
    clearTimeout(timeout);
  }
});

app.listen(process.env.PORT || 3000);

The example intentionally returns a generic provider error. Do not forward raw upstream errors if they can contain request content, account information, or internal details.

Add provider adapters

Providers do not share one exact request or response schema. Keep each translation in a small adapter:

const adapters = {
  providerA: { buildRequest, normalizeResponse, normalizeUsage },
  providerB: { buildRequest, normalizeResponse, normalizeUsage }
};

Normalize only what your application needs. A lowest-common-denominator interface can hide useful capabilities such as tool calls, reasoning controls, structured output, or prompt caching.

Rate limits and budgets

Production controls should consider:

• requests per user and tenant;
• concurrent generations;
• input and output tokens;
• daily or monthly spend;
• model-specific quotas;
• background-job capacity.

An in-memory counter is not sufficient across multiple instances. Use a shared store or a managed gateway. See rate limiting AI APIs and how rate limiting works.

Fallbacks require policy

Do not retry every failure against every model. A safe policy distinguishes:

• a transient network error;
• provider throttling;
• invalid credentials;
• a rejected input;
• an exceeded context window;
• a client cancellation.

Fallback only when the second model is acceptable for that task. Record which model ultimately answered. Never retry a consequential tool action unless it is idempotent.

Logging without leaking data

Log metadata such as request ID, tenant, task, provider, model alias, latency, status, token usage, estimated cost, and fallback count.

Do not log provider keys. Avoid full prompts and outputs by default. If content logging is required for evaluation, establish retention, access control, redaction, consent, and deletion rules.

Streaming and asynchronous jobs

For streaming, preserve cancellation: when the browser disconnects, abort the upstream request when practical. For long-running media generation or agent jobs, return a job ID and deliver state through polling or reliable webhooks.

Build or buy?

Build a small gateway when you need a narrow internal contract and want to understand the control plane. Consider a managed or open-source gateway when you need high availability, distributed rate limits, provider integrations, analytics, policy management, caching, or enterprise identity.

Whichever path you choose, keep the application contract yours. Continue with AI Application Architecture, AI Security, and API authentication.