Kubernetes can operate a shared AI inference platform across CPU and GPU nodes, but it is not the default answer for every AI application. If you call hosted model APIs, or one Docker host can serve your local model reliably, Kubernetes may add more operational work than value.

It becomes useful when placement, capacity, rollout, and isolation decisions must be automated across a cluster.

When Kubernetes is justified

Good signals include:

• several models or inference services share heterogeneous GPU nodes;
• replicas must roll out without dropping long-running requests;
• queue depth, token throughput, or GPU utilization should drive scaling;
• model downloads and cold starts need coordinated caching;
• online inference, batch evaluation, and background agents share infrastructure;
• a platform team can own upgrades, networking, storage, policy, and observability.

If none apply, compare hosted APIs, a managed inference endpoint, or one Docker host first. The beginner Kubernetes decision guide starts from that choice.

Core objects, translated to inference

• Pod: one or more tightly coupled containers, such as a model server and metrics sidecar.
• Deployment: a replicated stateless API or model gateway.
• Service: a stable endpoint in front of replaceable pods.
• Job: a finite evaluation, embedding, or batch-inference run.
• ConfigMap and Secret: non-sensitive configuration and credentials kept outside images.
• PersistentVolume: model cache or data that must survive pod replacement.

Kubernetes manages desired state; it does not understand tokens, model readiness, KV caches, or hallucinations unless you expose those signals.

Requesting GPUs

Kubernetes exposes vendor devices through device plugins. Its official GPU scheduling guide documents GPUs as custom resources such as nvidia.com/gpu:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      nodeSelector:
        accelerator: nvidia-l40s
      containers:
        - name: inference
          image: registry.example.com/model-server:tested
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: 16Gi
            limits:
              nvidia.com/gpu: 1

Install the matching vendor driver and device plugin on nodes. Labels, affinity, and taints keep expensive GPU workloads on compatible hardware. Do not assume a generic GPU request captures VRAM size, interconnect topology, or quantization compatibility.

Model serving is more than a Deployment

A plain Deployment is fine for a small internal service. At scale you also need model acquisition, readiness, routing, autoscaling, canary releases, and metrics.

KServe extends Kubernetes for predictive and generative inference. Its current generative stack supports LLM serving, OpenAI-compatible endpoints, model caching, multi-node inference, and metrics such as token throughput or queue depth. The KServe deployment-mode guide recommends standard Kubernetes deployments for GPU-heavy generative inference and distinguishes them from scale-to-zero predictive workloads.

For direct runtime configuration, start with serving LLMs using vLLM. A controller does not remove the need to understand the runtime.

Readiness, startup, and model caches

Large weights can take minutes to download and load. Configure separate probes:

• startupProbe allows slow initialization without restart loops;
• readinessProbe removes a pod from traffic until the model can answer;
• livenessProbe detects a stuck process after startup.

Cache models on suitable local or persistent storage, but version the cache key. A stale cache serving the wrong model is worse than a slow cold start.

Scaling AI workloads

The built-in HorizontalPodAutoscaler handles CPU, memory, and custom metrics. Kubernetes documents workload and event-driven approaches in its autoscaling guide.

CPU is often a poor proxy for GPU inference demand. Better signals can include:

• queued requests;
• time to first token;
• active sequences;
• tokens generated per second;
• GPU utilization and memory pressure;
• provider or downstream saturation.

Scaling pods is not enough if no compatible node has capacity. Pair workload scaling with node autoscaling and account for GPU provisioning and model-load delay. For tightly coupled distributed workloads, topology matters; Kubernetes now documents topology-aware scheduling for workloads including distributed AI/ML, but feature maturity must be checked against your cluster version.

Rollouts without model surprises

Treat a model change like an application release:

1. identify model, tokenizer, runtime, quantization, and prompt/config versions;
2. warm the candidate before sending traffic;
3. run health and quality evaluations;
4. shift a small traffic percentage;
5. watch latency, errors, cost, and task-quality signals;
6. preserve a fast rollback path.

Kubernetes can perform the rollout. It cannot decide whether the new model is behaviorally acceptable.

Troubleshooting sequence

Start with:

kubectl get pods -o wide
kubectl describe pod <name>
kubectl logs <name> --previous
kubectl top pod <name>

Then separate scheduling, image/model acquisition, startup, runtime, and traffic problems. AI Kubernetes troubleshooting covers that diagnostic flow, while vLLM CUDA out-of-memory focuses on accelerator pressure.

Security and observability

Use workload identities instead of long-lived cloud keys, restrict egress, apply least-privilege RBAC, isolate namespaces according to threat boundaries, and avoid placing secrets in images or ConfigMaps. Tool-using agents need a separate action policy; Kubernetes isolation does not make their tools safe.

Measure infrastructure and model behavior together: request rate, queue time, time to first token, inter-token latency, tokens per second, GPU memory, restart count, model version, cost, and quality/evaluation results. The broader production agent deployment guide covers application-level controls.

Final decision

Kubernetes earns its place when AI infrastructure needs shared scheduling, controlled rollouts, hardware-aware placement, and a platform team to operate it. It is unnecessary overhead when a hosted API or one well-managed inference server solves the problem. Make that decision from workload constraints, not from the size of the Kubernetes ecosystem.