Kubernetes is a control system for running containerized workloads across a cluster. You describe the desired state—images, replicas, resources, networking, and rollout policy—and controllers keep working to make reality match it.
For AI teams, its distinctive value is not “more containers.” It is coordinating scarce hardware, slow model startup, multiple inference services, and changing demand across machines. Many AI applications still do not need it.
The core objects
- Cluster: the machines and control plane Kubernetes manages.
- Node: one machine, possibly with a particular GPU type.
- Pod: the smallest scheduled unit, containing one or more tightly coupled containers.
- Deployment: keeps a chosen number of replaceable pods running and manages rollouts.
- Service: gives changing pods a stable network endpoint.
- Job: runs finite work such as evaluation, embedding, or batch inference.
Kubernetes’ official workloads documentation describes these general primitives. AI platforms add model-aware controllers and metrics on top.
What Kubernetes changes
Without orchestration, operators choose servers, start processes, replace failures, and coordinate releases. Kubernetes automates much of that infrastructure loop. It does not automatically:
- choose a good model;
- know whether output quality regressed;
- make a model fit in GPU memory;
- secure an agent’s tools;
- create an economical scaling policy.
Those remain application and platform decisions.
AI workload decision table
| Workload | Start with | Kubernetes becomes plausible when |
|---|---|---|
| App calling hosted model APIs | serverless/PaaS/container | many internal services need shared policy and operations |
| One local model on one GPU server | Docker/system service | replicas or several GPU node types must be scheduled |
| Bursty inference | managed endpoint | custom runtime, hardware, or scaling control is required |
| Shared LLM platform | managed inference or K8s | teams need routing, rollouts, quotas, caches, and observability |
| Multi-node training/inference | managed training platform | you have the expertise to operate topology-aware workloads |
The number of microservices is not a reliable threshold. Operational constraints are.
How GPUs fit
Kubernetes exposes GPUs through vendor device plugins. Pods request a custom resource such as nvidia.com/gpu; labels and placement rules select compatible nodes. The official GPU scheduling guide documents the mechanism and its constraints.
Kubernetes schedules the device. Your model runtime still owns batching, quantization, tensor parallelism, memory management, and token throughput.
Model-serving platforms
A plain Deployment can expose a model API, but production systems also need model loading, readiness, scaling, routing, and controlled releases. KServe provides Kubernetes resources for predictive and generative inference, including LLM runtimes, streaming endpoints, model caching, and AI-oriented autoscaling.
If that is your problem, continue with Kubernetes for AI inference. It covers GPU manifests, probes, KServe, scaling, rollouts, and observability.
Alternatives first
- Hosted model API: no inference infrastructure; pay per use and accept provider constraints.
- Managed inference endpoint: deploy your model without owning the orchestration layer.
- Docker Compose: excellent for local development and one-server services.
- Serverless/PaaS: suitable for the application layer around hosted models.
- Managed Kubernetes: removes control-plane work, not workload, networking, security, or cost work.
Compare AI compute economics in serverless versus dedicated GPU.
Questions to answer before adopting it
- Which workload cannot be operated safely on a simpler platform?
- Who owns upgrades, networking, RBAC, storage, observability, and incident response?
- Which GPU types, model sizes, and startup times must the scheduler account for?
- What metric should scale capacity: queue depth, latency, active sequences, or tokens?
- How will a model release be evaluated and rolled back?
- What is the idle cost of keeping suitable GPU capacity available?
Kubernetes is justified when its scheduling and control loops solve constraints you actually have. It is not a badge of production maturity. If you already operate a cluster, AI Kubernetes troubleshooting provides a workload-specific diagnostic path.