A Pending pod has been accepted but cannot yet run. In an AI cluster, the usual causes are unavailable GPU capacity, incompatible node selection, requests larger than any node, quota, taints, or storage that cannot bind.

Read the scheduler message first

kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp
kubectl get nodes -o wide

The Events section often identifies Insufficient nvidia.com/gpu, insufficient memory, an unmatched node selector, untolerated taint, or unbound PersistentVolumeClaim. Fix that condition rather than applying a generic restart.

GPU availability

Confirm that GPU nodes exist, are Ready, expose the expected schedulable resource, and have compatible drivers, runtime integration, and device plugins. A physical GPU is not schedulable until the cluster advertises it correctly.

kubectl describe node NODE_NAME
kubectl get nodes -L accelerator

Check whether another workload holds the device and whether autoscaling can provision the requested GPU type in the selected region. Cloud quota and actual capacity are separate constraints.

Requests that cannot fit

A pod requesting one GPU and 96 GiB RAM needs a node satisfying both at once. Lowering requests only to make scheduling succeed can create an immediate OOM failure. Size the model, runtime, KV cache, and concurrency first; then request honest capacity.

Selectors, affinity, and taints

Review nodeSelector, node affinity, topology constraints, and tolerations together. Labels may refer to a GPU family or zone that no current node provides. Broad tolerations can schedule expensive or sensitive workloads in unintended pools, so change policy deliberately.

Storage binding

Model weights or caches may use a PVC restricted to a zone or storage class. With delayed binding, scheduling and volume placement interact. Inspect the claim and storage events rather than assuming GPU scarcity.

Capacity planning

Track Pending duration by GPU type, requested RAM, queue depth, node provisioning time, model load time, and idle accelerator cost. Keep latency-sensitive inference separate from batch jobs when they compete for scarce devices. Apply admission limits so a burst does not create an unbounded expensive queue.

Resolution checklist

• Scheduler event identified and recorded.
• Requested GPU resource is advertised by Ready nodes.
• RAM, CPU, and GPU requests can fit on one eligible node.
• Selectors, affinity, taints, quota, and namespace policy agree.
• PVC and zone constraints can bind.
• Autoscaler has cloud quota and real regional capacity.
• The resulting pod has enough memory to avoid OOMKilled.

Connect this diagnosis to Kubernetes memory failures, pod evictions, and the AI Operations hub.