ImagePullBackOff means Kubernetes failed to obtain a container image and is delaying repeated attempts. The pod has not reached application startup, so changing model-server probes or GPU settings will not fix the pull itself.

Read the pod events first

kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp

The event usually distinguishes:

• image or tag not found;
• registry authentication failure;
• missing imagePullSecret;
• DNS, TLS or network failure;
• registry throttling or outage;
• incompatible image manifest or node architecture;
• disk pressure while unpacking layers.

Do not recreate pods repeatedly before reading the message; backoff is a symptom, not the cause.

Verify the immutable image identity

Check the deployed image:

kubectl get pod POD_NAME -n NAMESPACE \
  -o jsonpath='{.spec.containers[*].image}{"\n"}'

Confirm repository, registry, tag and digest. For production inference, prefer a reviewed immutable digest so latest or a moved tag cannot silently change the runtime, CUDA stack or model server.

Fix private-registry authentication

The pull secret must exist in the pod’s namespace and be referenced by the pod template or ServiceAccount:

kubectl get secret -n NAMESPACE
kubectl get serviceaccount SERVICE_ACCOUNT -n NAMESPACE -o yaml
kubectl get deployment DEPLOYMENT -n NAMESPACE -o yaml

Do not print or commit decoded registry credentials. Rotate the secret if it is expired and roll out the workload through the normal secret-management process.

Check node and registry compatibility

AI images are often large and architecture-specific. Verify:

• node CPU architecture matches the image manifest;
• required GPU runtime and drivers are managed separately from image pull;
• nodes can resolve and reach the registry;
• private endpoints, proxies and CA trust are configured on every relevant node;
• node image filesystem has enough space for layers;
• registry policy allows the node identity to pull.

An image can pull successfully and still fail later because CUDA, driver or model requirements do not match. That later failure is CrashLoopBackOff or another container-state problem, not ImagePullBackOff.

Verify the rollout

kubectl rollout status deployment/DEPLOYMENT -n NAMESPACE
kubectl get pods -n NAMESPACE -w
kubectl describe pod NEW_POD -n NAMESPACE

Confirm the resolved image ID and run the inference readiness or smoke test. Monitor pull duration, registry errors and node disk pressure during rollout.

Connect this page to Kubernetes CrashLoopBackOff for AI workloads, AI container deployment and the AI Operations hub.

Prevent recurrence by pinning artifacts, scanning them before promotion, validating manifests in CI, testing registry access from new node pools and alerting on pull failures before a full rollout stalls.