No space left on device

On an AI server, a full disk can interrupt model downloads, corrupt an incomplete cache, prevent containers from starting or stop logs and databases from accepting writes. The cause may be model weights, Docker layers, build caches, inference logs—or exhausted inodes rather than bytes.

Do not begin with a broad delete command. Identify the filesystem and data owner first.

Immediate diagnosis

df -h
df -i
sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /var 2>/dev/null | sort -h
docker system df -v

df reports filesystem capacity; du reports reachable files. A large difference can mean deleted files are still held open:

sudo lsof +L1

Restart or rotate the owning service deliberately rather than deleting unrelated data.

Model weights and inference caches

Local model systems can keep multiple quantizations, revisions and partial downloads. Before removing anything:

1. Identify which runtime owns the directory.
2. List models through that runtime where possible.
3. Confirm which revisions production uses.
4. Remove unused models with the runtime’s supported command.
5. Keep persistent model data outside ephemeral container layers.

Do not assume every large file in a home directory is disposable. A cache may be shared by active services. For layout decisions, see the Linux foundation for AI servers and AI Deployment & Hosting.

Docker images, layers and volumes

Inspect before pruning:

docker image ls
docker container ls -a
docker volume ls
docker builder du

Remove only a confirmed unused image, stopped container or build cache. docker system prune can help, but -a widens the scope and --volumes may remove persistent data. Never schedule indiscriminate volume pruning on a production AI host.

Model servers often create huge writable layers when weights download during startup. Prefer a named volume, host mount or managed model store with explicit retention.

Logs, traces and evaluation artifacts

AI workloads can produce large request logs, token traces and evaluation outputs. Find the source:

sudo du -xhd1 /var/log 2>/dev/null | sort -h
journalctl --disk-usage
find /var/log -xdev -type f -size +500M -print

Use the service’s rotation mechanism. For systemd journals:

sudo journalctl --vacuum-size=1G

Truncating an active file may hide incident evidence. Preserve relevant logs first and check whether sensitive prompts or model outputs require shorter retention.

Inode exhaustion

If df -h shows space but df -i is full, millions of small cache, session or evaluation files may have exhausted inodes.

sudo find /var -xdev -type f -printf '%h\n' 2>/dev/null \
  | sort | uniq -c | sort -rn | head

Remove files through the owning application or a reviewed retention rule. Increasing disk size alone may not repair a poor small-file lifecycle.

Emergency recovery order

1. Stop nonessential jobs generating more data.
2. Preserve incident evidence and database integrity.
3. Free a small, known-safe amount from disposable caches or old logs.
4. Restore the affected service and confirm writes succeed.
5. Perform measured cleanup after stability returns.
6. Add capacity and retention alerts before resuming batch jobs.

If a pod is restarting, follow CrashLoopBackOff for AI workloads. If the failure is RAM rather than disk, use the AI workload out-of-memory guide.

Prevention for AI production

• Alert on bytes, inodes and growth rate.
• Separate OS, container and model-data storage where practical.
• Set retention for model revisions, logs and evaluation artifacts.
• Keep model downloads out of container writable layers.
• Budget temporary space for conversion and rolling deployments.
• Test cleanup without production volumes attached.
• Track usage by workload when several agents share a host.

Disk cleanup is an operational policy, not a one-time command. The AI Operations hub connects storage pressure with memory, containers, Kubernetes and deployment reliability.

Primary documentation

• Docker disk usage
• systemd journal controls