Linux knowledge becomes operationally important when an AI application moves beyond a hosted API. Local models, GPU servers, inference containers and background workers all depend on the operating system behaving predictably.

This guide is not a Linux command encyclopedia. It covers the decisions and failure points that matter when you run AI workloads. Start with local AI if you are still choosing a runtime, or the deployment and hosting guide if you are deciding where the workload belongs.

Start with the workload

Before configuring a server, identify what it must run:

• A local development model can tolerate manual restarts and limited concurrency.
• A shared inference endpoint needs service supervision, health checks and access control.
• A RAG worker needs durable storage and enough temporary disk for document processing.
• A GPU service needs compatible drivers, runtime libraries and capacity monitoring.
• An agent worker needs narrow permissions and a controlled workspace.

Do not install every AI tool directly on the host. Separate the base system, container runtime, model data and application state so each can be upgraded or restored independently.

Verify CPU, memory, disk and GPU

lscpu
free -h
df -h
lsblk
nvidia-smi

nvidia-smi confirms that the kernel driver can see an NVIDIA GPU. It does not prove that a container or Python environment has the correct CUDA libraries. Validate each layer separately: host driver, container runtime, framework and model.

For hosted hardware, compare capacity and operational control in the cloud GPU provider guide. Size for model weights, runtime overhead, KV cache and concurrent requests—not only the model download size.

Treat model storage as infrastructure

Model weights can consume hundreds of gigabytes. Keep them outside application release directories:

sudo mkdir -p /srv/models /srv/ai-data
sudo chown -R ai-runtime:ai-runtime /srv/models /srv/ai-data
df -h /srv/models
du -sh /srv/models/*

Use separate volumes when model caches would otherwise fill the root filesystem. Monitor inode use as well as bytes. Define retention for superseded weights, temporary uploads and generated artefacts.

Back up application state and configuration. Re-downloading public model weights is often cheaper than backing them up, but private adapters and evaluation data are not replaceable.

Run inference as a service

A process started in an SSH session disappears when the session or server ends. Use a service manager for host processes:

[Unit]
Description=AI inference service
After=network-online.target

[Service]
User=ai-runtime
WorkingDirectory=/opt/inference
EnvironmentFile=/etc/ai-inference.env
ExecStart=/opt/inference/bin/start
Restart=on-failure
RestartSec=5
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Store secrets in a protected environment file or secret manager, not in the unit. Run under a dedicated unprivileged account. Add readiness checks before sending traffic to a newly restarted model.

sudo systemctl daemon-reload
sudo systemctl enable --now ai-inference
systemctl status ai-inference
journalctl -u ai-inference -f

Containers and GPU access

Containers make dependencies reproducible, but they do not create GPU memory or fix incompatible drivers. Confirm GPU access inside the container before debugging the model:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Pin images, set memory and CPU limits deliberately, mount model data read-only where possible, and keep generated data on a durable volume. The Ollama Docker setup covers a practical local-model stack.

Kubernetes becomes useful when you genuinely need scheduling, isolation and coordinated rollout. It does not remove the need to understand node memory, GPU allocation, model warm-up or persistent storage. Use the AI operations hub for the production layer.

Permissions for agents and model services

Avoid running an agent, notebook or inference server as root. Separate accounts by responsibility:

• deployment automation can replace releases but not read unrelated secrets;
• inference can read approved model files but not modify system configuration;
• document workers can write to their job directory but not other tenants’ data;
• human administrators use personal SSH keys rather than shared passwords.

Check the effective identity and permissions instead of applying chmod 777:

id
namei -l /srv/models/example/model.gguf
sudo -u ai-runtime test -r /srv/models/example/model.gguf

For the access layer, continue with SSH for AI servers and AI security.

Logs and monitoring

Collect four different signals:

1. System health: disk, RAM, load, temperatures and GPU utilisation.
2. Runtime health: restarts, queue depth and request latency.
3. Model behaviour: first-token latency, tokens per second and errors.
4. Product health: task success, fallback rate and cost.

Useful first checks include:

journalctl -u ai-inference --since "30 minutes ago"
dmesg -T | tail -100
free -h
df -h
nvidia-smi
docker stats

An out-of-memory kill may appear in kernel logs while the application reports only a lost connection. Diagnose the full path before changing retry settings.

Safe update workflow

For production hosts:

1. Snapshot configuration and state.
2. Validate disk capacity and driver compatibility.
3. Deploy a pinned release beside the current one.
4. Warm the model and run a representative request.
5. Shift traffic only after readiness passes.
6. Keep a rollback path that does not require rebuilding the server.

Linux should make the AI workload observable and recoverable. If operating the host consumes more effort than the application warrants, choose a managed platform instead of treating self-hosting as a requirement.