Permission Denied on AI Servers and Containers: Models, Volumes and Linux Users
Permission errors in AI systems often appear when a service reads model weights, writes a cache, mounts a Docker volume, accesses a GPU device, or drops privileges in production. The safe fix is to align ownership and required accessโnot to make everything writable or run every container as root.
Identify the exact actor and object
Inside the failing environment, inspect:
id
namei -l /models/my-model/config.json
stat /models/my-model/config.json
Ask four questions:
- Which UID/GID runs the process?
- Which path or device is denied?
- Is it inside the image, a bind mount, a named volume, or network storage?
- Does the operation need read, write, execute, or directory traversal access?
Directory execute permission controls traversal. A readable model file is still inaccessible when a parent directory cannot be traversed.
Model weights and caches
Mount immutable model weights read-only where possible. Give writable caches and generated artifacts separate locations with explicit ownership. Do not recursively chmod 777 a model directory that may contain proprietary weights, tokens, or configuration.
chown -R 10001:10001 /srv/ai-cache
chmod -R u=rwX,g=rX,o= /srv/ai-cache
Use the real service UID/GID and the narrowest permissions your workload needs.
Docker volume mismatches
A container user may not match ownership on a host bind mount. Inspect the container identity:
docker run --rm --entrypoint id your-image:tag
docker inspect your-container
Set an explicit non-root user in the image or deployment and pre-arrange volume ownership. Prefer read-only mounts for weights and configuration. --privileged is not a permissions repair; it dramatically expands access.
Named volumes and bind mounts have different host relationships. See Docker vs Kubernetes and the Linux foundation.
Kubernetes security contexts
Align runAsUser, runAsGroup, and storage policy with the image and volume. fsGroup can provide group access for supported volumes, but it should not substitute for understanding ownership. Also check read-only root filesystems, admission policy, SELinux/AppArmor, and device-plugin access for GPUs.
Deployment and secret permissions
Private keys, environment files, service-account tokens, and model-provider credentials should be readable only by the workload identity. If a deployment script cannot read a secret, fix its identity or secret mountโnot the secretโs permissions globally.
Environment variables and AI secrets covers credential handling. API authentication and MCP authentication cover application-level identity; filesystem permissions do not replace them.
Safe repair checklist
- Reproduce as the actual service user.
- Locate the first denied path or device.
- Inspect every parent directory and mount boundary.
- Align UID/GID and least-required access.
- Keep model weights and secrets read-only where possible.
- Restart through the normal deployment path.
- Test model loading, cache writes, and rollback.
Connect these controls to AI Security, deployment design to AI Deployment & Hosting, and production diagnostics to AI Operations.