Ollama is running on CPU when you have a GPU. Here is how to fix it.

Fix 1: Verify GPU is detected

First, confirm your GPU is visible to the system:

# NVIDIA
nvidia-smi

# AMD
rocm-smi

# Apple Silicon (no command needed - Metal is automatic)
system_profiler SPDisplaysDataType

If nvidia-smi fails, your NVIDIA drivers are not installed correctly.

Fix 2: Install NVIDIA drivers (Linux)

# Ubuntu/Debian
sudo apt update
sudo apt install nvidia-driver-535
sudo reboot

# Verify after reboot
nvidia-smi

For CUDA toolkit:

# Install CUDA toolkit
sudo apt install nvidia-cuda-toolkit

# Verify CUDA
nvcc --version

Fix 3: Fix Docker GPU passthrough

If Ollama runs in Docker:

# Check if Docker can see GPU
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

# If this fails, install nvidia-container-toolkit
sudo apt install nvidia-container-toolkit
sudo systemctl restart docker

Docker compose setup:

services:
  ollama:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    runtime: nvidia

Fix 4: Apple Silicon (macOS)

Metal support is automatic on Apple Silicon. If it is not working:

# Verify Apple Silicon
uname -m  # Should show arm64

# Reinstall Ollama
brew uninstall ollama
brew install ollama

# Start Ollama
ollama serve

Intel Macs do not support GPU acceleration in Ollama.

Fix 5: AMD GPU (Linux)

AMD support requires ROCm:

# Install ROCm (Ubuntu)
sudo apt install rocm-dev

# Set environment
export ROCm_VISIBLE_DEVICES=0

# Restart Ollama
ollama serve

ROCm support is experimental. Check Ollama’s AMD documentation for your specific GPU.

Fix 6: Environment variables

Sometimes Ollama needs explicit GPU configuration:

# Force CUDA usage
export CUDA_VISIBLE_DEVICES=0
ollama serve

# For multi-GPU setups
export CUDA_VISIBLE_DEVICES=0,1
ollama serve

Fix 7: Reinstall Ollama

If nothing works, a clean install often fixes GPU detection:

# Stop Ollama
pkill ollama

# Remove installation
sudo rm -rf /usr/local/bin/ollama
sudo rm -rf ~/.ollama

# Reinstall
curl -fsSL https://ollama.com/install.sh | sh

# Start and verify
ollama serve

Verify GPU is working

# Start Ollama and run a model
ollama run llama3.2

# Check GPU usage in another terminal
nvidia-smi

# You should see ollama using GPU memory

Still not working?

1. Check Ollama logs — OLLAMA_DEBUG=1 ollama serve
2. Try a different model — Some models have better GPU support
3. Update drivers — Outdated drivers cause detection issues
4. Check hardware compatibility — Not all GPUs are supported

FAQ

How do I know if Ollama is using my GPU?

Run nvidia-smi in a separate terminal while Ollama is running a model. You should see Ollama listed as a process using GPU memory. If it is not there, Ollama is running on CPU.

Does Apple Silicon work with Ollama?

Yes. Apple Silicon Macs automatically use Metal for GPU acceleration. No additional setup is needed. Intel Macs do not support GPU acceleration in Ollama.

Why does Ollama work on CPU but not GPU?

Common causes: CUDA not installed (NVIDIA), wrong drivers, Docker without GPU passthrough, or the model is too large for your VRAM. Check each of these in order.

Related: Ollama Complete Guide · How Much VRAM for AI · Best GPU for AI 2026 · Ollama Out of Memory Fix · Ollama Slow Inference Fix