Your Ollama model is taking 10+ seconds per token. Here is how to fix it.

Fix 1: Check if GPU is being used

The most common cause: CPU inference when GPU is available:

# Check GPU utilization
nvidia-smi  # NVIDIA
rocm-smi   # AMD

# Check Ollama's GPU status
ollama ps

If Ollama shows CPU-only, enable GPU:

# NVIDIA: ensure CUDA is installed
nvidia-smi  # Should show GPU info

# If CUDA is missing, install it
# Ubuntu/Debian
sudo apt install nvidia-cuda-toolkit

# macOS: Metal is automatic on Apple Silicon

Fix 2: Use a smaller model

Bigger models are slower. Match model size to your hardware:

Hardware	Recommended model	Speed
4GB RAM	Qwen3 1.7B	30-50 tok/s
8GB RAM	Qwen3 8B	15-25 tok/s
16GB RAM	DeepSeek R1 14B	10-20 tok/s
32GB RAM	Qwen 3.5 27B	5-15 tok/s
64GB RAM	Llama 4 Scout	3-10 tok/s

Fix 3: Use smaller quantization

Q4 is faster than Q8 with minimal quality loss:

# Slower but higher quality
ollama pull qwen3:27b          # Q4 default

# Faster with slight quality loss
ollama pull qwen3:27b-q3_k_m  # Q3 quantization

Fix 4: Reduce context window

Large context windows slow inference:

# Default context (often 8192 or larger)
ollama run qwen3:8b

# Reduced context (faster)
ollama run qwen3:8b --num-ctx 2048

For coding tasks, 4096-8192 context is usually sufficient.

Fix 5: Close other applications

Other apps compete for GPU/CPU resources:

# Check GPU usage
nvidia-smi

# Kill other GPU-heavy processes
# Chrome, Discord, video players use GPU

On macOS, Activity Monitor shows GPU usage. Close unnecessary apps.

Fix 6: Update Ollama

Older versions may have performance bugs:

# Check version
ollama --version

# Update
curl -fsSL https://ollama.com/install.sh | sh

# Or on macOS
brew upgrade ollama

Fix 7: Use GPU layers properly

For models that partially fit in VRAM:

# Set GPU layers (higher = more on GPU)
OLLAMA_NUM_GPU=35 ollama run qwen3:27b

# Check how many layers the model has
ollama show qwen3:27b --modelfile | grep PARAMETER

Still slow?

If none of these help, your hardware may not support fast inference. Consider:

1. Cloud API — Use OpenRouter for faster responses
2. Smaller model — Trade quality for speed
3. Hardware upgrade — More VRAM or faster GPU

FAQ

Why is my Ollama model so slow?

The most common cause is running on CPU instead of GPU. Check with nvidia-smi to see if Ollama is using your GPU. Other causes include: too-large model for your hardware, large context window, or other apps using GPU resources.

How many tokens per second should I expect?

Depends on your hardware: 4GB RAM gets 30-50 tok/s with 1.7B models, 16GB RAM gets 10-20 tok/s with 14B models, 32GB RAM gets 5-15 tok/s with 27B models. Apple Silicon is generally faster than NVIDIA for the same model size.

Does quantization affect speed?

Yes. Smaller quantizations (Q3, Q4) are faster than larger ones (Q5, Q8) because they use less memory and compute. For speed-critical tasks, use Q4_K_M as a good balance.

Related: Ollama Complete Guide · How Much VRAM for AI · Best GPU for AI 2026 · Ollama Out of Memory Fix · Best Local AI Models 2026