Running LLMs on a Raspberry Pi 5 is possible, but you need to set expectations right. This is not a speed demon. It’s a $80 computer that can run small language models at 2-8 tokens per second. That’s slow, but it’s real, local, private AI for under $150 total.
Here’s how to set it up, what models work, and what to expect.
What you need
| Component | Price | Notes |
|---|---|---|
| Raspberry Pi 5 (8GB) | $80 | 4GB works but limits model size |
| AI HAT (Hailo-8) | $70 | Optional, 26 TOPS NPU |
| microSD (64GB+) | $10 | Or NVMe HAT + SSD |
| Power supply (27W USB-C) | $12 | Official RPi PSU recommended |
| Case with fan | $15 | Thermal throttling is real |
| Total | $187 | Without AI HAT: $117 |
Setup
Install Raspberry Pi OS
# Flash Raspberry Pi OS (64-bit) to microSD
# Use Raspberry Pi Imager: https://www.raspberrypi.com/software/
# Boot and update
sudo apt update && sudo apt upgrade -y
# Verify hardware
cat /proc/cpuinfo | grep "Raspberry Pi 5"
free -h # Should show ~8GBInstall Ollama
# Install Ollama for ARM64
curl -fsSL https://ollama.com/install.sh | sh
# Verify installation
ollama --versionPull a model
# Start with a small model
ollama pull gemma4:2b
# Test it
ollama run gemma4:2b "What is the capital of France?"Recommended models
| Model | Params | RAM Needed | Speed | Quality |
|---|---|---|---|---|
| Qwen 3.6 0.5B | 0.5B | 1GB | 15-20 tok/s | Basic |
| Gemma 4 2B | 2B | 2GB | 5-8 tok/s | Good |
| Phi-3 Mini 3.8B | 3.8B | 3GB | 3-5 tok/s | Good |
| Llama 3.2 3B | 3B | 3GB | 4-6 tok/s | Good |
| Qwen 3.6 1.5B | 1.5B | 2GB | 8-12 tok/s | Decent |
| Llama 3.1 8B | 8B | 6GB | 1-2 tok/s | Best (slow) |
Recommendation: Gemma 4 2B or Qwen 3.6 1.5B for the best balance of speed and quality.
With AI HAT (Hailo-8 NPU)
The Raspberry Pi AI HAT adds a Hailo-8 NPU with 26 TOPS. For LLMs, the NPU helps with specific operations but doesn’t transform the experience.
# Install Hailo SDK
sudo apt install hailo-all
# Verify NPU
hailortcli fw-control identify
# Use with Ollama (experimental NPU support)
# Note: NPU acceleration for LLMs is still experimental
ollama run gemma4:2b "Hello"The AI HAT excels at vision tasks (YOLO, classification) more than LLMs. For LLMs specifically, the improvement is modest (maybe 20-30% faster).
Performance benchmarks
| Task | Model | Speed | Notes |
|---|---|---|---|
| Chat | Gemma 4 2B Q4 | 6 tok/s | Usable, not fast |
| Coding | Qwen 3.6 1.5B Q4 | 10 tok/s | Decent for simple tasks |
| Classification | Phi-3 Mini Q4 | 4 tok/s | Good for batch processing |
| Vision (NPU) | YOLO v8 Nano | 15 FPS | Real-time with AI HAT |
| Speech | Whisper Tiny | 1x realtime | Barely keeps up |
Power consumption
| State | Power Draw |
|---|---|
| Idle | 3W |
| CPU inference | 8-12W |
| CPU + NPU | 10-15W |
| Peak | 15W |
At 10W average, running 24/7 costs about $0.35/month (at $0.15/kWh).
Optimization tips
1. Use quantized models: Q4_K_M quantization is the sweet spot. Q2 is faster but quality drops significantly.
2. Enable ZRAM: Swap extension for better memory management:
sudo apt install zram-tools
sudo systemctl enable zramswap3. Overclock (with cooling):
# In /boot/firmware/config.txt
arm_freq=2800
gpu_freq=800
over_voltage_delta=500004. Use NVMe instead of microSD: NVMe is 10x faster for model loading.
5. Close unnecessary services:
sudo systemctl disable bluetooth
sudo systemctl disable cupsReal-world use cases
1. Offline chatbot: A simple Q&A bot that works without internet. Slow but private.
2. Text classification: Classify emails, documents, or messages. Batch processing works well.
3. Code completion: Simple code suggestions in a local IDE. Expect 2-5 second latency.
4. Voice assistant: Whisper for speech-to-text + small LLM for responses. Works at 1x realtime.
5. IoT brain: Process sensor data and make decisions locally. No cloud dependency.
When to use a Raspberry Pi for LLMs
- You need offline/private AI
- Budget is under $150
- Speed is not critical (batch processing)
- Power consumption matters
- You’re building an IoT device
When to use something else
- Faster inference: Jetson Orin Nano ($249, 10-15 tok/s)
- Better quality: Any GPU-equipped PC with Ollama
- Cloud API: GPT-5.6 Luna at $0.20/$1.20 per 1M tokens
- Desktop: Mac Mini M4 ($599, 20-30 tok/s)
My take
Running LLMs on a Raspberry Pi 5 is a fun experiment and a viable option for specific use cases (offline, private, low-power). But it’s slow. 6 tok/s with a 2B model is usable for batch processing but painful for interactive chat.
For serious local AI, spend the extra $100 and get a Jetson Orin Nano Super ($249). The 10-15 tok/s with 3B models is a much better experience. Or spend $599 on a Mac Mini M4 for 20-30 tok/s.
The Raspberry Pi is great for vision tasks with the AI HAT (15 FPS YOLO). For LLMs specifically, it’s the cheapest option but not the best experience.
FAQ
Can the Raspberry Pi 5 run 7B models?
Technically yes, but at 1-2 tok/s it’s painfully slow. Stick to 1.5B-3B models for usable performance.
Do I need the AI HAT for LLMs?
No. The AI HAT helps with vision tasks but has limited LLM acceleration. For LLMs specifically, the improvement is modest (20-30% faster).
How much RAM do I need?
8GB recommended. 4GB works for 1.5B models but limits you. 8GB gives headroom for 3B models.
Can I run Ollama on a Raspberry Pi 4?
Yes, but it’s much slower. The Pi 4’s CPU is 2-3x slower than the Pi 5. Expect 1-3 tok/s with 2B models.
Is it worth it for production?
For low-volume, always-on, private inference: yes. For anything with users waiting for responses: no. The latency is too high for interactive applications.