Jul 24, 2026 · 6 min read

How to Run GLM-5.2 Locally: Hardware, Quantization, and Setup Guide (2026)

GLM-5.2 is Z.ai’s 744B parameter MoE model with 40B active parameters, a 1M token context window, and MIT open weights. Running it locally means no API costs, no rate limits, and full data privacy. But at 744B parameters, this is not a casual install.

This guide covers what hardware you actually need, which quantization to pick, and how to get GLM-5.2 running on your own machine. If you just want API access, see our GLM-5.2 complete guide.

Hardware reality check

GLM-5.2 at full precision needs ~1.5TB of memory. Nobody is doing that at home. The practical path is quantization, and even then you need serious hardware.

Memory requirements by quantization

Quantization	Disk size	RAM/VRAM needed	Hardware example
BF16 (full)	~1.5 TB	~1.5 TB	8x H100 80GB
FP8	~750 GB	~780 GB	4x H100 80GB
Q4_K_M	~400 GB	~420 GB	4x RTX 4090 + 256GB RAM
IQ2_XXS (Unsloth dynamic)	~239 GB	~260 GB	Mac Studio 256GB or 4x RTX 3090
IQ1_S (minimum)	~190 GB	~210 GB	192GB RAM + CPU offload

Can I actually run this?

Yes, if you have:

Mac Studio M2/M3/M4 Ultra with 192GB+ unified memory (IQ2_XXS, ~10-15 tok/s)
4x RTX 3090/4090 with 256GB system RAM (Q4_K_M with split across GPUs)
Cloud GPU rental: RunPod 4x A100 80GB (~$8/hr)

No, if you have:

A single consumer GPU (even a 4090 with 24GB)
Less than 128GB total system RAM
A laptop (any laptop)

For consumer hardware, consider GLM-5.1 (same architecture, smaller), Qwen 3.6-27B (fits in 16GB), or Gemma 4 12B (runs on any modern machine). Our Best AI Models for Coding Locally guide has a full ranking of options by hardware tier.

Setup with Ollama (easiest path)

Ollama has community-uploaded GLM-5.2 GGUF models:

# Pull the model (will download ~240GB for IQ2_XXS)
ollama pull frob/glm-5.2

# Or import a specific quantization
# First download from HuggingFace:
huggingface-cli download unsloth/GLM-5.2-GGUF \
    --include "*IQ2_XXS*" --local-dir ./models/

# Create Modelfile
echo 'FROM ./models/GLM-5.2-UD-IQ2_XXS.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 32768' > Modelfile

# Import into Ollama
ollama create glm-5.2 -f Modelfile

# Run
ollama run glm-5.2 "Refactor this function to use async/await"

Ollama will automatically use GPU layers where available and fall back to CPU for the rest. On a 192GB+ Mac, it loads the entire model into unified memory.

Context window note: The full 1M context window requires proportionally more memory. Start with num_ctx 32768 and increase if your hardware handles it. At 128K context, expect roughly 50% more memory usage than the base model size.

Setup with llama.cpp (most control)

For multi-GPU setups or fine-grained control over layer placement:

# Build llama.cpp with CUDA support
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j \
    --clean-first --target llama-cli

# Download Unsloth dynamic GGUF (best quality-per-byte)
pip install huggingface_hub hf_transfer
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download \
    unsloth/GLM-5.2-GGUF \
    --include "*IQ2_XXS*" \
    --local-dir ./models/glm-5.2/

# Run with MoE layers on CPU, attention on GPU
./llama.cpp/build/bin/llama-cli \
    --model ./models/glm-5.2/GLM-5.2-UD-IQ2_XXS.gguf \
    --threads 32 \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    -no-cnv \
    --prompt "Your prompt here"

The -ot ".ffn_.*_exps.=CPU" flag offloads MoE expert layers to CPU while keeping attention on GPU. This is the key trick for making large MoE models fit on fewer GPUs.

Multi-GPU split (4x RTX 4090)

# Split layers across 4 GPUs
./llama.cpp/build/bin/llama-cli \
    --model ./models/glm-5.2/GLM-5.2-UD-Q4_K_M.gguf \
    --threads 32 \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    --tensor-split 25,25,25,25 \
    --temp 0.7 \
    -no-cnv \
    --prompt "Your prompt here"

Setup with vLLM (production serving)

For serving GLM-5.2 as an API to multiple users:

pip install vllm

# Serve on 4x A100 80GB with tensor parallelism
python -m vllm.entrypoints.openai.api_server \
    --model zai-org/GLM-5.2 \
    --tensor-parallel-size 4 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.95 \
    --port 8000

This gives you an OpenAI-compatible API endpoint running locally. Use this for team deployments or when you need concurrent requests.

Performance expectations

Based on community reports:

Setup	Quantization	Speed	Context
Mac Studio M2 Ultra 192GB	IQ2_XXS	~8-12 tok/s	32K
Mac Studio M4 Ultra 256GB	IQ2_XXS	~12-18 tok/s	64K
4x RTX 4090 (96GB VRAM)	Q4_K_M	~20-30 tok/s	32K
4x A100 80GB	FP8	~60-80 tok/s	128K
8x H100	BF16	~150+ tok/s	1M

These are estimates based on community benchmarks. Your actual speed depends on context length, batch size, and system configuration.

Thinking modes

GLM-5.2 has two thinking modes that affect quality and speed:

High — balanced reasoning, faster. Good for code review, documentation, straightforward tasks.
Max — deep reasoning, slower. Use for complex refactors, debugging, architectural decisions.

In llama.cpp, trigger Max mode with a system prompt:

You are GLM-5.2 in Max thinking mode. Think step by step through complex problems before providing your answer.

When to self-host vs use the API

Factor	Self-host	API
Cost at scale	Cheaper above ~$500/month API usage	Cheaper below that
Data privacy	Full control, nothing leaves your network	Data goes to Z.ai servers
Speed	Depends on hardware	~100 tok/s
Context window	Limited by your RAM	Full 1M tokens
Setup effort	High (hours to days)	Minutes
Maintenance	You handle updates	Managed

Self-host if: You process sensitive code, have existing GPU infrastructure, or your API bill exceeds $500/month.

Use the API if: You need the full 1M context, want maximum speed, or do not have multi-GPU hardware.

Cheaper alternatives that run on consumer hardware

If GLM-5.2’s hardware requirements are too steep:

Model	Size	RAM needed	Speed on M4 Max	Quality vs GLM-5.2
Qwen 3.6-27B	27B dense	16GB	~50 tok/s	~75% (coding)
DeepSeek V4 Flash	284B MoE, 13B active	32GB (Q4)	~25 tok/s	~70% (general)
Gemma 4 12B	12B dense	8GB	~80 tok/s	~60% (multimodal)
Poolside Laguna XS 2.1	33B MoE, 3B active	16GB	~60 tok/s	~55% (coding)

For most developers on consumer hardware, Qwen 3.6-27B offers the best balance of quality and accessibility.

FAQ

How much does it cost to rent hardware for GLM-5.2?

A 4x A100 80GB setup on RunPod costs approximately $8/hour. For occasional use, this is cheaper than self-hosting. For daily use (8+ hours), buying hardware becomes more economical within 6-12 months.

Can I run GLM-5.2 on Apple Silicon?

Yes, but only on high-end configurations. Mac Studio M2/M3/M4 Ultra with 192GB+ unified memory can run the IQ2_XXS quantization at ~8-18 tok/s. MacBook Pro (even with 96GB) is too tight for comfortable use.

What is the best quantization for GLM-5.2?

IQ2_XXS from Unsloth is the sweet spot for most self-hosting scenarios. It fits in ~260GB of memory while retaining most of the model’s quality. If you have more hardware, Q4_K_M offers noticeably better output quality at ~420GB.

Does the 1M context window work locally?

Only with enough memory. At 1M context on Q4_K_M, you need roughly 800GB+ of memory. Most local setups are limited to 32K-128K context. For full 1M context, use the API or a cloud GPU cluster.

Should I use GLM-5.2 or DeepSeek V4 Pro locally?

Both are similar in size (744B vs 1.6T total, but 40B vs 49B active). DeepSeek V4 Pro scores higher on coding benchmarks (80.6% vs ~72% SWE-bench). GLM-5.2 has the longer context window (1M vs 128K). Choose based on whether you need context length or coding accuracy.