How to Run Tencent Hy3 Locally: Hardware, Ollama, and Quantization Guide

Tencent Hy3 is open weight (Apache 2.0) and available on Hugging Face. That means you can run it on your own hardware with zero API costs and complete data privacy. The catch: a 295B parameter MoE model needs serious hardware. This guide walks through your options.

Understanding Hy3’s Memory Requirements

Hy3 is a Mixture of Experts model. It has 295 billion total parameters but only 21 billion are active per token (8 of 192 experts fire at once). This architecture creates a specific memory pattern:

Storage requirement: All 295B parameters must be in memory (or accessible) even though only 21B activate per token. At FP16, that is approximately 590GB of memory. At BF16, same story.

Compute requirement: Only 21B parameters are computed per token. This means inference per token is fast (comparable to a 21B dense model) once the full model is loaded.

The tradeoff: You need lots of memory to hold the model, but you do not need lots of compute to run it. This favors high-memory, moderate-compute setups.

Hardware Options

Option 1: Multi-GPU Workstation (Full Precision)

For FP16/BF16 inference without quantization:

GPUs	VRAM Total	Fits?
4x NVIDIA A100 80GB	320GB	No (need ~590GB)
8x NVIDIA A100 80GB	640GB	Yes (tight)
4x NVIDIA H100 80GB	320GB	No
8x NVIDIA H100 80GB	640GB	Yes
4x NVIDIA A6000 48GB	192GB	No

Full precision requires at minimum 8x 80GB GPUs. This is server-grade hardware costing $100K+. Not practical for individual developers.

Option 2: Quantized (Recommended for Most Users)

Quantization reduces precision to shrink the model. For MoE models, quantization works well because the routing logic remains precise while expert weights tolerate lower precision.

Quantization	Model Size	Min VRAM	Quality Impact
FP16 (none)	~590GB	640GB	None
INT8	~295GB	320GB	Minimal
INT4 (GPTQ)	~148GB	192GB	Small
Q4_K_M (GGUF)	~150GB	160GB+	Small-moderate
Q3_K_M (GGUF)	~112GB	128GB	Moderate
Q2_K (GGUF)	~75GB	96GB	Significant

Recommended sweet spot: INT4/Q4_K_M. This keeps quality high for coding tasks while fitting on hardware that costs $10-20K rather than $100K+.

With Q4_K_M quantization and a setup like:

• 2x NVIDIA RTX 4090 (48GB total) + system RAM offloading: Possible but slow
• 4x NVIDIA RTX 4090 (96GB total): Workable for the lower quants
• 2x NVIDIA A6000 (96GB total): Good fit for Q3/Q4 quants
• 4x NVIDIA A6000 (192GB total): Comfortable for Q4 and above

For detailed VRAM guidance, see our how much VRAM for AI models guide.

Option 3: CPU + RAM Offloading

If you have a machine with 256GB+ system RAM but limited GPU VRAM, you can offload most layers to CPU and keep only some on GPU. This works but is slow:

• Inference speed drops to 1-5 tokens/second (vs 20-50 on full GPU)
• Acceptable for batch processing, painful for interactive use
• Requires fast memory (DDR5 preferred)

A workstation with 512GB RAM and a single GPU for the hot layers can run Hy3 with Q4 quantization. You will not get real-time completions, but for batch code generation or review, it works.

Setting Up with Ollama

Ollama is the simplest path to running Hy3 locally. Once the model is available in Ollama’s registry:

# Pull the quantized model
ollama pull tencent-hy3:q4_k_m

# Run interactively
ollama run tencent-hy3:q4_k_m

# Serve via API
ollama serve

If Hy3 is not yet in Ollama’s registry, you can import GGUF files manually:

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./tencent-hy3-q4_k_m.gguf
PARAMETER num_ctx 65536
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

# Create the model in Ollama
ollama create tencent-hy3 -f Modelfile

# Run it
ollama run tencent-hy3

Ollama Configuration for MoE

MoE models benefit from specific Ollama settings:

# Set number of parallel experts to load
# (affects memory but not compute per token)
OLLAMA_NUM_PARALLEL=4 ollama serve

For Hy3 specifically, you want to ensure the full expert parameters fit in memory even though only 8 experts activate. If memory is tight, Ollama will offload experts to CPU, which hurts performance on the tokens that route to offloaded experts.

Setting Up with vLLM

For higher throughput serving (multiple users, batched requests), vLLM handles MoE models efficiently:

# Install vLLM with MoE support
pip install vllm

# Launch server
python -m vllm.entrypoints.openai.api_server \
  --model tencent/Hy3 \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --trust-remote-code

vLLM’s tensor parallelism splits the model across GPUs efficiently. For MoE models, it handles expert routing across GPU boundaries.

Key vLLM settings for Hy3:

• --tensor-parallel-size: Match to your GPU count
• --max-model-len: Set based on your available memory (256K max, but reduce if memory is tight)
• --quantization gptq: Add if using a GPTQ-quantized variant

Setting Up with llama.cpp

For GGUF format on CPU/GPU hybrid setups:

# Clone and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j LLAMA_CUDA=1

# Run with partial GPU offloading
./main -m tencent-hy3-q4_k_m.gguf \
  -ngl 40 \
  -c 65536 \
  -t 16 \
  --moe-experts 8

The -ngl flag controls how many layers go to GPU. More layers on GPU = faster inference but more VRAM. Experiment to find the sweet spot for your hardware.

Performance Expectations

Realistic inference speeds based on hardware:

Setup	Quantization	Tokens/sec	Use Case
8x A100 80GB	FP16	40-60	Production serving
4x A6000 48GB	Q4_K_M	15-25	Development workstation
2x RTX 4090	Q4_K_M	8-15	Personal (workable)
1x RTX 4090 + CPU offload	Q3_K_M	3-6	Batch only
512GB RAM + 1 GPU	Q4_K_M	2-4	Background processing

For interactive coding assistance (tab completions, inline edits), you want at least 15 tokens/second. Below that, the latency becomes disruptive to your workflow.

For code review, batch generation, or background tasks, even 3-4 tokens/second is acceptable since you are not waiting in real-time.

Quantization Quality for Coding

Not all quantization levels are equal for coding tasks. Code is more sensitive to precision because:

• Small changes in token probabilities can produce syntactically invalid code
• Variable names and API methods need exact spelling
• Indentation and formatting matter for Python and YAML

Based on community testing of similar MoE models:

• Q4_K_M: Almost no quality degradation for coding. This is the recommended default.
• Q3_K_M: Slight quality loss. Occasional syntax errors in complex code. Still very usable.
• Q2_K: Noticeable degradation. More frequent errors in long code blocks. Use only if hardware forces it.

For coding specifically, do not go below Q3 unless you are just experimenting.

Connecting to Your IDE

Once Hy3 is running locally via Ollama or vLLM, connect it to your IDE:

VS Code (Continue extension):

{
  "models": [{
    "title": "Tencent Hy3 Local",
    "provider": "ollama",
    "model": "tencent-hy3:q4_k_m"
  }]
}

Cursor (custom model): Configure as an OpenAI-compatible endpoint pointing to your local server (localhost:11434 for Ollama, localhost:8000 for vLLM).

Terminal-based tools: Any tool supporting OpenAI-compatible APIs can point to your local inference server.

Cost Comparison: Local vs API

Is running locally worth it? Depends on usage:

Usage Level	API Cost (OpenRouter)	Local Amortized	Winner
Light (1M tokens/month)	$0.14	$500+/month (hardware)	API
Medium (50M tokens/month)	$7	$500+/month	API
Heavy (500M tokens/month)	$70	$500+/month	Depends
Very heavy (5B tokens/month)	$700	$500+/month	Local

At $0.14/M tokens, Hy3 via API is so cheap that local deployment only makes financial sense at extremely high volumes or when data privacy requirements make API usage impossible.

The privacy argument is stronger than the cost argument for most teams. If your code cannot leave your network, local is the only option regardless of cost.

Troubleshooting Common Issues

Out of memory during loading: Reduce --max-model-len or use a more aggressive quantization. The model needs memory for both weights and KV cache (context).

Slow first token: MoE models sometimes have slower time-to-first-token because the routing decision adds a step. This improves with GPU warmup.

Inconsistent speed: Some tokens route to experts that are offloaded to CPU. If speed varies wildly, more GPU memory (to keep all experts on GPU) solves it.

Ollama crashes on start: Check that Ollama’s allocated memory matches your available VRAM. Update to the latest Ollama version for best MoE support.

Further reading

If you are still deciding whether Hy3 is the right local model, compare it against other options in best AI models for coding locally 2026 and check the VRAM requirements guide for different model sizes. For the full Hy3 story, start with the Tencent Hy3 complete guide and see how it stacks up in Hy3 vs DeepSeek V4 and Hy3 vs Qwen 3.7. For Ollama setup patterns that apply to any model, see the Ollama complete guide.

FAQ

What is the minimum hardware to run Hy3 locally?

At Q4_K_M quantization with partial CPU offloading: 2x RTX 4090 (48GB VRAM) plus 128GB system RAM gives a functional but slow setup (8-15 tokens/sec). For comfortable interactive use, aim for 4x A6000 or equivalent (192GB+ VRAM).

Is the API cheaper than running locally?

For most usage volumes, yes. At $0.14/M tokens, you need to generate billions of tokens monthly before local hardware pays for itself. Run locally for privacy, not cost savings.

Does quantization hurt coding quality?

Q4_K_M shows almost no degradation for coding tasks. Q3_K_M has slight quality loss. Avoid going below Q3 for code generation. See the quantization quality section above for details.

Can I run Hy3 on Apple Silicon?

With unified memory MacBooks (M2 Ultra with 192GB, M3 Ultra with 192GB+), you can run lower quantizations. Performance will be below dedicated NVIDIA GPUs but functional for light use. Expect 3-8 tokens/second depending on quantization and context length.

How does running Hy3 locally compare to running DeepSeek V4 locally?

DeepSeek V4 is a dense model requiring more compute per token but similar total memory. Hy3’s MoE architecture gives faster per-token inference once loaded. For local deployment comparisons, see our DeepSeek V4 local guide.