Bonsai 2 27B is interesting because compact weights can make a large model practical on more local hardware. The first deployment decision, however, is the runtime, not the download size. This model uses specialized ternary packing, and treating its GGUF file like an ordinary quantized model can produce an incompatible or incorrect setup.
This is a documentation-based deployment guide. We have not benchmarked Bonsai 2 ourselves. Start with the official demo, establish correct output, then measure your workload before replacing a working local model or a cloud service.
What model are you downloading?
Prism ML’s announcement identifies Bonsai 2 27B as derived from Qwen3.8-27B. It is not the earlier Gemma-based Bonsai family or an official Qwen release. The official model card lists Apache 2.0 licensing and a 262,144-token context window. Text and image use require the appropriate components; a long advertised context is not a promise that your laptop can serve it at full length.
The current release offers specialized GGUF weights and an MLX 2-bit distribution. Model lineage, license and runtime support are separate checks. A permissive weight license does not establish that a deployment meets your organization’s privacy or operational requirements.
Formats and runtime: use the model-specific path
| Choice | Practical implication |
|---|---|
| PTQ1_0 GGUF | 5.95 GB language weights, 1.75 bits/weight; specialized compatible runtime required |
| PQ2_0 GGUF | 7.21 GB language weights, 2.13 bits/weight; current demo default |
| MLX 2-bit | Separate Apple-silicon path; follow its model-specific instructions |
| Vision projector | Additional component and storage for image input, not included merely by quoting language-weight size |
The official demo README says Bonsai 2’s GGUF bands currently need the Prism llama.cpp fork. Its installer selects that runtime. Upstream support for older Bonsai formats is not evidence that stock llama.cpp supports Bonsai 2. Hugging Face’s generic library launch snippets are not a substitute for the model-specific instructions.
For background on serving tradeoffs, use our vLLM, Ollama and llama.cpp comparison. Do not assume an Ollama import or an arbitrary GGUF application will understand a newly introduced packing type.
Storage is not working memory
Budget three separate resources before installation:
- Disk for the weight file, optional vision projector, runtime and auxiliary applications.
- Working memory for loaded weights, runtime buffers and any other processes.
- Context memory and compute for your actual prompt length and concurrent requests.
A small weight file cannot tell you all three. The demo describes approximately 7.8 GB for its default PQ2_0 weights plus projector, with optional applications adding further downloads. Check the current artifacts rather than promising a universal installation footprint.
On unified-memory machines, the operating system and other applications share that budget. On discrete-GPU machines, system RAM and VRAM are different constraints. Offloading may keep a model usable but change latency substantially. Begin with one request and a short context, then increase one variable at a time.
Our local-model hardware guide provides broader budgeting considerations. If local hardware cannot meet your workload, compare cloud GPU options rather than forcing an unstable installation into production.
Install through the official demo
For macOS or Linux, the project’s documented starting path is:
git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./setup.sh
./scripts/start_llama_server.shReview setup.sh before running it. It installs/downloads components; it is not simply a model-file fetch. Use a non-production development environment without production credentials. Do not run an unfamiliar installer with elevated privileges merely because a guide includes a command.
Open the documented local interface at http://localhost:8080 after the server starts. The README also provides a terminal-only smoke path:
./scripts/run_llama.sh -p "What is the capital of France?"The installer supports skipping optional Open WebUI and code-interpreter components through BONSAI_OPENWEBUI=0 and BONSAI_CODE_INTERPRETER=0. That can reduce installation scope when all you need is basic inference. Recheck the current README before changing options; a project installer can evolve independently from the model weights.
Check hardware support before choosing a backend
The demo documents Mac/Metal, Linux and Windows GPU paths, and CPU operation. The Bonsai 2 model card specifically lists CUDA, Metal and CPU; do not assume every demo backend has identical model-specific performance or support. Follow the branch for your operating system and hardware rather than transplanting CUDA commands onto Apple silicon. MLX is an Apple-silicon route; GGUF serving follows the compatible fork and its available backend.
Record the runtime revision, backend, weight artifact and configuration in your test notes. If the project supplies a prebuilt binary, confirm its source and intended platform. A server that starts successfully is only the first check: output correctness and memory behavior still need validation.
For local serving architecture, see AI Deployment & Hosting. For lifecycle ownership, logs and rollback, use AI Operations.
Validate before integrating
Use a small acceptance set drawn from your application, not only a chat greeting. Include an answerable question, a deliberately unanswerable question, a structured-output task, and a representative longer prompt. Add images only if your integration actually needs vision and the projector is configured.
Keep the same prompts when comparing against your previous model. Inspect schema validity, unsupported claims, latency, peak memory and failure behavior. Our AI Testing & Evaluation foundation explains how to keep a repeatable evaluation set. The JSON parsing foundation helps separate malformed output from a transport or runtime problem.
Then exercise cancellation and overload. If a request fails halfway through, your application must not treat a partial stream as a completed answer. Keep a known-good serving configuration so rollback means restoring a tested artifact, not downloading a different model during an incident.
How to read Prism’s benchmark claims
Prism reports retention of much of its reference model’s quality across its own evaluation suite. That is a project benchmark, not our independent reproduction and not a universal claim of lossless quantization. Different task sets, reasoning settings and runtime builds can change the comparison.
Avoid turning an aggregate retention percentage into a guarantee for legal analysis, coding or every image task. Likewise, do not transfer a speed figure from a different hardware configuration or superseded runtime build into your buying decision. Measure your own end-to-end request path.
My take
The best first use is a controlled local experiment where compact weights matter and you can tolerate a model-specific runtime. It is a weaker fit when your deployment requires only stock upstream packages or an already-validated enterprise serving stack. Bonsai 2 can be worth evaluating without being the default recommendation for every local AI project.
Keep this deployment guide separate from choosing local models by task: the latter is a selection decision, while this page owns installation compatibility and practical deployment limits.
FAQ
Can I use ordinary stock llama.cpp?
Do not assume so. The current Bonsai 2 demo specifies Prism’s fork for its specialized GGUF bands. Older Bonsai upstream support is a different compatibility claim.
Does the download size equal required RAM?
No. Account for runtime buffers, context, concurrency and optional vision components, plus the operating system. Disk footprint and peak working memory are different measurements.
Is the full 262,144-token context practical on every laptop?
No. An advertised context limit says what the model supports, not what your memory budget or acceptable latency permits.
Have we independently reproduced the quality benchmarks?
No. The figures belong to Prism’s evaluation. Use a workload-specific acceptance set before treating the model as a replacement.
Should I expose the local demo publicly?
Not as an unreviewed default. Review authentication, network binding and resource limits first. A development demo is not automatically a hardened production service.