Running an AI agent fully locally, no cloud API, no per-token bill, no data leaving your machine, has usually meant a real quality tradeoff. Small local models could follow instructions but struggled with reliable tool calling, which is the backbone of any real agent. Liquid AI’s LFM2.5-2.6B is built specifically to close that gap, and Liquid AI documents Hermes Agent by name as one of the agent harnesses it’s trained to work inside.
This guide walks through the exact setup: serving LFM2.5-2.6B locally, then pointing Hermes Agent at it, based on Liquid AI’s own deployment documentation.
Why this pairing makes sense
Hermes Agent is model-agnostic by design. It talks to any OpenAI-compatible /v1 endpoint, whether that’s Nous Portal, OpenRouter, OpenAI, or a self-hosted server, and you switch providers with hermes model, no code changes required. That flexibility is exactly what makes a fully local setup possible: Hermes doesn’t need LFM2.5 to be a first-class supported provider, it just needs a local server exposing the standard chat completions API with tool calling enabled.
LFM2.5-2.6B is the model half of the equation. It’s a 2.6B dense model trained specifically for agentic workloads, with:
- 128K token context window, enough for tool traces and multi-step workflows without constant truncation
- Native tool calling, trained into the model rather than bolted on via prompting tricks
- On-device footprint, small enough to run on a laptop or phone-class hardware
- Roughly 5.9 GB VRAM at FP16, and much less with quantization
Liquid AI’s own documentation states plainly that LFM2.5-2.6B “runs on edge devices and it is trained to work reliably inside agent harnesses like Hermes Agent, OpenClaw, and Pi.” That’s a direct compatibility statement from the model vendor, not an inference on our part, and Liquid AI publishes a complete walkthrough for the Hermes Agent pairing specifically.
Hardware requirements
| Quantization | Size | Notes |
|---|---|---|
| Q4_K_M (GGUF) | 1.67 GB | Best balance of size and quality, recommended starting point |
| Q6_K (GGUF) | 2.22 GB | Better quality, still light |
| Q8_0 (GGUF) | 2.87 GB | Near-lossless, safe choice for tool-heavy agentic work |
| BF16 (full precision) | 5.4 GB | Maximum fidelity, useful for benchmarking |
Any of these fit comfortably on a laptop GPU or even CPU inference for the smaller quants. If you hit context-overflow errors during long agent sessions, the fix is either raising the served context window or trimming the agent’s history, LFM2.5-2.6B supports up to 128K tokens, but you don’t have to serve the full window if you’re memory-constrained. A 32K window is plenty for a single agent task.
Step 1: Serve LFM2.5-2.6B locally
Pick one inference backend. All of them expose an OpenAI-compatible endpoint, which is what makes this whole setup work with an unmodified Hermes Agent install.
Option A: llama.cpp (recommended for most setups)
Install:
brew install llama.cpp # macOS
winget install llama.cpp # WindowsRun (the -hf flag auto-downloads the GGUF):
llama-server -hf LiquidAI/LFM2.5-2.6B-GGUF:Q4_K_M \
--jinja \
--port 8080 \
-c 131072 \
-fa on \
-ngl 99 \
--temp 0.1 \
--top-k 50 \
--repeat-penalty 1.1The --jinja flag is not optional here, it enables tool calling via the model’s chat template. Without it, Hermes Agent will not be able to make reliable tool calls against this model.
Option B: LM Studio
Download LM Studio, search for LFM2.5-2.6B in the model catalog, and download the Q4_K_M GGUF. In the Developer / Local Server tab: load the model, enable tool use in the model settings, set your context length, and click Start Server. It serves at http://localhost:1234.
Option C: vLLM or SGLang (GPU servers)
For a GPU server rather than a laptop:
pip install vllm
vllm serve LiquidAI/LFM2.5-2.6BSGLang requires an explicit tool-call parser flag:
uv pip install "sglang>=0.5.10"
sglang serve \
--model-path LiquidAI/LFM2.5-2.6B \
--host 0.0.0.0 \
--port 30000 \
--tool-call-parser lfm2Confirm your server is reachable before moving on (adjust the port to match your backend):
curl http://localhost:8080/v1/modelsStep 2: Install and configure Hermes Agent
Install Hermes Agent:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setupPoint it at your local server using the interactive wizard:
hermes model
# choose "Custom endpoint (self-hosted / vLLM / etc.)"
# API base URL: http://localhost:8080/v1
# API key: (leave empty for local)
# Model name: LFM2.5-2.6BOr configure it directly:
hermes config set model.provider custom
hermes config set model.base_url http://localhost:8080/v1
hermes config set model.default LFM2.5-2.6B
hermes config set model.context_length 131072
hermes config set model.api_mode chat_completions
hermes config set agent.tool_use_enforcement trueThat last line matters more than it looks. Per Liquid AI’s own setup guide: without tool_use_enforcement enabled, a small model like this “tends to describe actions instead of calling tools.” This is the single most common failure mode when pairing a small local model with an agent harness, the model talks about what it would do instead of actually invoking the tool. Enable enforcement and you get real tool calls.
Step 3: Run it
hermesYou now have Hermes Agent running entirely on your own hardware, with no API key, no per-token cost, and no data leaving your machine.
One troubleshooting note directly from Liquid AI’s documentation: if web_search is missing from the model’s available tools, check whether search or browser are listed in agent.disabled_toolsets. Remove both entries via hermes config edit and restart Hermes.
What to expect from a 2.6B model as an agent
Be realistic about where this setup shines and where it doesn’t. LFM2.5-2.6B is not going to out-reason a frontier model on complex multi-step planning or ambiguous instructions. What it’s good at is exactly what Liquid AI built it for: reliable tool calling, straightforward multi-step workflows, and staying responsive on modest hardware.
If you’re building a fully local agent for well-scoped, repeatable tasks (file organization, simple data extraction, scheduled checks, structured lookups), this pairing is a genuinely practical option. If your agent needs to handle open-ended reasoning or judgment calls, pair Hermes with a larger model instead, that’s the same hermes model command, just pointed at OpenRouter or Nous Portal instead of a local endpoint.
FAQ
Is Hermes Agent actually compatible with LFM2.5-2.6B, or does it just technically work?
It’s documented compatibility, not just theoretical. Liquid AI’s own documentation names Hermes Agent as a supported agent harness for LFM2.5-2.6B and publishes a complete setup guide with exact configuration commands, which is what this article is based on. This isn’t Hermes’ generic “works with any OpenAI-compatible endpoint” claim stretched to cover an untested pairing, Liquid AI tested and documented this specific combination.
Why do I need to enable tool_use_enforcement?
Small models are more prone to describing an action in text rather than emitting a proper tool call. Liquid AI’s documentation flags this explicitly for LFM2.5-2.6B inside agent harnesses. Enabling agent.tool_use_enforcement in Hermes fixes this by requiring the model to actually invoke tools rather than narrate them.
How much VRAM do I actually need?
As little as 1.67 GB for the Q4_K_M quantization, which Liquid AI recommends as the best balance of size and quality. Even the full BF16 precision weights only need about 5.4 GB. This runs on a laptop GPU, and the smaller quants work reasonably well on CPU-only inference too.
Can I use a different local model instead of LFM2.5-2.6B with Hermes Agent?
Yes, Hermes Agent works with any OpenAI-compatible endpoint, so any locally served model works in principle. The reason this guide specifically uses LFM2.5-2.6B is that it’s the smallest model we found with documented, vendor-confirmed compatibility and a tested setup guide for Hermes Agent specifically, rather than just theoretical API compatibility.
Does this work on a phone or only on a laptop?
This guide covers laptop/desktop setup via llama.cpp, LM Studio, vLLM, or SGLang. LFM2.5-2.6B is also small enough for phone-class hardware, but running the full Hermes Agent gateway (which expects a persistent server process) on a phone is a different, more involved setup than what’s covered here.