NVIDIA Nemotron 3.5 Lightning: The Execution Model for Long-Running AI Agents
Most coverage of a new model release asks âhow smart is it.â Thatâs the wrong question for NVIDIA Nemotron 3.5 Lightning. The right question is âhow cheaply and quickly can it execute the same well-defined step, thousands of times, inside a longer-running agent.â Lightning is not trying to be your smartest model. Itâs trying to be the model that does the boring, repetitive, high-volume work inside an agent loop while a bigger model does the thinking.
This is the planner/executor pattern, and Lightning is one of the first models explicitly built and marketed around it rather than around general chat benchmarks.
What Nemotron 3.5 Lightning actually is
Nemotron 3.5 Lightning is a 30B-parameter open Mixture-of-Experts model with only 3B active parameters per token, released by NVIDIA and developed with the Nemotron Coalition. Itâs distilled from NVIDIAâs larger Nemotron 3 Ultra model, which matters more than it sounds: distillation from a stronger teacher model is how you get a 3B-active model that doesnât fall apart on real tasks.
Key specs:
- Total parameters: 30B
- Active parameters: 3B per token
- Architecture: Hybrid LatentMoE, interleaving Mamba-2 and MoE layers with select attention layers
- Context window: up to 1,000,000 tokens
- Speculative decoding: DFlash, plus multi-token prediction (MTP)
- License: Open (open weights and training recipes)
- Hardware targets: RTX PCs, RTX PRO workstations, DGX Spark, DGX Station, datacenter, and cloud
The hybrid Mamba-2/MoE/attention design is what makes the 3B-active number meaningful rather than a marketing number. Mamba-2 layers handle long-range sequence processing with linear (not quadratic) compute scaling, MoE layers keep total capacity high without activating all of it, and the select attention layers preserve precise local reasoning where it matters. NVIDIA is explicit that this is not a general-purpose reasoning flagship: itâs built specifically for âhigh-volume, low-latency execution in always-on AI agents and agentic workflows.â
This guide is different from our Nemotron 3 family guide, which covers NVIDIAâs earlier Nano/8B/Super 120B lineup released around GTC 2026. Lightning is a newer, separate release focused specifically on the agent-execution role rather than being a smaller or larger version of the same family for general use.
The planner/executor pattern, explained
If youâve built or used a coding agent, youâve already seen the problem Lightning is trying to solve, even if you didnât name it. A single frontier model handling every step of a long agent loop is doing two very different jobs at once:
- Planning: deciding what to do next, reasoning about tradeoffs, interpreting ambiguous instructions, recovering from unexpected errors.
- Execution: running a specific, well-scoped tool call, formatting a response, extracting a value from a document, applying a known transformation, checking a condition.
Planning needs a strong, expensive model. Execution, in a well-designed agent, is usually a narrow, repeatable task that a much smaller model can handle correctly, if itâs been trained specifically for that role. Most teams donât split this today. They use one frontier model (Claude Sonnet 5, GPT-5.6, DeepSeek V4 Pro, whatever) for every single step of the loop, planning and execution alike, because thatâs the default and itâs simpler to build.
The cost problem shows up at scale. A long-running agent that makes hundreds or thousands of tool calls in a session pays frontier-model prices for every trivial execution step, not just the handful of genuinely hard planning decisions. Latency compounds the same way: if your agent architecture requires many small steps in sequence, each one waiting on a large modelâs inference time adds up.
Lightningâs pitch is to sit underneath a planner model as the execution layer: the planner (a frontier model) decides the strategy and hands off discrete, well-defined steps, and Lightning executes them fast and cheap, then reports back. You only pay planner-tier cost and latency for the steps that actually need planner-tier intelligence.
Why this matters for cost and latency, not just benchmarks
The economics are the actual story here, more than any single benchmark number:
- 3B active parameters means inference cost and latency scale closer to a 3B dense model than a 30B one, even though the model has 30B worth of total capacity to draw on via MoE routing.
- DFlash speculative decoding and MTP further reduce latency per generated token, which matters disproportionately in agent loops where youâre making many small, sequential calls rather than one long generation.
- Consumer hardware support (RTX PCs, not just DGX-class datacenter gear) means the execution layer of an agent can run locally, next to your application, instead of round-tripping to a cloud API for every trivial step.
Put together, this is the argument for treating âplanner costâ and âexecutor costâ as separate line items in your agentâs economics, the same way youâd separate a databaseâs read-heavy queries from its write-heavy queries and route them differently.
Where Lightning fits versus a frontier model doing everything
| Frontier model doing every step | Lightning as executor + frontier model as planner | |
|---|---|---|
| Cost per trivial tool call | Full frontier-model rate | Fraction of frontier-model rate |
| Latency per trivial step | Frontier-model inference time | Faster, especially with local deployment |
| Planning quality | High | Unchanged, planner model is unchanged |
| Architecture complexity | Simple, one model | Higher, requires a routing/handoff layer |
| Best for | Short agent sessions, low call volume | Long-running, high-call-volume agents |
The tradeoff is honest: this pattern adds architectural complexity. You need a way to route steps between the planner and the executor, decide which steps qualify for handoff, and handle cases where the executor gets something wrong and the planner needs to intervene. For a simple agent that makes a handful of tool calls per session, that complexity isnât worth it, just use one capable model. The pattern earns its cost when an agent runs for a long time and racks up a large number of narrow, repeatable execution steps, which is increasingly the norm for persistent agent frameworks and always-on automation rather than one-off chat sessions.
Getting started
Lightning is available through Ollama, Together AI, DeepInfra, and NVIDIAâs own NIM microservices, in addition to raw weights on Hugging Face.
Ollama (local)
ollama run nemotron3.5-lightning
vLLM (self-hosted, GPU)
pip install vllm
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
--max-model-len 1000000
NVIDIA NIM
NVIDIA publishes a dedicated getting-started guide for deploying Lightning via NIM, including recommended speculative decoding configurations (MTP, DFlash, or DSpark) for H100, H200, B200, and GB200 targets. If youâre already running other models through NIM, this is the path with the least new infrastructure to learn.
When to actually use this pattern
Reach for a planner/executor split, with Lightning or a similar small execution model, when:
- Your agent runs long sessions with many tool calls per session, not just a handful
- A meaningful share of those calls are narrow and repeatable (parsing, formatting, simple lookups, condition checks) rather than requiring judgment
- Youâre running cost-sensitive agent infrastructure at volume, where frontier-model pricing on every step adds up
- You have (or are willing to build) a routing layer that decides which steps go to which model
Skip it when your agent sessions are short, low-volume, or when most steps genuinely require planner-level reasoning. In that case, the added complexity of a two-model architecture costs more in engineering time than it saves in inference cost.
FAQ
Is Nemotron 3.5 Lightning a replacement for frontier models like Claude Sonnet 5 or GPT-5.6?
No, and NVIDIA doesnât position it that way. Itâs designed as an execution layer underneath a planner model, not a general-purpose replacement. Use a frontier model for planning and complex reasoning, and Lightning for the high-volume, narrow execution steps in between.
How does Lightningâs architecture keep costs low?
It activates only 3B of its 30B total parameters per token through a hybrid Mamba-2/MoE/attention design, and adds DFlash speculative decoding plus multi-token prediction to cut latency per generated token. The combination means inference cost and speed track closer to a 3B model than a 30B one.
Can I run Nemotron 3.5 Lightning on consumer hardware?
Yes. NVIDIA explicitly supports RTX PCs and RTX PRO workstations in addition to DGX-class and datacenter deployments, and itâs available via Ollama for straightforward local setup.
Whatâs the difference between this and NVIDIAâs earlier Nemotron 3 family?
The Nemotron 3 family (Nano 4B, 8B, Super 120B) targets general on-device and datacenter use cases across a range of sizes. Lightning is a separate, newer release specifically architected and marketed for the execution role inside long-running agent loops, with a hybrid Mamba-2/MoE architecture the earlier family doesnât use.
Do I need to build a custom planner/executor router myself?
Today, largely yes. Thereâs no universal standard for planner/executor handoff yet. Most teams building this pattern write their own routing logic based on task type (e.g., âis this a tool call with a well-defined schema, or does it require judgmentâ) rather than relying on an off-the-shelf framework. Expect this tooling to mature as small execution models become more common.