NVIDIA Nemotron 3.5 Lightning: The Execution Model for Long-Running AI Agents
Most coverage of a new model release asks “how smart is it.” That’s the wrong question for NVIDIA Nemotron 3.5 Lightning. The right question is “how cheaply and quickly can it execute the same well-defined step, thousands of times, inside a longer-running agent.” Lightning is not trying to be your smartest model. It’s trying to be the model that does the boring, repetitive, high-volume work inside an agent loop while a bigger model does the thinking.
This is the planner/executor pattern, and Lightning is one of the first models explicitly built and marketed around it rather than around general chat benchmarks.
What Nemotron 3.5 Lightning actually is
Nemotron 3.5 Lightning is a 30B-parameter open Mixture-of-Experts model with only 3B active parameters per token, released by NVIDIA and developed with the Nemotron Coalition. It’s distilled from NVIDIA’s larger Nemotron 3 Ultra model, which matters more than it sounds: distillation from a stronger teacher model is how you get a 3B-active model that doesn’t fall apart on real tasks.
Key specs:
- Total parameters: 30B
- Active parameters: 3B per token
- Architecture: Hybrid LatentMoE, interleaving Mamba-2 and MoE layers with select attention layers
- Context window: up to 1,000,000 tokens
- Speculative decoding: DFlash, plus multi-token prediction (MTP)
- License: Open (open weights and training recipes)
- Hardware targets: RTX PCs, RTX PRO workstations, DGX Spark, DGX Station, datacenter, and cloud
The hybrid Mamba-2/MoE/attention design is what makes the 3B-active number meaningful rather than a marketing number. Mamba-2 layers handle long-range sequence processing with linear (not quadratic) compute scaling, MoE layers keep total capacity high without activating all of it, and the select attention layers preserve precise local reasoning where it matters. NVIDIA is explicit that this is not a general-purpose reasoning flagship: it’s built specifically for “high-volume, low-latency execution in always-on AI agents and agentic workflows.”
This guide is different from our Nemotron 3 family guide, which covers NVIDIA’s earlier Nano/8B/Super 120B lineup released around GTC 2026. Lightning is a newer, separate release focused specifically on the agent-execution role rather than being a smaller or larger version of the same family for general use.
The planner/executor pattern, explained
If you’ve built or used a coding agent, you’ve already seen the problem Lightning is trying to solve, even if you didn’t name it. A single frontier model handling every step of a long agent loop is doing two very different jobs at once:
- Planning: deciding what to do next, reasoning about tradeoffs, interpreting ambiguous instructions, recovering from unexpected errors.
- Execution: running a specific, well-scoped tool call, formatting a response, extracting a value from a document, applying a known transformation, checking a condition.
Planning needs a strong, expensive model. Execution, in a well-designed agent, is usually a narrow, repeatable task that a much smaller model can handle correctly, if it’s been trained specifically for that role. Most teams don’t split this today. They use one frontier model (Claude Sonnet 5, GPT-5.6, DeepSeek V4 Pro, whatever) for every single step of the loop, planning and execution alike, because that’s the default and it’s simpler to build.
The cost problem shows up at scale. A long-running agent that makes hundreds or thousands of tool calls in a session pays frontier-model prices for every trivial execution step, not just the handful of genuinely hard planning decisions. Latency compounds the same way: if your agent architecture requires many small steps in sequence, each one waiting on a large model’s inference time adds up.
Lightning’s pitch is to sit underneath a planner model as the execution layer: the planner (a frontier model) decides the strategy and hands off discrete, well-defined steps, and Lightning executes them fast and cheap, then reports back. You only pay planner-tier cost and latency for the steps that actually need planner-tier intelligence.
Why this matters for cost and latency, not just benchmarks
The economics are the actual story here, more than any single benchmark number:
- 3B active parameters means inference cost and latency scale closer to a 3B dense model than a 30B one, even though the model has 30B worth of total capacity to draw on via MoE routing.
- DFlash speculative decoding and MTP further reduce latency per generated token, which matters disproportionately in agent loops where you’re making many small, sequential calls rather than one long generation.
- Consumer hardware support (RTX PCs, not just DGX-class datacenter gear) means the execution layer of an agent can run locally, next to your application, instead of round-tripping to a cloud API for every trivial step.
Put together, this is the argument for treating “planner cost” and “executor cost” as separate line items in your agent’s economics, the same way you’d separate a database’s read-heavy queries from its write-heavy queries and route them differently.
Where Lightning fits versus a frontier model doing everything
| Frontier model doing every step | Lightning as executor + frontier model as planner | |
|---|---|---|
| Cost per trivial tool call | Full frontier-model rate | Fraction of frontier-model rate |
| Latency per trivial step | Frontier-model inference time | Faster, especially with local deployment |
| Planning quality | High | Unchanged, planner model is unchanged |
| Architecture complexity | Simple, one model | Higher, requires a routing/handoff layer |
| Best for | Short agent sessions, low call volume | Long-running, high-call-volume agents |
The tradeoff is honest: this pattern adds architectural complexity. You need a way to route steps between the planner and the executor, decide which steps qualify for handoff, and handle cases where the executor gets something wrong and the planner needs to intervene. For a simple agent that makes a handful of tool calls per session, that complexity isn’t worth it, just use one capable model. The pattern earns its cost when an agent runs for a long time and racks up a large number of narrow, repeatable execution steps, which is increasingly the norm for persistent agent frameworks and always-on automation rather than one-off chat sessions.
Getting started
Lightning is available through Ollama, Together AI, DeepInfra, and NVIDIA’s own NIM microservices, in addition to raw weights on Hugging Face.
Ollama (local)
ollama run nemotron3.5-lightningvLLM (self-hosted, GPU)
pip install vllm
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
--max-model-len 1000000NVIDIA NIM
NVIDIA publishes a dedicated getting-started guide for deploying Lightning via NIM, including recommended speculative decoding configurations (MTP, DFlash, or DSpark) for H100, H200, B200, and GB200 targets. If you’re already running other models through NIM, this is the path with the least new infrastructure to learn.
When to actually use this pattern
Reach for a planner/executor split, with Lightning or a similar small execution model, when:
- Your agent runs long sessions with many tool calls per session, not just a handful
- A meaningful share of those calls are narrow and repeatable (parsing, formatting, simple lookups, condition checks) rather than requiring judgment
- You’re running cost-sensitive agent infrastructure at volume, where frontier-model pricing on every step adds up
- You have (or are willing to build) a routing layer that decides which steps go to which model
Skip it when your agent sessions are short, low-volume, or when most steps genuinely require planner-level reasoning. In that case, the added complexity of a two-model architecture costs more in engineering time than it saves in inference cost.
FAQ
Is Nemotron 3.5 Lightning a replacement for frontier models like Claude Sonnet 5 or GPT-5.6?
No, and NVIDIA doesn’t position it that way. It’s designed as an execution layer underneath a planner model, not a general-purpose replacement. Use a frontier model for planning and complex reasoning, and Lightning for the high-volume, narrow execution steps in between.
How does Lightning’s architecture keep costs low?
It activates only 3B of its 30B total parameters per token through a hybrid Mamba-2/MoE/attention design, and adds DFlash speculative decoding plus multi-token prediction to cut latency per generated token. The combination means inference cost and speed track closer to a 3B model than a 30B one.
Can I run Nemotron 3.5 Lightning on consumer hardware?
Yes. NVIDIA explicitly supports RTX PCs and RTX PRO workstations in addition to DGX-class and datacenter deployments, and it’s available via Ollama for straightforward local setup.
What’s the difference between this and NVIDIA’s earlier Nemotron 3 family?
The Nemotron 3 family (Nano 4B, 8B, Super 120B) targets general on-device and datacenter use cases across a range of sizes. Lightning is a separate, newer release specifically architected and marketed for the execution role inside long-running agent loops, with a hybrid Mamba-2/MoE architecture the earlier family doesn’t use.
Do I need to build a custom planner/executor router myself?
Today, largely yes. There’s no universal standard for planner/executor handoff yet. Most teams building this pattern write their own routing logic based on task type (e.g., “is this a tool call with a well-defined schema, or does it require judgment”) rather than relying on an off-the-shelf framework. Expect this tooling to mature as small execution models become more common.