Laguna S 2.1 Benchmarks: Why 8B Active Params Beats Models 14x Its Size
The benchmark results for Laguna S 2.1 are genuinely unusual. An MoE model with 8B active parameters per token should not be beating models with 49B active params on real-world coding benchmarks. But it does, and understanding why tells us something important about where AI coding models are headed.
This article breaks down each benchmark, what it actually measures, and why Laguna’s numbers are so remarkable relative to its size.
The Full Benchmark Picture
| Benchmark | Laguna S 2.1 | Score Context |
|---|---|---|
| Terminal-Bench 2.1 (thinking) | 70.2% | #1 in weight class |
| Terminal-Bench 2.1 (no thinking) | 60.4% | Still beats DeepSeek V4 (64.0% is V4’s thinking score) |
| SWE-bench Pro | 59.4% | Beats DS-V4 (55.4%), close to Sonnet 5 (63.2%) |
| SWE-bench Multilingual | 78.5% | Excellent polyglot performance |
| DeepSWE | 40.4% | Destroys DS-V4 (9.0%), strong for open-weight |
Now let me explain what each of these actually measures and why Laguna’s scores matter.
Terminal-Bench 2.1: 70.2%
What It Measures
Terminal-Bench tests a model’s ability to work as a terminal-based coding agent. That means:
- Navigating file systems
- Reading and understanding code in context
- Writing and editing files
- Running commands and interpreting output
- Iterating on solutions when the first attempt fails
- Multi-step problem solving across turns
This is not a “generate a function” benchmark. It is a “here is a real development task, figure it out using a terminal” benchmark. It tests exactly the workflow you use when running a model through Claude Code, the pool CLI, or Aider.
Where 70.2% Sits
| Model | T-Bench 2.1 | Active Params | Size Ratio vs Laguna |
|---|---|---|---|
| Kimi K3 | 88.3% | Large (2.8T total) | 24x |
| Claude Fable 5 | 88.0% | Closed | N/A |
| Muse Spark 1.1 | 80.0% | Closed | N/A |
| Tencent Hy3 | 71.7% | 21B | 2.6x |
| Laguna S 2.1 | 70.2% | 8B | 1x |
| DeepSeek V4 Pro Max | 64.0% | 49B | 6.1x |
| Inkling | 63.8% | 41B | 5.1x |
Laguna S 2.1 beats DeepSeek V4 Pro Max (6x more active params) and Inkling (5x more active params). It nearly matches Tencent Hy3 (2.5x more active params). The only models that significantly outperform it are either closed-source (Fable 5, Muse Spark) or 24x larger (Kimi K3).
The efficiency per active parameter is unmatched in the industry.
The Thinking Mode Factor
Without thinking: 60.4%. With thinking: 70.2%. That is a 9.8 percentage point jump (16.2% relative improvement). Thinking mode gives Laguna time to plan multi-step approaches before executing. For Terminal-Bench tasks that require sequential reasoning, this planning step is critical.
Read the full thinking mode guide for implementation details.
SWE-bench Pro: 59.4%
What It Measures
SWE-bench Pro takes real GitHub issues from real open-source projects and asks models to produce correct patches. The issues come from repositories like Django, Flask, scikit-learn, and other well-known projects. Success means:
- Correctly understanding the issue description
- Navigating the relevant codebase
- Producing a patch that resolves the issue
- Passing the existing test suite
This is closest to “can this model actually fix bugs in production code?” of any benchmark.
Where 59.4% Sits
| Model | SWE-Pro | Cost (in/out per 1M) |
|---|---|---|
| Claude Fable 5 | 80.3% | Closed, expensive |
| Grok 4.5 | 64.7% | $2/$6 |
| Claude Sonnet 5 | 63.2% | $2/$10 |
| Laguna S 2.1 | 59.4% | $0.10/$0.20 (free at 256K) |
| Tencent Hy3 | 57.9% | $0.50/$2.00 |
| DeepSeek V4 Pro Max | 55.4% | $0.40/$1.60 |
Laguna S 2.1 at 59.4% is only 3.8 points behind Sonnet 5 ($2/$10) and 5.3 points behind Grok 4.5 ($2/$6). Those models cost 20-50x more. For the price-performance ratio, Laguna dominates. See the full Laguna vs Sonnet 5 comparison for a detailed breakdown.
DeepSWE: 40.4%
What It Measures
DeepSWE is the hardest agentic coding benchmark. It tests deep, multi-step problem solving that requires:
- Extended reasoning chains
- Navigating unfamiliar codebases
- Making architectural decisions
- Recovering from wrong approaches
- Sustained coherent execution across many steps
If Terminal-Bench is “can you use a terminal effectively?” and SWE-bench Pro is “can you fix this bug?”, DeepSWE is “can you architect and implement a complex feature from scratch?”
Why Laguna’s Score Matters
| Model | DeepSWE |
|---|---|
| Laguna S 2.1 (thinking) | 40.4% |
| Laguna S 2.1 (no thinking) | 16.5% |
| DeepSeek V4 Pro Max | 9.0% |
The gap between Laguna (40.4%) and DeepSeek V4 Pro Max (9.0%) is staggering. A model 14x smaller absolutely destroys the larger model on the hardest benchmark. This is not a minor difference. Laguna is 4.5x better on DeepSWE.
This aligns with Poolside’s case studies:
- Building a browser engine from scratch (50 minutes, 181 steps) requires exactly the sustained multi-step execution DeepSWE tests
- Optimizing a training harness (5.2% faster, 70% less memory) requires deep architectural reasoning
- Re-deriving Erdos problem #397 (50+ year open problem) requires extended mathematical reasoning
The Thinking Mode Impact on DeepSWE
The thinking mode impact on DeepSWE is the most dramatic of any benchmark: 16.5% without thinking to 40.4% with thinking. That is a 145% relative improvement. For deep reasoning tasks, thinking mode is not optional. It is transformational.
SWE-bench Multilingual: 78.5%
What It Measures
Same concept as SWE-bench Pro, but across multiple programming languages. This tests whether a model can handle codebases in Python, JavaScript, TypeScript, Java, Go, Rust, and other languages.
Why 78.5% Is Impressive
This is Laguna’s highest benchmark score and demonstrates strong polyglot capability. Many code models are Python-dominant because training data skews heavily toward Python. Scoring 78.5% across languages suggests Poolside’s 409K training environments included significant diversity in languages and frameworks.
For comparison, XS 2.1 scores 63.1% on this benchmark (up from 57.7% on XS 2.0), showing that even the smaller model handles multilingual code well.
What Explains the Efficiency?
How does 8B active params beat 49B active params? Three training innovations:
RLCEF (Reinforcement Learning with Code Execution Feedback)
Traditional RLHF uses human preferences to guide training. RLCEF uses actual code execution results. The model writes code, runs it, sees if it passes tests, and learns from the outcome. This produces a model optimized for “code that works” rather than “code that looks good to a human reviewer.”
409K Training Environments
Scale of training environments matters more than scale of parameters. Each environment is a realistic software engineering scenario where the model must operate. 409K unique environments provide extraordinary coverage of real-world coding patterns.
Multi-harness Rollouts
During training, the model attempts each problem multiple ways. It learns not just from successes but from failures. This teaches robust reasoning and recovery strategies, which is exactly what DeepSWE and Terminal-Bench require.
The combination means Laguna extracts more capability per parameter than models trained with conventional methods. It is not that 8B params is inherently enough. It is that Poolside’s training methodology makes 8B active params sufficient for this performance level.
Comparison Summary Table
| Model | T-Bench | SWE-Pro | DeepSWE | Active Params | Cost |
|---|---|---|---|---|---|
| Claude Fable 5 | 88.0% | 80.3% | N/A | Closed | Expensive |
| Kimi K3 | 88.3% | N/A | N/A | Large | $3/$15 |
| Muse Spark 1.1 | 80.0% | N/A | N/A | Closed | $1.25/$4.25 |
| Tencent Hy3 | 71.7% | 57.9% | N/A | 21B | $0.50/$2.00 |
| Laguna S 2.1 | 70.2% | 59.4% | 40.4% | 8B | Free-$0.20 |
| Grok 4.5 | N/A | 64.7% | N/A | Closed | $2/$6 |
| Claude Sonnet 5 | N/A | 63.2% | N/A | Closed | $2/$10 |
| DeepSeek V4 | 64.0% | 55.4% | 9.0% | 49B | $0.40/$1.60 |
For complete pricing across all models, see our AI API pricing comparison.
What These Benchmarks Do Not Tell You
No benchmark is perfect. Things Laguna’s scores do not capture:
- Inference speed: Benchmarks measure accuracy, not tokens per second. Local deployments vary based on hardware.
- Instruction following precision: Sometimes you want a model to do exactly what you say, not what it thinks is best.
- Code style and readability: Passing tests does not mean the code is maintainable.
- Long conversation coherence: Benchmarks are typically single-task. Real conversations drift.
The case studies partially address these gaps (a 181-step browser engine build tests conversation coherence), but keep in mind that benchmarks are indicators, not guarantees.
For guidance on choosing between models in practice, see our best AI coding tools guide and best open-source coding models.
FAQ
What is Terminal-Bench 2.1 and why does it matter?
Terminal-Bench tests models as terminal coding agents: navigating files, writing code, running commands, and solving problems across multiple steps. It is the closest benchmark to how developers actually use AI coding assistants in tools like the pool CLI or Claude Code.
Is DeepSWE harder than SWE-bench Pro?
Yes, significantly. SWE-bench Pro tests bug fixing in existing codebases. DeepSWE tests deep, multi-step problem solving that requires architecture decisions, sustained reasoning, and recovery from mistakes. Laguna’s 40.4% on DeepSWE vs 59.4% on SWE-Pro reflects this difficulty gap.
Does thinking mode always improve performance?
For simple, short tasks the improvement is marginal and adds latency. For complex multi-step tasks, thinking mode is transformational (16.5% to 40.4% on DeepSWE). Use non-thinking for quick completions and thinking for anything requiring planning.
How reliable are these benchmarks for predicting real-world performance?
They correlate well but are not perfect. The case studies (browser engine, harness optimization, Erdos problem) demonstrate that benchmark performance translates to real capability. But your specific use case may differ from benchmark distributions.
Why does Laguna beat much larger models?
Training methodology. RLCEF with 409K environments and multi-harness rollouts teaches the model to write working code through execution feedback, not just to produce plausible-looking code. This approach extracts more capability per parameter than conventional training.