By Vilius Vystartas | May 2026
Ten more models through the same 10 agent coding tasks. Two tied the all-time record. One cost $0.0002. The other hit the score at $0.0018 — cheaper than most models scoring 70%.
Batch 10 was the cheapest one yet.
The Leaders
Two models scored 90% with zero hard fails, joining MiniMax M2 Her and Baidu Ernie 4.5 300B as the highest-scoring models on this benchmark:
Qwen3 Coder 30B A3B — 90% in 28 seconds, $0.0004. An efficient coder that doesn't burn budget on thinking tokens it doesn't need.
DeepSeek Chat (original) — 90% in 59 seconds, $0.0018. The original DeepSeek Chat still competes with modern models on agent coding. Newer doesn't always mean better.
The Surprises
LFM 2 24B A2B (85%, $0.0002, 15s) is the cheapest model I've ever tested. Liquid's debut family is absurdly cost-effective. A full 10-task benchmark for literally $0.0002. At this price/performance ratio, there's no excuse not to test a model before committing to a more expensive alternative.
Mistral Small 3.2 (85%, $0.0004) is a clear upgrade. The Small line went 75% → 85% across versions — a ten-point jump at the same budget tier. Mistral keeps improving the right things.
Qwen3 14B scored 0% across all 10 tasks. Mandatory thinking mode that can't be suppressed at 300 tokens means every request times out before producing output. Skip for agent coding.
Cydonia 24B V4.1 (80%, $0.001) debuts a new family from TheDrummer. Zero hard fails. Watch this one.
The Duds
Qwen3.7 Max (85%, $0.13, 295 seconds) scored the same as budget models costing 300x less. Thinking mode tax at work — the accuracy is there, but you'll wait five minutes and pay for every second.
Claude Opus 4 (80%, $0.10, 76s) had one hard fail. For a top-tier premium model at $0.10 per 10 tasks, that's below expectations. It's not a bad model — it's overkill for agent coding at a tight token budget.
Aion 1.0 (80%) had two hard fails and was the slowest at 160 seconds. The architecture is interesting, but it's not ready for production agent work.
My Picks
- Best overall: Qwen3 Coder 30B A3B — 90%, 28s, $0.0004
- Best value: LFM 2 24B A2B — 85%, $0.0002 total. Ridiculous price/performance.
- Fastest: LFM 2 24B A2B — 15 seconds flat
- Most improved: Mistral Small 3.2 — 75% → 85% across versions
- Skip entirely: Qwen3 14B for agent tasks
Methodology
Ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing — tested against each model via OpenRouter. Max tokens: 300. Temperature: 0.1. Results scored by pattern matching against expected outputs. Pre-flight verification caught 2 models (Ernie 4.5 21B — HTTP 429, Trinity Mini — empty content) before they wasted the batch.
Total batch cost: $0.14 across 9 models. Qwen3.7 Max alone accounted for $0.13 of that — thinking tax.
Total models tested: 148 (up from 138).
Full results and per-task scores: benchmarks.workswithagents.dev
Because you should.
