I gave Opus a cheap coder to save money. Accuracy dropped and the bill went up

2026-11-03 · 4 min · eval

To save money on coding tasks, I gave Opus a cheap DeepSeek model to write the code. Accuracy dropped and the bill went up. Running Opus 5.5 alone reached 81% pass@1 at $1.20 per task, while pairing Opus with DeepSeek V4.1 Flash fell to 67% and raised the cost to $1.29 per task.

DeepSeek's Super-agent post tested agent scheduling using simulated coding, so I wanted to see how the pattern held up in practice. I tested 7 team setups on 12 real, hard coding tasks, and no team beat Opus alone on pass@1. The full table with 39 setups is here.

pass@1 on 12 tasks x 3 runs, with 95% CI. Blue bars show teams (Opus + another model) and black bars show single models.

TL;DR

• Single models offer the best value, with GPT-6.1 Terra reaching 78% and Sol 75% at $0.17 per task each.
• One prompt line made an Opus team cheaper than Opus alone, though accuracy still lagged behind the solo model.
• Teams pay to re-cache Opus's context while waiting for the worker to finish, which ends up costing more than the worker itself.

How I built the benchmark

My earlier mini-benchmarks built from my own agent trajectories had become too easy. For this evaluation, I pulled 12 tasks that match my daily work from Terminal-Bench 4, DeepSWE, SWE-rebench, SWE-bench Pro V2 and Longtail-SWE.

After inspecting the failed runs, I fixed broken verifiers and ambiguous task prompts in 9 of the 12 tasks. I also scored 14 apparent "solutions" as 0 because the agent simply downloaded the upstream fix. Every setup was then scored on the fixed tasks across 3 runs per task, 1,188 agent runs across 41 setups in total.

Going in, I was looking for two things: an orchestration setup that beats a single model without a big price jump, or a setup that runs much cheaper than a frontier model with almost no quality loss.

The setups

In the main setup, Opus runs in Claude Code to plan and review, and a rule blocks it from editing code directly. A cheaper worker writes the implementation instead: Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash or Qwen3.8 27B. The other six setups are shown in the diagram below.

Black boxes show Opus 5.5 in Claude Code, grey outlines show the cheap worker, and blue outlines show GPT-6.1 Sol in pi. Each card shows the best result of that setup.

Across these configurations, pairing Opus with a cheap worker solved an average of 55% of tasks at $1.25 per task. Even the strongest team topped out at 69%, behind solo Opus at 81% and $1.20 per task.

Why teams lose

71% of team failures happened on tasks that the worker also failed when running alone. In most failures, the worker was the limit.

Teams also failed on tasks where both models succeeded on their own. One task required building a tool that lets one agent hand work to another. The task prompt says nothing about empty input, and the hidden test accepts two behaviours: raising a stream error or running the sub-agent.

Acting as planner, Opus wrote DeepSeek a ticket with a third rule, "return an error message, but don't raise". DeepSeek followed it, and Opus approved the diff after noting that "all 8 new tests pass". The hidden test rejected the submission.

On that task, Opus alone passed 3/3 runs and DeepSeek alone passed 3/3, yet together they scored 2/3.

Where the money goes

The extra cost comes down to prompt caching. An agent resends its full context on every step and relies on the prompt cache to keep those repeated reads cheap. The cache lives for about 5 minutes, so when Opus waits 10-15 minutes for a worker to finish, the cache expires and Opus has to write its full context again on the next turn.

On our gateway, a cache write costs 25x a cache read ($5 vs $0.20 per million tokens). Across the team runs, Opus spent $0.52-0.84 per task on cache writes alone, while the whole worker run cost just $0.06-0.19. When Opus worked solo, cache writes came to $0.33.

Cost of one task, split by part, with Opus cache writes shown in blue.

What helped

No team beat solo Opus, but a few changes improved either accuracy or cost.

• Sol reviewing a cheap model. A review pass from Sol lifted DeepSeek from 56% to 64% and Qwen from 28% to 39%, for about $0.07 per task in review cost.
• Letting Opus code while telling it "your tokens cost 20x more than the worker's". With this line instead of the hard editing ban, the worker still wrote 94% of the code and pass@1 stayed at 67%. Opus's own spend fell from $1.19 to $0.59, bringing the total to $0.71 per task.
• The reviewer matters. Sol reviewing Opus at low effort fixed 3 runs and broke none, while Opus reviewing Sol fixed none and broke 4. Self-review helped neither model.

Limits

• 12 tasks leave a wide CI of about ±20 pp on pass@1, so differences under ~10 pp are noise unless a paired test says otherwise.
• Models are partly compared together with their harness: Claude Code for Anthropic models, Codex or pi for OpenAI, and OpenCode for DeepSeek.
• DeepSeek and Terra are priced at the public list price from token counts rather than a bill, and Qwen is unpriced.

Next

To see whether longer cache retention removes the handoff penalty, I plan to test a 1-hour cache TTL for Opus and measure the total bill. If there is another multi-agent setup you want to see, send me the models and how they should split the work.