Same score, different agents

2026-09-27 · 2 min · eval

Most configurations on my mini-benchmark now pass almost every task, so the score alone says little. I went through the trajectories — 12 configurations, 360 runs — to see how the agents work.

Opus 5.5 vs GPT-6 Luna

Two working styles, same result. Opus 5.5 reads everything, writes the solution in one go, checks it once and writes a long report: 4–7 steps per task. GPT-6 Luna explores in parallel, writes, then runs many small check-and-cleanup loops: 11–14 steps, and about half of its shell commands are housekeeping like rm -rf __pycache__, py_compile and git diff --check.

Higher effort mostly buys more checking. From low to high effort, Opus goes from 1.3 to 2.8 code runs per task; Luna goes from 3.5 to 5.1 between medium and xhigh. The first draft doesn't get noticeably better, and the pass rate was already at the ceiling — so here extra effort mainly costs time and tokens.

GLM, Qwen and DeepSeek

Weaker models think instead of act. GLM-5.3-Flash and Qwen3.8-27B write 120–170k characters of reasoning per task; Opus and Luna stay under 12k. Every GLM failure is the same: one message that hits the 64,000-token output limit without a single line of code — one of them contains "let me" 245 times and "Wait" 49 times. Qwen loops on the same edge case ("Actually…" 145 times), ends with "Now, let me write it all out." — and stops without calling a tool.

More testing doesn't make a model stronger. DeepSeek-V4.1-Flash runs code more than anyone — 8.4 times per task versus 1.3 for Opus 5.5 low — and gets to 93% by iterating until its tests are green. What separates models is the quality of the first write, and whether the model starts acting at all.

Harnesses

The model keeps its style. In Claude Code, Opus 5.5 high works exactly as in Pi: 6.5 steps instead of 6.7, one big write, about three code runs, 30 of 30 passed. Oh My Pi pushes GPT-6 Luna further into its incremental style — 28 tool calls per task instead of 18, almost every change as a small edit — and it fails one task in all three runs that Luna passes every time in Pi.

The harness sets the token bill. Every step re-reads a much larger context: about 30k tokens per step for Opus in Claude Code versus 11k in Pi, most likely the bigger system prompt and tool definitions. The same work costs 227k tokens per task instead of 98k, and Luna in Oh My Pi uses 776k instead of 290k.

On tasks of this size, the interesting differences are cost, speed and working style — not the score. Current numbers are on the leaderboard.